Jérôme David

Jérôme Euzenat
email: jerome.david|euzenat@inrialpes.fr

On fixing semantic alignment evaluation measures

The evaluation of ontology matching algorithms mainly consists of comparing a produced alignment with a reference one. Usually, this evaluation relies on the classical precision and recall measures. This evaluation model is not satisfactory since it does not take into account neither the closeness of correspondances, nor the semantics of alignments. A first solution consists of generalizing the precision and recall measures in order to solve the problem of rigidity of classical model. Another solution aims at taking advantage of the semantic of alignments in the evaluation. In this paper, we show and analyze the limits of these evaluation models. Given that measures values depend on the syntactic form of the alignment, we first propose an normalization of alignment. Then, we propose two new sets of evaluation measures. The first one is a semantic extension of relaxed precision and recall. The second one consists of bounding the alignment space to make ideal semantic precision and recall applicable.

Introduction

With the semantic Web, many related but heterogenous ontologies are being created. In such an open context, there is no reason why two domain-related applications would share the same ontologies. In order to facilitate the exchange of knowledge between such applications, ontology matching aims at discovering a set of relations between entities from two ontologies. This set of relations is called an alignment.

Many different matching algorithms have been designed [START_REF] Euzenat | Ontology matching[END_REF]. In order to compare the performance of such algorithms, some efforts are devoted to the evaluation of ontology matching tools. Since 2004, the Ontology Alignment Evaluation Initiative1 (OAEI) organizes, every year, an evaluation of ontology matching methods. The evaluation of matching algorithms consists of comparing a produced alignment with a reference one. This evaluation often relies on two classical measures used in information retrieval: precision and recall [START_REF] Joost | Information Retrieval[END_REF]. Precision measures the ratio of correct correspondences in the evaluated alignment. Recall measures the ratio of reference correspondence found by the evaluated alignment.

In the context of alignment evaluation, precision and recall present the drawbacks to be all-or-nothing measures [START_REF] Ehrig | Relaxed precision and recall for ontology matching[END_REF] and they do not consider neither the semantic of alignment relations, nor those of ontologies. Then, an alignment can be very close to the expected result and have low precision and recall values. Two approaches have been proposed for correcting these drawbacks. [START_REF] Ehrig | Relaxed precision and recall for ontology matching[END_REF] introduced a generalization of precision and recall measures. This approach relies on syntactic measures relaxing the all-or-nothing feature of classical measures in order to take into account close correspondences. [START_REF] Euzenat | Semantic precision and recall for ontology alignment evaluation[END_REF] has introduced semantic precision and recall measures which rely on a semantic of alignments

We will show that semantic precision and recall are still dependent on the alignment syntax and, as a consequence, they can assign different values to semantically equivalent alignments. [START_REF] David | AROMA : une méthode pour la découverte d'alignements orientés entre ontologies à partir de règles d'association[END_REF] proposes to use the ideal semantic precision and recall measures introduced in [START_REF] Euzenat | Semantic precision and recall for ontology alignment evaluation[END_REF] restricted to alignments containing only simple correspondences, i.e. only between named entities.

In this paper, we show and analyze the limits and problems of the semantic precision and recall measures. To overcome their drawbacks, we first investigate an approach allowing to normalize alignments. This normalization relies on algebra of alignment relations and can partially resolves problems encountered by the evaluation measures.

In addition, we propose two adaptations of the relaxed and semantic measures. The first adaptation makes use of the generalization framework of [START_REF] Ehrig | Relaxed precision and recall for ontology matching[END_REF] and allows to locally consider the semantic of alignments. The second one is a restriction of Semantic closures of alignment. This restriction makes the ideal semantic measures proposed in [START_REF] Euzenat | Semantic precision and recall for ontology alignment evaluation[END_REF] useable.

This paper is organized as follows: a first section introduces the definitions related to the syntax and semantics of alignments. In the second section, we first present and introduce five properties that an ideal model should satisfy. Then, we present the classical evaluation measures and the semantic evaluation measures which satisfy three of the five desired properties. In the following section, we explain why these semantic measures do not satisfy the two last properties. The last section proposes three ways for fixing the semantic measures: a normalization of alignments, new relaxed semantic precision and recall measures, and Λ-bounded semantic evaluation measures.

2 Ontology alignment: syntax and semantic

Definition and syntax

An alignment groups correspondences between entities or formulas from two ontologies o 1 and o 2 . Each element of correspondence can be associated to a quality value by a function q. We use the following syntax for representing an alignment: Definition 1 (Alignment). An alignment between two ontologies o 1 and o 2 is a set of correspondances holding between o 1 and o 2 . A correspondance, noted c = (x, y, R), is a triple where x, respectively y, are formulas (or entities) from o 1 , respectively from o 2 , and R is the relation holding between x and y. A correspondance c = (x, y, R) can be also written xRy This definition includes both simple alignments considering only matching relations between entities (classes or properties) and complex alignments containing relations between formula inferred from the ontologies.

Semantic of alignments: Semantic closure and semantic reduction

Alignments between ontologies can be helpful for reasoning with several ontologies. For enabling reasoning capabilities, a semantic for alignments must be defined. In this paper, we relies on the semantic proposed in [START_REF] Euzenat | Semantic precision and recall for ontology alignment evaluation[END_REF]. This semantic of alignment is function of the semantics of each individual ontology. The semantic of an ontology is given by its set of models.

γ o • m ∈ M(o), γ o • m ∈ M(o ) and γ o (m(e)), γ o (m (e )) ∈ R U This is denoted as m, m |= γ c.
Given an alignment between two ontologies, the semantics of the aligned ontologies can be defined as follows.

Definition 6 (Models of aligned ontologies). Given two ontologies o and o and an alignment A between these ontologies, a model m of these ontologies aligned by

A is a triple m, m , γ ∈ M(o) × M(o ) × Γ , such that m, m |= γ A.
We will consider a specific kind of consequence, α-consequences [START_REF] Euzenat | Semantic precision and recall for ontology alignment evaluation[END_REF], which are the correspondences holding for all models of aligned ontologies. Given this semantic, the semantic closure and semantic reduction of an alignment are given by the following definitions:

Definition 8 (Semantic closure). The semantic closure Cn(A) of an alignment A is the set of its α-consequences.

Obviously , the semantic closure of an alignment is unique but it has no reason to be finite.

Definition 9 (Semantic reduction). A semantic reduction (or minimal cover) A 0 of an alignment A is an alignment satisfying Cn(A 0 ) = Cn(A) and ∀c ∈ A 0 , Cn(A 0 -{c}) = Cn(A)

There could exist several semantic reductions for a given alignment. An alignment

A contains redundant elements if A is not a minimal cover. A correspondence c ∈ A is redundant if A -{c} |= c.

Evaluation models

Alignment evaluation is achieved by comparing the produced alignment with the reference one. This comparison usually relies on the precision (P ) and the recall (R) measures [START_REF] Joost | Information Retrieval[END_REF]. Intuitively, the precision aims at measuring the correctness of the evaluated alignment. The recall is used for quantifying the completeness of the evaluated alignment.

In the rest of this paper, we will consider two alignments between ontologies o 1 and o 2 : a reference alignment, noted A r , and an alignment produced by some matching method A e .

Desired properties of evaluation measures

If we consider that precision and recall should approximate correctness and completeness, an ideal model taking semantic into account, would respect the constraints given by [START_REF] Euzenat | Semantic precision and recall for ontology alignment evaluation[END_REF]:

-A r |= A e ⇒ P (A e , A r ) = 1 (max-correctness) -A e |= A r ⇒ R(A e , A r ) = 1 (max-completeness) -Cn(A e ) = Cn(A r ) iff P (A e , A r ) = 1 and R(A e , A r ) = 1 (definiteness)
Furthermore, in the evaluation context, one could be interested to compare several alignments produced by some matching algorithms against one reference alignment. Then, it would be useful that two semantically equivalent alignments have the same precision and recall values.

-

Cn(A e1 ) = Cn(A e2 ) ⇒ P (A e1 , A r ) = P (A e2 , A r ) and R(A e1 , A r ) = R(A e2 , A r ) (semantic-equality)
Finally, if the evaluated and the reference alignments share some common information then the precision and recall values must not be null:

-P (A e , A r ) = 0 and R(A e , A r ) = 0 iff Cn(A e ) ∩ Cn(A r ) = Cn(∅) (overlapping positiveness)

Classical evaluation model

The classical evaluation model is based on the interpretation of alignments as sets and considers the following sets :

-E: the set of all correspondences that could be generated between o 1 and o 2 . This set is a subset of the cartesian product of all entities which can be deduced from o 1 , those deductible for o 2 and the set of matching relations considered. -true-positives: the set of correspondences which are found by the matching method and contained in the reference alignment. -false-positives: the set of correspondences which are found by the matching method but not contained in the reference alignment. -false-negatives: the set of reference correspondences which are not found by the matching method. -true-negatives: the set of correspondences that are neither in the evaluated alignment nor in the reference alignment. The cardinalities of these sets are given in the contingency table 1. The sets of truepositives, false negatives, and false-positives are defined only from A e and A r . The set of true-negatives is also function of the set E which is not easily identifiable.

From these contingencies, the classical measure of precision and recall can be defined. The precision (P ) represents the proportion of found correspondences that are relevant:

P (A e , A r ) = |A e ∩ A r | |A e | (1)
The recall (R) represents the proportion of relevant correspondences that have been found :

R(A e , A r ) = |A e ∩ A r | |A r | (2)

Limitations of classical precision and recall

These two measures applied to this simple model have the advantages to be easily computable and understandable. However, they verify none of the constraints presented Section 3.1. This is because they do not consider the semantic of alignment relations, nor the semantic of ontologies. Firstly, they do not take into account the semantic of matching relations. For example, if the produced alignment A e contains the elements x y and x y, and the reference alignment A r contains the element x ≡ y, then the classical model will consider x y and x y as false-positives and x ≡ y as false-negative. In this case the precision and recall values are equals to 0 even if A e ≡ A r .

Secondly, this classical model does not take the semantic of ontologies into account. For example, A e contains the element x y, the reference alignment A r contains the element x ≡ y, and the ontology o 1 states x

x. Even if A r |= A e , the classical precision will be equal to 0 since the correspondence x y is considered as a falsepositive by this evaluation model.

Semantic evaluation models

In order to resolve the drawbacks of classical precision and recall, [START_REF] Euzenat | Semantic precision and recall for ontology alignment evaluation[END_REF] proposes to take into account the semantics of matching relations and ontologies. The author provides two extensions of precision and recall.

The ideal extension of the classical model consists of replacing A e and A r by their respective sets of α-consequences, Cn(A e ) and Cn(A r ). Table 2 show the new contingencies. From this extended model, ideal precision and recall measures, respectively named P i and R i , are :

P i (A e , A r ) = |Cn(A e ) ∩ Cn(A r )| |Cn(A e )| (3) R i (A e , A r ) = |Cn(A e ) ∩ Cn(A r )| |Cn(A r )| (4)
These measures correct the drawbacks of the classical model and all the properties given Section 3.1 are satisfied. However, they bring a new problem : as the semantic closures of alignments could be infinite, then the measures may be undefined.

In order to overcome this problem, [START_REF] Euzenat | Semantic precision and recall for ontology alignment evaluation[END_REF] introduces two new measures known as semantic precision and semantic recall.

Semantic precision measures the proportion of evaluated correspondances of A e that can be deduced from A r .

P s (A e , A r ) = |A e ∩ Cn(A r )| |A e | (5)
Semantic recall measures the proportion of reference correspondances of A r that can be deduced from A e .

R s (A e , A r ) = |Cn(A e ) ∩ A r | |A r | (6)
With these measures, the max-correctness, max-completeness, and definiteness properties are preserved. The values of semantic precision and semantic recall are greater than or equal to those of classical ones because

|A e ∩ Cn(A r )| > |A e ∩ A r | and |Cn(A e ) ∩ A r | > |A e ∩ A r |.

Limitations of semantic precision and recall

Semantic precision and recall correct some drawback of classical precision and recall measure since they satisfy the max-correctness, max-completeness, and definiteness properties. Nevertheless, they do not satisfy the semantic-equality and overlappingpositiveness properties which we have introduced. This is due to the fact that these semantic measures are still dependent on the syntactic form of the alignments.

Limitation concerning semantic-equality property

Two alignments A e1 and A e2 having the same closure and then, semantically equivalent, could have different precision and recall values according to A r . This due to the fact that the semantic precision and recall are directly function of the cardinalities of the correspondences sets which could be different for two semantically equivalent alignments.

We give two examples demonstrating that semantic evaluation measures do not satisfy the semantic-equality property. In the first example, we reason only with alignment. In the second example, we show that redundancy in alignment can break the satisfaction of semantic-equality property by precision measure.

In the first example, we consider two alignments A e1 = {x ≡ y, u ≡ v} and A e2 = {x y, x y, u ≡ v}. These two alignments are equivalent since we have only replaced the equivalence x ≡ y of A e1 by x y and x y in A e2 . According to a reference alignment A r = {x ≡ y}, the two alignments do not have the same precision values: P s (A e1 , A r ) = 1/2 and P s (A e2 , A r ) = 2/3.

In the second example, we now have the alignments A e1 = {x ≡ y, u ≡ v} and A e2 = {x y, x ≡ y, u ≡ v}, and the knowledge o 1 |= x x. These two alignments are equivalent since x y is redundant according to x ≡ y. Nevertheless, the semantic precision values will be different: P s (A e1 , A r ) = 1/2 and P s (A e2 , A r ) = 2/3

Limitation concerning overlapping-positiveness property

An alignment could have a null precision or/and recall value even if the intersection of its consequence sets and those of the reference is not the empty set. This due to the fact that the semantic precision and recall partially take the alignment semantic into account. A correspondance can entail several correspondances. Such a correspondance can be partially true-positive in the sense that it entails a true-positive element but also a false-negative or false-positive element. With the semantic precision and recall, such elements are entirely considered as false-positives or/and false-negatives.

For example, let be the two alignments A e = {a} and A r = {b}, another matching relation c and the properties a |= c, b |= c, a |= b and b |= a. On this trivial example, the semantic precision and recall values are both equals to 0 even if the intersection of their Semantic closures is not equals to the empty set (i.e. c ∈ Cn(A e ) ∩ Cn(A r )).

Corrections of semantic evaluation measures

In previous section, we highlighted some drawbacks of classical precision and recall and semantic precision and recall. The first kind of problems concerns the inability of classical and generalized precision and recall measures to reason with the alignment relations. The semantic precision and recall try to resolve this problem by using Semantic closures, but these measures are still defined on the alignment cardinality which is dependent on the syntactic form of the alignment. As a consequence, there are some cases where the semantic-equality property is not satisfied.

When this problem is entirely due to the syntactic form the alignments, we may try to resolve it by normalizing the alignment representation. We propose here a normalization strategy which relies on algebras of alignment relations [START_REF] Euzenat | Algebras of ontology alignment relations[END_REF].

Then, with the help of alignment normalization, we propose two new sets of evaluation measures. The first one concerns relaxed semantic measures based on the generalized precision and recall framework of [START_REF] Ehrig | Relaxed precision and recall for ontology matching[END_REF]. Contrarily to the original generalized precision and recall measures provided in the aforementioned paper, these new measures are not only based on the syntactic form of alignment, but also on the semantic of alignments.

The second set of measures is an adaptation of ideal semantic measures of [START_REF] Euzenat | Semantic precision and recall for ontology alignment evaluation[END_REF].

Normalization of alignments

For allowing measures to respect the semantic-equality property, it is useful to introduce a notion of a normal form for alignments. A normal form for alignments ensures that two semantically equivalent alignments have always the same syntax or form. Naturally, it is a very difficult problem but we can propose a partial solution which only considers the semantic of alignment relations. Our notion of normal form takes benefit of entailment capabilities provided by algebras of alignment relations and does not use any knowledge about the aligned ontologies.

An algebra of alignment relations [START_REF] Euzenat | Algebras of ontology alignment relations[END_REF] is a particular type of relation algebra [START_REF] Tarski | On the calculus of relations[END_REF] defined by the tuple 2 Γ , ∩, ∪, •, Γ, ∅, {≡}, -1 where Γ is the set of all elementary relations; ∩ and ∪ are set-operations used to meet and join two sets of relations, for example, if xRy or xR y then xR ∪ R y; • is the composition operator, i.e. an associative internal composition law with {≡} as unity element; -1 the converse operator. For instance, if Γ = { , , ≡, , ⊥}, all all elementary relations, except and , are there own converse and, -1 = and -1 = .

Such an algebra allows to write any relation between entities (or formulas) as a disjunction of elementary relations. For example, x y would be written x{ , ≡}y. With the help of this relation algebra, any pair of entities or formulas will appear at most once in the alignment.

Definition 10. An alignment in normal form is an alignment A = (V, q) where the set of correspondances V satisfies the following properties:

1. V ⊂ {xRy|x ∈ o 1 ∧ y ∈ o 2 ∧ R ⊆ Γ }:
all relations between two entities (or formulas) are written with a disjunction of elementary relations. 2. ∀xRy ∈ V, ∃xR y ∈ V, R = R : any pair of entities (or formulas) appear at most once in the alignment.

Using such a normalization allows to correct classical and semantic precision and recall when relations between entities or formulas are split into several correspondances. For example, let be A e = {x y, x y} and A r = {x ≡ y}. By rewriting these alignments using disjunction of elementary relations, A e = {x{ , ≡} ∩ { , ≡}y} = {x{≡}y} and A r = {x{≡}y} will be syntactically equivalent.

Of course, when this problem is due to the semantic of ontologies such a normalization is not sufficient. For example, let be A e = {x y} and A r = {x z} and the axiom y ≡ z ∈ o 2 . These alignments are equivalent (given the previous axiom), but their normalization (A e = {x{ , ≡}y} and A r = {x{ , ≡}z}) are not equal.

Relaxed semantic precision and recall

In generalized precision and recall framework, evaluation measures are function of a measure quantifying the proximity between two correspondences [START_REF] Ehrig | Relaxed precision and recall for ontology matching[END_REF]. We propose new proximity measures σ dealing partially with the semantic of alignments. We want such measures to locally respect the max-correctness and maxcompleteness properties contrarily to those provided in [START_REF] Ehrig | Relaxed precision and recall for ontology matching[END_REF]:

-if x R y |= xRy then σ prec (xRy, x R y ) = 1 (local max-correctness) -if xRy |= x R y then σ rec (xRy, x R y ) = 1 (local max-completeness)
In order to propose such measures, we suggest to take advantage of an algebra of alignment relations as presented in the previous section (Section 5.1). Following the example of the relaxed precision and recall, which are oriented, we introduce two new σ measures: σ prec for precision and σ rec for the recall. In these two σ measures, we do not consider the confidence values.

In a first instance, we only consider the case where we have two correspondances aligning the same entities or formulas. Let xRy and xR y be two such correspondances. From the algebra of alignment relations, we have the following properties:

-if xR y |= xRy, then R ⊆ R, -if xRy |= xR y, then R ⊆ R .
Hence, σ prec and σ rec are defined by:

σ prec (R, R ) = |R ∩ R | |R | (7) σ rec (R, R ) = |R ∩ R | |R| (8)
Now, for extending these measures to correspondances which do not align the same entities or formulas, we propose to use this relation algebra also with the ontologies.

Definition 11 (Relaxed semantic proximity measures). Given an evaluated relation xRy, a reference relation x R y , and relations deduced from ontologies, o 1 |= xR 1 x and o 2 |= yR 2 y , the relaxed semantic proximities σ prec and σ rec are defined by:

σ prec (xRy, x R y ) = |R ∩ R 1 • R • R -1 2 | |R 1 • R • R -1 2 | (9) σ rec (xRy, x R y ) = |(R -1 1 • R • R 2 ) ∩ R | |R -1 1 • R • R 2 | (10)
The relaxed semantic proximity measures satisfy the local max-correctness and local max-completeness properties. As a consequence, they allow to provide relaxed semantic precision and recall measures which partially deals with alignment semantics. However, such measures do not consider the whole alignment semantic and then, they do not necessarily satisfy any property mentioned Section 3.1.

In our opinion, these semantic proximity measures are a first step for providing new semantic evaluations measures satisfying the desired properties. However, for satisfying these properties, it would be essential to propose new generalized precision and recall measures.

Restriction of ideal precision and recall

In order to deals with the semantic of alignments on one hand, and ideal precision and recall on the other hand, we first propose to use a partial closure of alignment instead of its full closure (α-consequence set). This partial closure has the advantage to be finite but in counterpart, it is defined relatively to a set of alignments. As a consequence, the ideal precision and recall can be computed, but their values depend on the set of considered alignments Λ. In a the case of evaluation campaigns, the set Λ = A e1 ∪ ... ∪ A en ∪ A r will contain all correspondences provided by the participants, and the reference alignment.

Definition 12 (Bounded closure of an alignment). The bounded closure of an alignment V given an alignment Λ (V ⊆ Λ) is defined as a set of correspondances issued from Λ which can be deduced from V .

V +/Λ = Cn(V ) ∩ Λ (11)
The bounded closure V +/Λ of an alignment V is finite when Λ is finite (i.e. each alignment in Λ is finite). From this bounded closure definition, we provide Λ-bounded precision and recall.

Definition 13 (Λ-bounded precision measure). Given a set of considered correspondences Λ, the precision of an alignment A e ⊆ Λ in comparison to a reference alignment A r ⊆ Λ is: With these measures the max-correctness, max-completeness, definiteness are verified. The semantic-identity property is also satisfied for each semantically equivalent alignments belonging to Λ (but not necessarily for the others). Still the overlappingpositiveness is not satisfied: A +/Λ e ∩ A +/Λ r = ∅ =⇒ Cn(A e ) ∩ Cn(A r ) = ∅ These measures are defined in the case of expressive alignments but they are dependent of Λ and consequently the precision and recall value are not absolute. Hence, these measures are useful for comparing a finite set of systems, but do not provide an absolute measure of precision and recall with regard to a reference alignment.

P Λ (A e , A r

Conclusion

In this paper, we presented and analyzed several ontology alignment evaluation propositions. Actually, no concrete evaluation measure respects the semantic-equality and the overlapping-positiveness properties that an ideal semantic model should satisfy. More precisely, the semantic precision and recall measures cannot respect the semanticequality due to the facts they still depend on the syntactic representation of alignments. To overcome these limitations, we first introduced alignment normalization principles which partially resolve the problem of semantic-equality. Then, we also proposed two new sets of evaluation measures. The first set of measures is built upon the generalized precision and recall framework and allows to locally consider the semantics of alignments. These measures can be seen as semantic-relaxed precision and recall. The second set of measures is proposed from an adaptation of ideal semantic measures. This adaptation makes the ideal semantic measures useable but in counterpart they do not verify the overlapping-positiveness property any more.

Definition 2 (

 2 Model). a model m = I, D of o is a function I from the terms of o to a domain of interpretation D, which satisfies all the assertions in o: ∀δ ∈ o, m |= δ The set of models of an ontology o is denoted as M(o). Because the models of various ontologies can have different interpretation domains, we use the notion of an equalising function, which helps make these domains commensurate. Definition 3 (Equilising function). Given a family of interpretations I o , D o o∈Ω of a set of ontologies Ω, an equalising function for I o , D o o∈Ω is a family of functions γ = (γ o : D o -→ U ) o∈Ω from the ontology domains of interpretation to a global domain of interpretation U . The set of all equalising functions is called Γ . The relations used in correspondences do not necessarily belong to the ontology languages. As a consequence, a semantics for them must be provided. Definition 4 (Interpretation of alignment relations). Given R an alignment relation and U a global domain of interpretation, R is interpreted as a binary relation over U , i.e., R U ⊆ U × U . The definition of correspondence satisfiability relies on γ and the interpretation of relations. It requires that in the equalised models, the correspondences are satisfied. Definition 5 (Satisfied correspondence). A correspondence c = x, y, R is satisfied for an equalising function γ by two models m, m of o, o if and only if

  Definition 7 (α-Consequence of aligned ontologies). Given two ontologies o and o and an alignment A between these ontologies, a correspondence δ is a α-consequence of o, o and A (noted A |= δ) if and only if for all models m, m , γ of o, o and A, m, m |= γ δ (the set of α-consequences is noted by Cn(A)).

  relevant not relevant found |Cn(Ae) ∩ Cn(Ar)| |Cn(Ae) -Cn(Ar)| |Cn(Ae)| true-positives false-positives not found |Cn(Ar) -Cn(Ae)| |(E -Cn(Ar)) -Cn(Ae)| |E -Cn(Ae)| false-negatives true-negatives |cn(Ar)| |E -Cn(Ar)|

  bounded recall measure). Given a set of considered correspondences Λ, the recall of an alignment A e ⊆ Λ in comparison to a reference alignment A r ⊆ Λ is: R Λ (A e , A r ) = |A

Table 1 .

 1 Contingency of sets Ae and Ar.

	relevant	not relevant
	found |Ae ∩ Ar|	|Ae -Ar|	|Ae|
	true-positives false-positives
	not found |Ar -Ae| |(E -Ar) -Ae| |E -Ae|
	false-negatives true-negatives
	|Ar|	|E -Ar|

Table 2 .

 2 Contingencies of the ideal extension of the classical model.

http://oaei.ontologymatching.org