HOURYA SINACEUR ALFRED TARSKI: SEMANTIC SHIFT, HEURISTIC SHIFT IN METAMATHEMATICS

Metamathematics was created by Hilbert in a series of papers published between 1905 and 1931. Hilbert used the term 'metamathematics' for the first time in 1922 as synonymous with 'proof theory' (Beweistheorie). The idea, but not the term, of proof theory was already present in his 1905 paper and described in more detail in 1918. In the latter paper, Hilbert outlined a programme for opening a new branch of mathematical inquiry, namely the study of the concept of mathematical proof. He listed some problems in the "theory of [mathematical] knowledge" which, he believed, could be solved within the framework of this "new mathematics": first of all, the problem of the consistency of arithmetic, and also the questions of the solvability of every mathematical problem, of decidability by a finite number of steps, of checking results, of defining a criterion for testing the simplicity of mathematical proofs, and finally the question of determining the relations between content and formalism (Inhaltlichkeit und Formalismus)in mathematics and logic. 1

As

Hilbert explicitly required in the case of the consistency of elementary arithmetic, the solution must be a mathematical one.

2 Thus, metamathematics consists in the use of mathematical tools to study logical or epistemological questions concerning mathematical structures or methods. Hilbert had such a definition in mind and this definition still suited the work of Hilbert's followers, namely, Gödel, Tarski, Church, Kleene or Kreisel (among others). One needs only to add that the meaning of 'mathematical tools' may be so wide as to include logical ingredients, since logic itself became in the meantime, and thanks precisely to the work of the above mentioned logicians and some others, much more involved in other mathematical branches.

With his proof theory, Hilbert created one of the three fundamental trends of metamathematical research which, along with recursion theory and model theory, nowadays constitute the scope of mathematical logic. I shall not look at recursion theory here. Rather, I will examine how Tarski developed Hilbertian metamathematics in order to create the basis of a new domain of mathematical logic, i.e., model theory.

TARSKIAN METAMATHEMATICS

Tarski knew very well the results and methods of what he himself named the "Göttingen School" of logic. [START_REF] Tarski | Tarski[END_REF] As we can see by reading his early writings, he used the expression "methodology of deductive sciences" to refer to his own work, borrowing this expression from K. Ajdukiewicz and stressing that it is synonymous with the Hilbertian term "metamathematics". [START_REF] Tarski | For example[END_REF] On his part, he defined the methodology of deductive sciences, or "methodology of mathematics", as the study of "formalized deductive theories" such as Peano's arithmetic or Hilbert's axiomatic geometry. 5 However, he soon came to dislike the term 'methodology', the meaning of which did not fit his current ideas: 6 Tarski wanted neither to develop "the science of method" nor to apply logic to philosophy. He emphasized indeed that "the analysis and the critical evaluation of the methods, which are applied in the construction of deductive sciences ceased to be the exclusive, or even the main task of methodology". 7 Eager to apply mathematics to logic and conversely logic to mathematics, he adopted the Hilbertian term from 1935 onwards. 8 He stressed however that the expression 'proof theory' seemed to him not to be very good and in any case less appropriate than 'metamathematics'. [START_REF] Tarski | [END_REF] Moreover, he strove to construct for the latter a meaning which no longer coincided with the Hilbertian one. If Hilbert grosso modo identified metamathematics with proof theory, Tarski needed to widen the scope of metamathematics in order to make it fit his own ideas and work. For him, the issue at stake was not only mathematical proof, but the whole field consisting of logic and mathematics -mathematics as generally practised by the working mathematician. This is one reason why Tarski more particularly rejected the restriction to "finitistic methods", i.e., methods for solving mathematical problems in a finite number of steps and with no appeal to the so-called "actual infinite".

In fact, after Gödel's results on the incompleteness of any first-order theory incorporating number theory (1931), metamathematical investigations could no longer be centered on Hilbert's main problem as formulated between 1904 and 1930, namely the consistency of elementary arithmetic, which was thought of as the foundational problem in mathematics. Among other Hilbertian metamathematical problems, the decidability of the firstorder theory of quantification, viewed as the fundamental problem for symbolic logic, and the solvability of every mathematical question were the dominant concerns for logicians in the late 1920s. About both issues there were, however, strong misgivings. [START_REF]Gödel[END_REF] As for the problem of the relations between a formal system and its content, it was Tarski who gave a radically new outlook, developing an original back-and-forth method between a formal language ant its interpretations and obtaining very important results by means of this method. One can say that developing this method, -which was going to be specific to model theory -was Tarski's own way of dealing with the Hilbertian problem of exploring the relations between content and formalism.

How did Tarski come to his views? What did he aim for in his own metamathematical investigations?

A General Theory of Deductive Science

As early as 1930, Tarski observed that "strictly speaking, metamathematics is not to be regarded as a single theory. For the purpose of investigating each deductive discipline a special metadiscipline should be constructed". [START_REF] Tarski | Tarski[END_REF] This observation means that there is no universal metatheory for the whole field of mathematics. In Tarski's view, it is possible, however, to construct the domain of "general metamathematics", whose task is to define the meaning of a series of metamathematical concepts which are common to the special metadisciplines, and to establish the basic properties of these concepts. Among these concepts we find consistency, completeness, decidability and also satisfiability and truth of a formula, [START_REF]Defined in Tarski[END_REF] definability, [START_REF] Tarski | Tarski[END_REF] model, consequence, 14 theory, 15 etc. all of whose definitions the model theorists today mainly owe to Tarski. It was this precise definition of semantic concepts used informally by some great logicians who preceded him -such as Skolem or Gödel -, which constituted a crucial shift in metamathematics. Let me give one single example. Gödel used the term 'model' in the Introduction of his paper on 'The completeness of the calculus of logic' (1929). [START_REF] Gödel | [END_REF] As far as I know, this was the first occurrence of the term, but Gödel did not define it. In German the usual words were 'Realisierung', 'Erfüllbarkeit' or 'Erfüllung'. Hilbert, for instance, used to speak of the 'Erfüllbarkeit' or 'Erfüllung eines Axiomensystems'. Before 1935 he used the term 'model' only once, namely, in Hilbert and Bernays (1934, 18). On his part, Skolem generally used the periphrases "domain (Bereich), in which such proposition is satisfied (erfüllt)" or "set of elements in which such axioms are true". Skolem used the term 'modèle' (in French) with nearly the same meaning as the German "Erfüllbarkeit", in a single paper, read at the Entretiens de Zürich, in 1938. 17 At that time Tarski had already defined the concept of model in terms of the concept of satisfaction in his address at the International Congress of Scientific Philosophy in Paris (1936d). [START_REF] Tarski | [END_REF] He had stressed that the given definition coincides with the mathematical use of the word [START_REF]While mathematicians were familiar with the notion of a model for a long time, they did not use the term very early. For example, Beltrami, Klein and Poincaré used the term 'interpretation'. Hilbert (1899), who systematically resorted to building (algebraic) models for testing compatibility, consistency, and independence of different sets of geometrical axioms, used the periphrase "system von Dingen, in dem sämtliche Axiome [der Geometrie] erfüllt sind[END_REF] and he had stated the definition of logical consequence which has become classic:

The sentence X follows logically from the sentences of the class K iff every model of the class K is also a model of the sentence X.

In chapter VI, paragraph 37 of his Introduction to Logic and to the Methodology of Deductive Sciences (1936b), Tarski had thoroughly explained the concept of model through the very simple example of congruent segments; he had specified that an axiomatic system can have several models and the way in which a theory can have a model in another theory.

In 1944 Tarski himself praised the merits of his semantical method: it allowed him to adequately define concepts which had been used until then only in an intuitive way. [START_REF] Tarski | Tarski[END_REF] Thus, Tarski achieved a very specific semantic shift from a view which focused on formal systems, axioms and rules of proof, to a view focused:

(1) on the formal definition and the axiomatization of semantic concepts, and (2) on the interplay between sets of elementary sentences and their mathematical models. In the model-theoretic tradition, which goes back to J. Lambert 21 in mathematics and to Ch. S. Peirce and E. Schröder in logic, [START_REF] Vaught | [END_REF] the relation of a sentence (or a set of sentences) to its models is the main concern. Tarski's specific contribution consisted in exploring the mutual relations between sentences and their mathematical models, in showing that semantic analysis is furthered, and not superseded, by syntactic analysis and in weaving metamathematical and mathematical threads in a much tighter fashion than anyone had done before him.

In a footnote to the last sentence of paragraph 42 of Tarski (1936b), Tarski dated the beginning of his general metamathematics to around 1920, and attributed it equally to the Göttingen School (with Hilbert and Bernays) and to the Warsaw School (with Leśniewski and Lukasiewicz among others). The originality and significance of his personal work, which may be rightly characterized as axiomatic semantics, 23 soon became evident to logicians. For instance, Rudolph Carnap stated in his Introduction to Semantics (1942) that it was Tarski who "first called my attention to the fact that the formal method of syntax must be supplemented by semantical concepts that can be defined by means no less exact than those of syntax". 24 About forty years later Jon Barwise declared that 'Tarski's view of logic has changed the way all of us think about the subject". 25

Examples of Tarski's Way of Defining a Concept and Solving a Problem

Tarski fruitfully developed a specific kind of conceptual analysis. He combined the syntactic analysis inherited from Hilbert's School and the semantic methods of the algebraists of logic (Peirce, Schröder, Löwen-heim, and Skolem). From Leśniewski -his dissertation adviser whose direct influence he acknowledged [START_REF] Tarski | Tarski[END_REF] -Tarski learned that any formalized theory consists of meaningful sentences. [START_REF] Tarski Stressed It | on his own part[END_REF] Tarski's semantics did reorient syntactic analysis; it did not reject it. Thus, attention was focused on the interrelations between formal sentences and their possible meaning (i.e., their possible interpretations) in some mathematical domain, and the aim was to define semantic concepts as well as solve metamathematical problems mathematically. Let me give two examples of Tarski's conceptual analysis, which is at the same time synctactical and semantical.

1.2.1. In Tarski's view, no precise definition could be constructed in a completely abstract and general way. Tarski did not address such general questions as "What is a number?" -a major question for Frege -or "What is a set?" -a question which was much discussed since Dedekind's and Cantor's work on set theory -. In fact, Tarski did not expect a uniform and unambiguous answer to these kinds of questions. He thought, rather, that the answer to a question or, at least, to one of its aspects, depended on the presupposed framework and tools used to formulate precisely that question or its selected aspect. Even when he came to address the question 'What are logical notions?' in a lecture given in 1966 and published twenty years later, he stressed that he was not aiming to discuss "the general question 'What is logic?' " but wanted instead to restrict his comments to "one aspect of the problem, the problem of logical notions". Moreover, he pointed out how much his approach distances itself from attempts to catch "the proper, true meaning of a notion, something independent of actual usage, and independent of any normative proposals, something like the platonic idea behind the notion". On his part, he would "make a suggestion or proposal about a possible use of the term 'logical notion' ". 28 Thus, Tarski constructed an algebraic definition for this term: a notion is logical if it is invariant under all possible one-one transformations of the universe of discourse onto itself. Then, he concluded: "The suggestion I have made does not, by itself, imply any answer to the question of whether mathematical notions are logical". According to Tarski, the latter question has a positive answer if one takes a higher-order underlying logic (e.g., as in Principia mathematica of Russell and Whitehead), and a negative answer if one takes a first-order underlying logic. Thus, the logicist philosophy of Frege and Russell was in fact closely tied to the logical framework they had adopted. Another framework, another philosophy! 1.2.2. One finds another important example of Tarski's specific kind of conceptual analysis in his treatment of the concept of definability. In his seminal paper of 1931, he declared that "the meaning of this notion is not at all precise: a given object may or may not be definable with respect to the deductive system in which it is studied, the rules of definition which are adopted, and the terms that are taken as primitive". [START_REF] Tarski | [END_REF] Then, he determined the domain within which he would make precise the meaning of the notion, i.e., the domain of interpretation of the notion. This domain is one of those most familiar to mathematicians, namely, real numbers. Thus, Tarski did not address (in this paper) the general question of definability, but the restricted question of what are definable sets of real numbers. The construction of the meaning of the term 'definable set of real numbers' is carried out in two steps. First -in the metamathematical step -, a set of real numbers is said to be 'definable' if it is determined by a first-order sentential function (first-order formula), [START_REF]Tarski borrowed the expression 'sentential function' from the Principia Mathematica of Russell and Whitehead. 31 Algebraic is a set whose elements are common solutions of a finite number of polynomial equations[END_REF] i.e., if it is defined by what logicians usually call an "explicit definition". In a second step, Tarski showed how to reconstruct this definition, within mathematics itself, by using instead of the metamathematical notion of formulas (of a first-order language), its mathematical analogue, namely, the concept of sets of sequences, and by defining on these sets operations corresponding to the logical operations adopted as primitive (negation, conjunction, disjunction, universal and existential quantification). Going further, Tarski proposed to use the concept of a polynomial, defined as a function associating a certain real number P (x 1 , x 2 , . . . , x n ), to every sequence x 1 , x 2 , . . . , x n of real numbers. Thus, a set E of real numbers is definable iff there is a polynomial P such that E is exactly the set of finite sequences x 1 , x 2 , . . . , x n satisfying the equation P (x 1 , x 2 , . . . , x n ) = 0 or the inequality P (x 1 , x 2 , . . . , x n ) > 0. In other terms:

E = {(x 1 ,x 2 ,. . . ,x n ) ∈ R n , P (x 1 ,x 2 ,. . . ,x n ) = 0 ∨ P (x 1 ,x 2 ,. . . ,x n ) > 0}.
Thus, Tarski stated for the ordered field of real numbers a notion analogous to that of algebraic set, 31 which was well-known for the field C of complex numbers and, more generally, for any commutative field. Tarski's statement is the very first definition of what modern mathematicians call real semi-algebraic sets, as I noted elsewhere. 32 Here, however, I only want to stress that the defining process is somewhat analogous to that which is used for the above-mentioned definition of the term 'logical notion'. Indeed, whenever it could be done, Tarski gave mathematical definitions, and preferably algebraic ones, for metamathematical concepts. He used likewise mathematical, and preferably algebraic tools, for solving metamathematical problems. For instance, he generalized Sturm's algorithm for counting the real roots of polynomials, by conceiving it as a decision procedure for the elementary theory of real numbers: not only the number of roots, but all questions formulated in the first-order language of the ordered field of real numbers are decided by applying Sturm's process in a generalized form. This result is a brilliant product of Tarski's conceptual analysis, which closely tied the study of syntactic properties of the axioms for the theory to the study of the structural properties of its model or models. 33 Tarski's ingenious trick was to understand Sturm's process as a mathematical instance of the logical method of effective elimination of quantifiers (EEQ). The older decision procedures by EEQ (Löwenheim, Skolem, Langford, Presburger, Herbrand) were also guided by the analogy with the successive elimination of unknowns in systems of polynomial equations, but they did not use any theorem borrowed from the theory of algebraic elimination. All the arguments of, say, Herbrand's proof for the decidability of the elementary theory of addition on integers, were of a purely logical nature. While Herbrand, among others, tried to mimic the algebraic elimination process, Tarski showed that the latter is a special case of effective elimination quantifier. 34 He noticed that from the point of view of its logical structure his proof was close to the older completeness proofs by quantifier elimination, but that it was singular in using "a much more powerful instrument". 35 Tarski's decision method for real algebra and geometry was indeed the first one being of a mathematical nature. In a historical perspective, one can say that Tarski reversed the reduction of mathematics to logic, using mathematical methods for defining logical concepts and proving logical theorems. Developing this "reverse logic" was nothing else, to some extent, than carrying out Hilbert's aim to establish a "new mathematics". But Tarski went much further, willing not only to construct a new logic with the help of mathematics, but also to give an evidence for the mathematical efficiency of model-theoretical reasoning. He restored logic to power, but on a technical as well asand perhaps more than -on a foundational level. He strove to extend our possibilities of mathematical reasoning.

A HEURISTIC SHIFT

Solomon Feferman, who studied with Tarski at Berkeley from 1948 to 1957, confirmed that Tarski did have "a very strong motivation" not only to make logic mathematical, but also "and at the same time to make it of interest to mathematicians". 36 In fact, from the beginning of his career in Poland, Tarski expressed again and again his will to make metamathematics fully mathematical. In 1936 he argued that "The methodology of the deductive sciences became a general theory of deductive sciences, in a sense analogous to that, in which arithmetic is the theory of numbers and geometry is the theory of geometrical figures". 37 Metamathematics, which more particularly includes metatheories for arithmetic and geometry, became itself a science comparable in every respect to arithmetics or geometry, the oldest mathematical sciences. Indeed, since metamathematical concepts can be reconstructed by mathematical means, "they do not differ at all from other mathematical notions . . . their study remains entirely within the domain of normal mathematical reasoning". 38 This explains how metamathematics became able to borrow from the rest of mathematics results and methods, which it could then apply within its proper perspective, thus yielding results that might in turn be fruitful for what Tarski called "ordinary" or "normal" mathematics. This is indeed the case for the definition of definable sets of real numbers -which was a by-product of the decision method for real algebra and geometry -, even though "ordinary" mathematicians did not recognize the mathematical fruitfulness of this definition before the late 1970s and did not create real algebraic geometry starting from Tarski's work, but rather from Łojasiewicz's semi-analytic sets. 39 In one word, the logician Tarski opened a new mathematical branch, the mere possibility of which no mathematician could even conceive of in the thirties.

Hilbert made a strong separation between mathematics and metamathematics, although he proposed to use mathematical tools to deal with metamathematical problems. But solving the latter was, according to him, eliminating them once and for all. In such a perspective, Hilbert could not conceive of the fruitfulness of metamathematics for mathematics. In Hilbert's view, metamathematics was to yield a guarantee (Sicherung) for mathematics. 40 Tarski thought that this aim to provide for mathematicians "a feeling of absolute security" was "far beyond the reach of any normal human science"; it pertained to "a kind of theology". 41 Tarski wanted to transform theology into mathematics. In his view, metamathematics became similar to any mathematical discipline. Not only its concepts and results can be mathematized, but they actually can be integrated into mathematics. Hence they can help solve mathematical problems or throw new light on older mathematical fields or notions, such as real numbers, orderings, Sturm's algorithm, polynomials, etc. -and they do so independently of what philosophical thoughts they may involve or suggest.

Tarski destroyed the borderline between metamathematics and mathematics. He objected to restricting the role of metamathematics to the foundations of mathematics. He also strove to give metamathematics a heuristic role. He did not want to put logic to service as a tool of conceptual analysis; he wanted rather to develop conceptual analysis as a tool of mathematical research. In the 1930s this was the beginning of a watershed in modern mathematical logic, which would become conspicuous from the 1950s onward. While the semantic shift in logic has been and still is the subject matter of many papers and studies, [START_REF]Semantic revolution" and the history of the word 'semantics' may be found in Wolenski[END_REF] it seems to me that historians of logic more rarely paid attention to the heuristic shift, even though outstanding logicians, such as E. Beth, A. Robinson and G. Kreisel, pointed it out. [START_REF][END_REF] Now, it is Tarski who initiated this shift and, as noted by Kreisel (1985), "the passage from the foundational aims for which various branches of modern logic were originally developed to the discovery of areas and problems for which logical methods are effective tools . . . did not consist of successive refinements, a gradual evolution by adaptation . . . , but required radical changes of direction, to be compared to evolution by migration".

Philosophical Attitude and Actual Doing

Tarski thought that the foundational perspective was mostly philosophical, i.e., to some extent foreign to the technical work of logicians. Thus he was led to a pragmatic point of view, stressing that his methods and results were independent of philosophical assumptions concerning the foundations of mathematics. [START_REF] Tarski | Tarski[END_REF] Carrying Hilbert's guideline further, Tarski sharpened the distinction between logic and philosophy, or, more accurately, between mathematical logic and philosophical logic. But unlike Hilbert, he abolished the distinction between mathematics and metamathematics, for he thought it irrelevant from a practical point of view. [START_REF] Tarski | Tarski[END_REF] Now, the practical point of view was dominant in Tarski's actual way of doing logic.

Truly, Tarski oscillated between different stands, explicitly or implicitly taken, on the nature of mathematical and logical knowledge and between different opinions on his own activity. In his paper on 'The completeness of elementary algebra and geometry ' (1939/1967), he noted that in order to determine whether or not a classical geometrical theorem belongs to his elementary system "it is only the nature of the concepts, not the character of the means of proof that matters". [START_REF] Tarski | Tarski[END_REF] What Tarski pointed out here is that a first-order theory may encompass concepts expressible and provable under non-elementary conditions which are satisfied in some particular model of the theory. That is to say that a first-order theory may grasp much more than first-order definable properties. From a logical (technical) point of view, this fact is by itself significant. Now, Tarski's insistence on a mathematical content independent of its possible formulation or proof might have led to a kind of realism. Although Tarski did not explicitly take such an attitude, it is compatible with that which is expressed in the final remarks of Tarski (1944). There, Tarski defended the intrinsic interest of metamathematical research, 47 arguing that it may be harmful to scientific progress to equate the importance of a work with its possible usefulness. He added, however, that this opinion did not affect the subject matter of his paper. The subject matter is also relatively independent of the formal language in the framework of which it is developed and totally unaffected by any informal philosophy associated with it.

Later on, Tarski changed his mind. Instead of stressing the independence of technical results from philosophical assumptions, he adopted the philosophical outlook fitting his actual practice. In particular, much as an "ordinary" mathematician, he accepted to raise the question of applicability and to show, for example, that his theory of arithmetical classes "has good chances to pass the test of applicability . . . [and to] be of general interest to mathematicians". [START_REF] Tarski | Tarski[END_REF] At the same time, he substituted a more pragmatic philosophy to his former "intuitionistic formalism" (borrowed from Leśniewski). [START_REF] Tarski | Tarski[END_REF] In the above-mentioned lecture, delivered in 1966 and published twenty years later, Tarski related the way in which the Platonistic approach was "so foreign and strange to [him]". [START_REF]This is supported by another passage, in which Tarski described himself an an "extreme anti-Platonist[END_REF] In Tarski's words, the Platonistic approach consisted in trying to catch "the proper, true meaning of a notion, something independent of actual usage, and independent of any normative proposals". To some extent this view of Platonism was shared by Leśniewski. [START_REF] Wolenski | [END_REF] But Tarski no longer agreed with Leśnewski's "intuitionistic formalism", proposing instead the attitude he described himself as follows:

What I shall do is make a suggestion or proposal about a possible use of the term 'logical notion'. This suggestion seems to me to be in agreement, if not with all prevailing usage of the term 'logical notion', at least with one usage which actually is encountered in practice. I think the term is used in several different senses and that my suggestion gives an account of one of them. 52 I think this declaration is very important for anyone who wishes to give a positive content to Tarski's negative statement that Lesniewski's philosophy did not "adequately reflect his present attitude" (footnote added in 1956 to Tarski (1930b). 53 ) One may already observe the presence of such a pragmatic attitude in Tarski's lecture at the Princeton University Bicentennial Conference on the Problems of Mathematics, back in 1946. [START_REF] Tarski | Jan Tarski found the draft of this talk in 1994 in his father's Nachlass in the Bancroft Library at Berkeley. This draft is to be published[END_REF] There, Tarski said that he would "use the term 'logic' pragmatically to denote the work of people who regard themselves as logicians -or those who are considered logicians by mathematicians generally". Now, a pragmatic attitude is, it seems to me, the most appropriate one to hold in defending the practical mathematical efficiency of metamathematics within mathematics itself.

A Cross-Fertilization Between Mathematics and Metamathematics

If methamathematics is itself a mathematical discipline in the fullest sense of the word 'mathematical', then, as Tarski stressed, "the results obtained in one deductive discipline can be automatically extended to any other discipline in which the given one finds an interpretation". 55 The process of interpretation -the formal definition of which would come in 1953 [START_REF] Tarski | Finding an interpretation of a theory T 1 in a theory T 2 is tantamount to constructing in T 2 a model of T 1[END_REF] -yields a fruitful co-operation between metamathematics and other branches of mathematics. The back-and-forth method is to be applied not only to sentences and their models, but also to different theories. The interplay between different areas of mathematics, metamathematics included, brought new mathematical cross-concepts and problems, along with novel hybrid tools to tackle them. Gödel made a significant step towards this cross-fertilization with his coding of metamathematical sentences by natural numbers. Tarski, who often stressed that he independently discovered the possibility of interpreting metalogical sentences about a structure S in the structure S itself, 57 systematized the use of interpreting one theory in terms of another 58 and was very eager to construct tools for "ordinary" mathematicians from metamathematical results. Well-known successes of the process of interpretation may be found: (1) in undecidability proofs, and (2) in constructing algebraic translations of metamathematical notions and theories. As an example of (1) let us mention Julia Robinson's proof of the undecidability of the elementary theory of rational numbers, which was tantamount to showing the definability of integers within the field of rationals. As an example of (2) one should be reminded of Tarski's aim at specifying the significance of first-order formulas by describing the closure properties of classes of models defined by them and the subsequent use of ultrapowers -which are of algebraic nature -by J. H. Keisler (in 1964) to describe classes of models which can be defined by a first-order formula. Last, but not least, let us mention another important example: Tarski developed in collaboration with L. Henkin and J. D. Monk cylindric algebras, which are an algebraic version of predicate logic.

CONCLUSION

One cannot say that advances in metamathematics revolutionized the whole field of "pure" mathematics, as Gödel believed it should do. [START_REF] According | [END_REF] However, Tarski's struggle to make metamathematics of interest to mathematicians succeeded, even though not immediately. Actually, new mathematical branches came out, which combine metamathematical and mathematical methods, such as model-theoretic algebra, model-theoretic analysis, real algebraic geometry, computer algebra, etc. These branches use many methods moulded by Tarski, and first of all tools deriving from Tarski's analysis of the notion of definability. Indeed, it may be the case that it is easier to state mathematical properties of some objects by analyzing the syntactical formula through which they can be defined rather than deducing these properties by purely mathematical means. This is the case, for instance, of proving that the projection of a semi-algebraic set is still semi-algebraic. 60 If metamathematics did not revolutionize the whole field of "pure" mathematics, there is nevertheless no doubt that much more of it has become an integral part of standard mathematics than "pure" mathematicians expected.

ACKNOWLEDGEMENTS

I wish to thank Elie Zaher, Liliane Beaulieu and Mohammed Melouk for reading over my paper and improving my English. NOTES


H Gailhanou
email: h.gailhanou@brgm.fr

S Gaboreau

M Debure

S Grangeon

P Blanc

N Marty

J M Grenèche

B Madé

S Altmann

C Tournassat

Geochemistryof Callovo-oxfordian argillite at 80°C:batch experimentsand modeling

Introduction

Arepository of high-level nuclear wastes in the Callovo-Oxfordian(COx) clayrock will lead to a temporary increase in temperature locally around the disposal cells, up to about 80°C. Thermally induced modifications of the pore-water chemistry of the clayrock must be assessedsince the pore water chemistry can affect the rates of corrosion of metallic materials used in the disposal facility [START_REF] Gaucher | A robust model for pore-water chemistry of clayrock: Geochim[END_REF][START_REF] Altmann | Geo'chemical research: a key building block for nuclear waste disposal safety cases[END_REF].In the present study, carried out as part of a collaborative BRGM/Andra research program, the results of long-term batch experimentsperformed at 80°C,are compared with compositions predicted using a model based on the one developed for 25°C.These experiments include a previous study [START_REF] Gailhanou | Influence of temperature on the geochemistry of Callovo-oxfordianargillite: batch experiments and modelling at 80[END_REF] that is completed by recent long-term batch experiments in order to verify the reproductibility of results and better understand the reactivity of the COx rock-solution system at 80°C. In this aspect, selectivity coefficients for exchange reactions Na/K, Na/Ca, Na/Mg and Na/Srin the clay-porewatersystem have been acquired at 80°C and have been considered to model the experiments.

Batch experiments

A first series of two experiments were performed with COx rock samples in contact with synthetic water, at 80°C, for durations at 6 and 20 months [START_REF] Gailhanou | Influence of temperature on the geochemistry of Callovo-oxfordianargillite: batch experiments and modelling at 80[END_REF]. Few changes were detected in the mineralogy between initial and final solids. The only noticeable change was the formation of needles of nanocrystalline goethite on pyrite particles in the 20 months experiment [START_REF] Gailhanou | Influence of temperature on the geochemistry of Callovo-oxfordianargillite: batch experiments and modelling at 80[END_REF]. A new series of experiments is being conducted to follow the reactivity of the clayrocksolution-gaz system at 80°C for durations from 20 months to 10 years. The results at 20 months are presented here and are compared with the previous ones. In addition, a last experiment was carried out with an initial CO 2 (g) partial pressure in the gaseous phase fixed at a higher value than the value predicted by modeling approach (log p CO2 = -0.6 instead of -1.1). This new 20 months experiment aims at verifying that the carbonate system re-equilibrates or not with the whole system independently of the initial pH. All these experiments were prepared at 25°C under anoxic atmosphere in closed reactors, with solution compositions close to the predicted pore water composition [START_REF] Gaucher | A robust model for pore-water chemistry of clayrock: Geochim[END_REF]. Afterwards, each system was heated at 80°C. p CO2 was monitored during the experiments. At the end of the experiments, the suspension was filtered at 0.1 µm at 80°C, under anoxic atmosphere. The pH was measured at 80°C; major cations (Ca, Mg, K, Na, Sr) and anions (notably Cl, SO4), TIC and minor and trace elements (notably Si, Al, Fe, Zn, Pb) were analysed.

In addition, selectivity coefficients were obtainedfor Na/K, Na/Ca, Na/Mg and Na/Srcationexchange at 80°C on the clay fraction (< 2 µm).

Modelling of batch experiments

The predictive model of composition as a function of temperature was developed based on the model from [START_REF] Gaucher | A robust model for pore-water chemistry of clayrock: Geochim[END_REF] and using the PHREEQC code and Thermoddem database [START_REF] Blanc | Thermoddem: a geochemical database focused on low temperature water/rock interaction and waste materials[END_REF].Modelling prediction were in fairly good agreement with the results of the first series of experiments, at 6 and 20 months, if the following primary mineral assemblage was considered: quartz, pyrite, carbonates (calcite and dolomite), sulfates (celestite), aluminosilicates (chlorite and illite) and goethite(Fig. 1). This model differs from the model of [START_REF] Gaucher | A robust model for pore-water chemistry of clayrock: Geochim[END_REF] at 25°C by the replacement of siderite by goethite (Fig. 1). This modification led to a noticeable improvement of the Fe concentration prediction. The introduction of goethite in the model instead of siderite, is in agreement with the presence of nanogoethite detected in the initial sample and in the solid after 20 months experiment.

Fig 1 .

 1 Fig 1. Results of the first series of experiments. Comparison between experimental and modeled concentrations (a) at 6 months at 80°C and (b) at 20 months at 80°C.


Astrid Duque-Ramos
email: astrid.duque@um.es

Jesualdo Tomás Fernández-Breis

Miguela Iniesta
email: miniesta@um.es

Michel Dumontier
email: michel_dumontier@carleton.ca

Mikel Egaña Aranguren

Stefan Schulz
email: stefan.schulz@medunigraz.at

Nathalie Aussenac-Gilles

Robert Stevens
email: robert.stevens@manchester.ac.uk

Evaluation of the OQuaRE framework for ontology quality

Keywords:

The increasing importance of ontologies has resulted in the development of a large number of ontologies in both coordinated and non-coordinated efforts. The number and complexity of such ontologies make hard to ontology and tool developers to select which ontologies to use and reuse. So far, there are no mechanism for making such decisions in an informed manner. Consequently, methods for evaluating ontology quality are required. OQuaRE is a method for ontology quality evaluation which adapts the SQuaRE standard for software product quality to ontologies. OQuaRE has been applied to identify the strengths and weaknesses of different ontologies but, so far, this framework has not been evaluated itself. Therefore, in this paper we present the evaluation of OQuaRE, performed by an international panel of experts in ontology engineering. The results include the positive and negative aspects of the current version of OQuaRE, the completeness and utility of the quality metrics included in OQuaRE and the comparison between the results of the manual evaluations done by the experts and the ones obtained by a software implementation of OQuaRE.

Introduction

The increasing importance of ontologies has resulted in the development of a large number of ontologies in both coordinated and non-coordinated efforts. Ontologies for domains very different like life sciences, e-commerce, city logistics or geospatial data can be found in [START_REF] Ashburner | Gene ontology: tool for the unification of biology[END_REF], [START_REF] Beisswanger | Biotop: an upper domain ontology for the life sciencesa description of its current structure, contents and interfaces to obo ontologies[END_REF], [START_REF] Hepp | Goodrelations: an ontology for describing products and services offers on the web[END_REF], [START_REF] Anand | Genclon: an ontology for city logistics[END_REF], or [START_REF] Tian | Enhance discovery and retrieval of geospatial data using soa and semantic web technologies[END_REF]. The National Center for Biomedical Ontology (NCBO) BioPortal (http://bioportal. bioontology.org/) has more than two hundred biomedical ontologies and controlled vocabularies, the TONES repository (http:// owl.cs.manchester.ac.uk/repository/) contains more than two hundred, and tools like Watson (http://kmi-web05.open.ac.uk/ WatsonWUI/) or Swoogle (http://swoogle.umbc.edu/) give access to thousands.

Many ontology builders and application developers have usually several options for which ontology to use to support a particular intelligent software application or to reuse for building a new ontology. Existing ontologies could fit for such purposes with little (or even no) additional development if they have the appropriate content. We regard the ideal scenario as the one in which users are able to select ontology artefacts from a large repository, however, currently not the state of play. Nowadays, ontology builders lack support for making an informed decision, because standardized methods have not been developed for evaluating the quality of ontologies.

In this work, the quality of an ontology is its degree of conformance to functional and non-functional requirements and we assume that such conformance can be measurable. Current work in ontology evaluation can be classified according to the particular evaluation aim: ranking, correctness, or quality evaluation. The ranking approaches range from generic ontology rankings to the selection of the most appropriate ontology for a particular task (see, for instance [START_REF] Alani | Ranking ontologies with aktiverank[END_REF][START_REF] Lozano-Tello | Ontometric: a method to choose the appropriate ontology. Ontological analysis, evaluation, and engineering of business systems analysis methods[END_REF][START_REF] Tartir | Ontology evaluation and ranking using ontoqa[END_REF]. The correctness category includes the approaches accounting for the formal correctness of the content represented in ontologies (see, for instance, [START_REF] Corcho | ODEVAL: a tool for evaluating RDF(S), DAML+OIL and OWL concept taxonomies[END_REF]Guarino & Welty, 2004, chapter 8;[START_REF] Sleeman | Cleanonto: evaluating taxonomic relationships in ontologies[END_REF][START_REF] Vrandecic | Ontology evaluation[END_REF]. Finally, the quality category is related to the evaluation of ontology quality (see for instance [START_REF] Gangemi | Modelling ontology evaluation and validation[END_REF][START_REF] Rogers | Quality assurance of medical ontologies[END_REF][START_REF] Sabou | Ontology selection: ontology evaluation on the real semantic web[END_REF][START_REF] Stvilia | A model for ontology quality evaluation[END_REF]. While each has been addressed from different, heterogeneous ways, none has become standard. The need for standardized methods for evaluating the quality of an ontology remains unfulfilled.

In this work, we also approach ontology evaluation as a tool for helping developers to evaluate their ontologies in order to build trust for sharing and reusing ontologies, which is one of the main objectives of ontology evaluation according to [START_REF] Brank | A survey of ontology evaluation techniques[END_REF]. For this purpose, our approach is in line with the effort done by the approaches of the quality category, although we propose the adaptation of an already existing standard from the Software Engineering community. In Fernández-Breis, Egaña Aranguren, and [START_REF] Fernández-Breis | A quality evaluation framework for bio-ontologies[END_REF], we presented an approach based on ISO 9126, which is an international standard for software product evaluation, which was applied to two versions of the same biomedical ontology. This evaluation process had the problem of the excessive workload and dependence on human judgment, since the method did not provide computer-support to the evaluator. More recently, the ISO/IEC 25000:2005, which is a standard for Software product Quality Requirements and Evaluation known as SQuaRE ISO (2005), was used by us for evaluating the same two biomedical ontologies, but in an automatic manner [START_REF] Duque-Ramos | Towards an square-based quality evaluation framework for ontologies[END_REF]. The results obtained in that work showed the benefits of using an automatic framework to support ontology evaluation processes, as presented in Duque-Ramos, Fernández-Breis, [START_REF] Duque-Ramos | Oquare: a SQuaRE-based approach for evaluating the quality of ontologies[END_REF].

The usage of a SQuaRE-based approach required the definition of components like quality model and quality metrics. A SQuaREbased quality model is comprised of a series of quality characteristics like functional adequacy or reliability. Each characteristic has a series of subcharacteristics associated. For instance, reusability can be a subcharacteristic of maintainability. Furthermore, each quality subcharacteristic has a series of metrics associated. An example of quality metrics for ontologies could be the mean number of properties per class. Thus, in order to design OQuaRE following such principles, a few decisions had to be taken, like the sets of characteristics, subcharacteristics and metrics, and the association of metrics with subcharacteristics. Scores in SQuaRE are provided in the range 1 (worst) and 5 (best), and the communities that apply SQuaRE must define how the scores of the metrics are mapped onto such range and how such scores are combined to provide scores for the subcharacteristics and characteristics. In our previous works we mentioned that such decisions should be the result of some community agreement: consequently, the objective of this work will be to evaluate such decisions and provide new insights about how ontology quality evaluation should be performed. It is clear that if the community wants ontology quality evaluation to be a engineering activity, there must be a standardization on how such evaluation processes must be performed.

For such purpose, we will describe in this paper the evaluation experiment for assessing both the quality model and quality metrics used in OQuaRE. The experiment will be based on the informed judgment of experts in ontology construction. The experiment will be divided in two rounds: the first one will evaluate the quality model and the second one will evaluate the quality metrics. In both rounds the experts will be asked to apply the evaluation method in a different manner, in order for the authors to acquire a thorough understanding of the different OQuaRE components.

The OQuaRE framework and the evaluation method used in this work will be described in the following section. In Section 3, the results of the OQuaRE evaluation will be presented. Results will be discussed in Section 4, which will also include the conclusions of this work.

Methods

This section will describe the main instruments used in this evaluation exercise. We will address the main aspects of OQuaRE and how it has been applied so far (Section 2.1) and the methodological approach for carrying out the evaluation process (Section 2.2).

OQuaRE

OQuaRE is the Ontology quality evaluation (OQuaRE) framework based on the software product quality SQuaRE. OQuaRE aims at defining all the elements required for ontology evaluation: evaluation support, evaluation process and metrics. However, the current version of OQuaRE includes the quality model and the quality metrics. This means that parts like evaluation requirements and evaluation reports, to date, have not been addressed in depth.

The main objective of OQuaRE is to provide an objective, standardized framework for ontology quality evaluation, which could be applied in a number of situations in a similar way, so the strengths and weaknesses of ontologies can be identified. OQuaRE reuses and adapts the following SQuaRE characteristics to evaluate ontologies: reliability, operability, maintainability, compatibility, transferability and functional adequacy. Most quality subcharacteristics suggested by SQuaRE were also adapted to ontologies in OQuaRE. However, SQuaRE does not include the structural characteristic, which is important in ontologies, as it can be drawn from the number of state-of-the art approaches that use it. Therefore, we included it into OQuaRE. The complete description of the quality model can be found at http://miuras.inf.um.es/evaluation/oquare. Next, we describe the OQuaRE quality characteristics and subcharacteristics.

The quality characteristics

We describe as follows the quality characteristics of OQuaRE. The definition of some subcharacteristics can be found in Table 1.

Structural: Formal and semantic properties that are important when evaluating ontologies. Some subcharacteristics are formalization, formal relations support, cohesion, tangledness, redundancy and consistency. These subcharacteristics account for the formal properties of the ontologies, the clarity of cognitive distinctions, the appropriate use of ontology modelling primitives and principles, etc.

Functional adequacy: The capability of ontologies to provide concrete functions that have been identified in literature. An ontology is evaluated for this criterion according to the degree of accomplishment of functional requirements, that is, the appropriateness for its intended purpose according to state-of-theart literature [START_REF] Stevens | Applications of ontologies in bioinformatics[END_REF]: reference ontology, controlled vocabulary, schema and value reconciliation, consistent search and query, knowledge acquisition, clustering and similarity, indexing and linking, results representation, classifying instances, text analysis, guidance and decision trees, knowledge reuse, inferencing, and precision.

Reliability: Capability of ontologies to maintain their level of performance under stated conditions for a given period of time. Recoverability and availability are some of its subcharacteristics.

Performance efficiency: Relationship between the level of performance and the amount of resources used, under stated conditions, taking into account elements such as the time response, or memory consumption. It has subcharacteristics like response time and resource utilization.

Operability: Effort needed for use, and in the individual assessment of such use, by a stated or implied set of users, and it is measured through subcharacteristics such as learnability.

Compatibility: The ability of two or more ontologies to exchange information and/or to perform their required functions while sharing the same hardware or software environment. Replaceability and interoperability are examples of subcharacteristics according to SQuaRE.

Maintainability: The capability of ontologies to be modified for changes in environments, in requirements or in functional specifications. Some subcharacteristics are modularity, reusability, analysability, changeability, modification stability and testability.

Transferability: degree to which the ontology can be transferred from one environment (e.g., operating system) to another. Portability and adaptability are example of its subcharacteristics.

Quality in use: Degree to which an ontology used by specific users meets their needs to achieve specific goals. It has subcharacteristics associated with usability and flexibility in use.

The quality metrics

OQuare does not attempt to develop new metrics for ontology quality evaluation but to reuse and adapt successful metrics from both ontology and software engineering communities. In this sense, some well-known metrics from Objected Oriented Programming have been included in its adapted form to ontologies (see, for instance [START_REF] Chidamber | A metric suite for object oriented design[END_REF][START_REF] Li | Object-oriented metrics that predict maintainability[END_REF]. In addition to this, metrics for the structural properties like the ones presented in [START_REF] Yao | Cohesion metrics for ontology design and application[END_REF] and [START_REF] Tartir | Ontology evaluation and ranking using ontoqa[END_REF] have been reused from the ontology community. Table 2 contains the definition of these metrics for OWL ontologies.

Experiences with OQuaRE

OQuaRE has been applied in two case studies in the last years, and the main results of such can be found at http://miuras.inf.um.es/evaluation/oquare. It was used for the evaluation of two versions of the Cell Type Ontology (see [START_REF] Bard | An ontology for cell types[END_REF]. A series of ontologies of units of measurement were evaluated. In both cases, the ontologies were firstly manually evaluated, and then automatically evaluated with a home-made software tool developed to run the evaluation experiments. Such tool takes an OWL ontology as input and generates a numeric report as output. This report includes the scores for all the characteristics, subcharacteristics and metrics involved in the evaluation process. The analysis of such results was done in two ways. First, the quality of the ontologies was evaluated and a series of strengths and weaknesses were identified. Second, we identified that doing a manual evaluation was perceived by the participants as very hard and that the results without human intervention did not seem too different. However, there might be several reasons for this, since an evaluation of the appropriateness of the OQuaRE configuration that leads to such results had not been externally evaluated. By OQuaRe configuration we mean the quality characteristics, quality subcharacteristics, quality metrics, which subcharacteristics contribute and how much to a characteristic, which metrics contribute and how much to a subcharacteristic, and how score are scaled in the [1, 5] range. Those are the main issues under investigation in this paper.

The evaluation method

In this section we describe the experimental approach used for evaluating OQuaRE, which follows the Goal/Question/Metric method van Solingen and Egon (1999).

Objectives

The research objectives of this experiment are to evaluate the appropriateness, completeness and usability of the quality model and quality metrics of OQuaRE. We also investigate the feasibility of automatic evaluation processes, through the comparison of manually and automatically calculated results. To achieve such objectives the following goals were defined:

Appropriateness of the quality subcharacteristics. Difficulty in applying the quality subcharacteristics. Relevance of metrics for measuring a particular subcharacteristic.

Usefulness of metrics for measuring a particular subcharacteristic.

Similarity between the results of the fully manual, metrics-supported manual, and fully automatic evaluation methods.

Participants

The set of participants were selected by applying the following criteria: all the participants should be experts in ontology construction, should be active and interested in ontology quality, should be from different research institutions and from different countries, should represent different ''ontology modelling schools'', and should not have played an active role in the design of OQuaRE. An additional constraint was imposed by the case study: given that the ontology under evaluation in the experiment was biomedical, background and expertise in biomedical ontologies was required. Three experts were selected for this experiment.

Experimental material

In this section we describe the ontology, documents and tools used in the experiment: Ontology: The existence of an ontology of units was considered important by the W3C Semantic Web Best Practices and Development working group (http://www.w3.org/2003/12/swa/ swbpd-charter). Since then, a series of ontologies of units of measurement have been developed. Ontologies of units are relevant not only for biomedical domains, but also in any other science and engineering domain (see [START_REF] Rijgersberg | How semantics can improve engineering processes: a case of units of measure and quantities[END_REF]. In [START_REF] Duque-Ramos | Oquare: a SQuaRE-based approach for evaluating the quality of ontologies[END_REF], a series of ontologies of units were evaluated, and we have selected one of them for this experiment, the Measurement Units Ontology SWEET 2.0 Scientific Units Ontology (SCIUNITS), which can be found at http:// sweet.jpl.nasa.gov/2.0/sciUnits.owl. This ontology has 326 classes, 54 object properties, 22 data properties, 103 individuals, 103 class assertion axioms, 134 object property assertion axioms and 76 data property assertion axioms. This ontology was selected because its size and design was found appropriate for this experiment, given our objectives and the constraints associated with the effort. Documents: The OQuaRE website was used by the experts as the reference for retrieving all the information about the framework, the quality model, the quality metrics, etc. That website also hosted a page describing the experiment and the forms to be filled by the experts.

Tools: A home-made software tool that implements the OQuaRE metrics was used. This tool takes an OWL ontology as input and generates a numeric report as output. This report includes the scores for all the characteristics, subcharacteristics and metrics involved in the evaluation process. This tool was not used by the participants for the evaluation although they were provided with the corresponding scores.

Data collection

As mentioned, this experiment was designed in two rounds, each of them having different objectives. Both rounds served us for collecting the data needed for analysing our evaluation goals. In the first round, the experts were asked:

Read the instructions and get used to the OQuaRE quality model, that is, characteristics and subcharacteristics.

Apply OQuaRE manually to the target ontology, that is, assigning values in the range [1,5] to each subcharacteristic.

Answer a questionnaire which contained questions about the difficulty and appropriateness of each subcharacteristic.

In the second round, the experts were asked for:

Read the instructions and get used to the OQuaRE quality metrics. This step includes to download from the website the scores for each metric calculated by our tool.

Apply OQuaRE to the target ontology, that is, assigning values in the range [1,5] to each subcharacteristic. For that purpose, they could use the metrics score as a reference.

Answer a questionnaire which contained questions about the usefulness and relevance of each metric.

Data analysis

The techniques used in the analysis of the data collected depend on the nature of data and on the particular goal. We have compared the results of the manual, computer-supported and automatic evaluation methods using descriptive statistics to obtain a basic description of the samples. ANOVA procedures have been used to test whether there are significant differences between sample means.

With respect to the data collected in round 1, we have applied agglomerative hierarchical clustering to the mean of the answers provided by the experts about the difficulty of getting a score for the quality subcharacteristics in the manual approach and the appropriateness of the quality subcharacteristics for grouping the subcharacteristics into classes. Hence, similar ones in terms of difficulty and appropriateness are in the same class. The procedure has been similar for round 2, looking for clusters of metrics based on their usefulness and relevance. Finally, k-means algorithms have been used to describe particular behaviours of the subcharacteristics in the three evaluation methods.

Results

In this section, we present the main results of this experiment. The complete results and the materials used in this experiment can be found at the following website: http://miuras.inf.um.es/evaluation/oquare/SCIUNITS/Experiment.html.

Hypotheses, variables and experimental design

Hypotheses and variables are established according to the research goals. The variables Difficulty and Appropriateness are obtained from the sample means of the values given by the experts about difficulty and appropriateness of the subcharacteristics in the questionnaire of round 1.

Our main statistical or null hypothesis says that there is no statistical difference on the mean score by using either the manual, manual assisted or the automatic method. The variables to contrast this hypothesis are obtained from the mean of the assigned values by the experts in each quality subcharacteristic in manual and assisted manual evaluation methods and the scores obtained by the automatic evaluation method, which are called M1, M2 and M3, respectively.

We have chosen an experimental design which tries to identify sources of variability in the experimental units (subcharacteristics), reduce the effect of such sources and improve the precision of the answers to questions of interest. This experimental design involves two factors, the method and the evaluated subcharacteristic. The method of evaluation will be the main factor whose measures (scores) are matched by subcharacteristic producing homogeneous blocks. In other words, the design is a multisample generalization of matched-pairs design. The subcharacteristics with no assigned value in M3 will be discarded in those procedures that need complete blocks.

Evaluating the answers of the expert

Before obtaining the variables of our study, we analysed the answers obtained from the methods M1 and M2. One way ANOVA applied to the corresponding data allows concluding that there are some pairs of sample means statistically different (P-value < 0.0001). The only two significant differences can be observed between Expert 1 with respect to Expert 2 and Expert 3 in M2. On the other hand, the experts do not modify significantly their mean judgment from M1 to M2.

Fig. 1 shows a basic description of the answers of experts in both methods by means of a boxplot, where the symbol inside the box represents the mean sample, and 95 percent confidence intervals for the means.

Comparison of the evaluation methods

Univariate description

We described the experimental data through summary statistics and graphics. Table 3 shows the basic statistics for the scores given to the case study ontology by using the three methods.

Those statistics are represented in the multiple boxplot in Fig. 2, where the line in the middle of each box corresponds to the median and the sides correspond to the first and third quartiles, respectively. The mean of M3 variable is greater than mean of M2 and M1 but we investigated whether some of these differences are significant. We analyse it in the next section.

Two way ANOVA

We have used the two way analysis of variance procedure with a single observation per cell, that is, by means of an additive model, to analyse whether the observed means scores of the three evaluation methods are statistically different, isolating and removing from the error term the variation attributable to the blocks (subcharacteristics). For this purpose, only complete blocks have been used, thus discarding subcharacteristics with missing values in M3.

The difference between the means of M1 and M3 methods is significant at 0.05 significance level but smaller than 0.04 (P-value = 0.044 in two-way additive ANOVA). Through this test we also obtain that the block factor does not explain a significant proportion of the total variance (P-value = 0.129 associated with this source of variation). According to this, if we only take into consideration the main factor there is not enough evidence that the means are different. Hence, the means score of the three methods are not statistically different at 0.05 of significance level (P-value = 0.055 in one-way ANOVA layout).

Given that the normality and homoscedasticity assumptions of ANOVA model cannot be accepted with this data set, non-parametric procedures were applied to obtain that the observed medians are not statistically different (P-value = 0.119 in Friedman test blocked by subcharacteristic).

The previous results show the lack of evidence to reject the null hypothesis, thus we can conclude that neither the mean nor the median depend on the evaluation method.

Trivariate description and classification

We calculated the matrix of sample correlations of M1, M2 and M3 variables, which are shown in Table 4 together with their associated P-values. The lack of correlation between M1 and M3 and between M2 and M3 and the significant correlation between M1 and M2 are useful because they mean that we could represent the experimental units (subcharacteristics) in two dimensions, discarding only a small proportion of the variation in the data and giving a proper interpretation of the new variables in terms of M1, M2 and M3.

Finally, we attempted to classify the subcharacteristics in three homogeneous groups according to their scores in M1, M2 and M3 by means of k-means clustering. The final partition of the iterative process produces three groups whose centroids appear in Table 5. The subcharacteristics belonging to Cluster 1 tend to have a profile with higher score than the mean in the three methods. The subcharacteristics belonging to Cluster 2 have a high score in M3 and have low score in M1 and M2. The subcharacteristics belonging to Cluster 3 tend to have a profile with lower score than the mean in the three methods.

Round 1: the quality model

We want to classify the subcharacteristics into clusters in such a way the profile of subcharacteristics in the Difficulty and Appropriateness variables in the same cluster are very similar, whereas the profile of subcharacteristics in different clusters are very different.

Cluster Analysis operates on the matrix of Euclidean distances between all pairs of characteristics and proceeds sequentially yielding a nested arrangement of characteristics in groups, in a hierarchical agglomerative process. We have also chosen Ward's linkage method to determine the distance between two clusters. We obtained a partition in three clusters or groups of subcharacteristics with a maximum of distance 3.46 between them. Table 6 shows the centroid of the clusters and the grand centroid. The subcharacteristics belonging to Cluster 1 have a profile similar to the grand centroid and can be described as appropriate and difficult.

The subcharacteristics of Cluster 2 can be described as difficult and not appropriate. Finally, the subcharacteristics of Cluster 3 can be described as appropriate and not difficult.

Table 7 shows the subcharacteristics bellowing to each group by applying of hierarchical cluster to Difficulty and Appropriateness variables. We analysed the answers provided by the experts about the relevance and usefulness of metrics in the questionnaire of round 2. Given that one metric may be associated with multiple subcharacteristics, the experts might have considered one metric useful or relevant for some subcharacteristics but not for some other ones. For each pair (subcharacteristic, metric), the experts were asked to give a value between 1 (lowest) and 5 (highest).

We applied k-means clustering to the mean of those answers for classifying those pairs (subcharacteristic, metric) in homogeneous groups according to their profiles in Relevance and Usefulness. The centroids of the three clusters obtained appear in Table 8. The pairs (subcharacteristic, metric) belonging to Cluster 1 tend to have profile with scores slightly higher than the mean in Relevance and Usefulness (see Table 9). The pairs belonging to Cluster 2 have profile with scores lower than the mean in Relevance and Usefulness (see Table 10). Finally, the pairs belonging to Cluster 3 have profile with scores greater than the mean in Relevance and Usefulness (see Table 11).

Discussion and conclusions

In this paper we have presented and evaluated the OQuaRe framework for ontology quality evaluation. The results have been presented in the previous section and the major findings and conclusions associated with such results are presented and discussed next.

Evaluation methods and evaluations of the experts

The ontology of the case study has been evaluated using three different methods. First, the experts were asked to evaluate manually the ontology, with no support except their own expertise and resources. Second, the experts were asked to repeat the process with the support of a series of pre-calculated metrics. Finally, the ontology was given a score by using only those metrics. The results show that we cannot say that there are significant differences between the scores of the three methods, and that the scores with the support of the metrics are higher. The scores of the second evaluation can be higher than in the first one due to several reasons. The experts might have become more familiar with the subcharacteristics due to the previous evaluation or due to the fact that the availability of the metrics may have contributed to a more precise understanding of the subcharacteristics and has provided additional information to the experts. The interesting result is that the same behaviour was found in the results of all the experts and that, even the mean of M2 is higher, there is no evidence of statistical difference, and there is no statistical difference between the scores in M1 and M2 of the experts when measured individually.

The quality model

According to the results, three clusters of quality subcharacteristics have been identified by grouping them in terms of difficulty and appropriateness. The largest cluster is Cluster 1, which includes the subcharacteristics that are moderately appropriate and moderately difficult, with means between 3 and 4, whereas the smallest is Cluster 2, which includes the difficult and not very appropriate ones. Consequently, it can be said that the set of subcharacteristics included in the quality model are of interest for the participating experts.

The experts found that the definition of some subcharacteristics should be improved since their understandability was part of the difficulty they found in their application. The lack of knowing the intended contexts of use of the ontology was key in the difficulty found by the experts in scoring some subcharacteristics. In the current model, the functional adequacy characteristics include subcharacteristics which might account for that, but this does not mean that they are the intended uses for the ontology under evaluation. SQuaRE includes a quality requirements module which has not been defined so far in OQuare and which would fulfill such requirement. Ontology developers would then determine which properties of the ontology should be evaluated, the functional requirements and the competency questions. The experts also made suggestions about the subcharacteristics. For instance, a division of structural accuracy into a series of subcharacteristics is proposed for being a broad notion. This deserves further study and analysis since some suggestions might be considered metrics instead of subcharacteristics.

The quality metrics

The objective of the second round was to evaluate the OQuaRE quality metrics and their usage in the framework. The associations between metrics and subcharacteristics have also produced three clusters. Cluster 1 includes the associations with intermediate relevance and usefulness, Cluster 2 contains the low relevance and usefulness metrics, and Cluster 3 contains the relevant and useful associations. It can be said that Clusters 1 and 3 contain positively considered associations and more than 50% of the associations belong to such clusters. Most metrics have been considered useful and relevant for at least one subcharacteristic, but three of them have been considered not useful for all its associations. Such results suggest that a redesign of the set of metrics is needed. The experts consider that some metrics makes sense only in a given context of use, and that making such distinction is important because, in the absence of additional information, it might be not possible to assess some task-based metrics. This affects specially to the functional category subcharacteristics and their metrics associated, which explains the bad results of such metrics associations. Thus, this result seems to be in line with the discussion of round 1.

The experts found difficult some metrics because of their definition in an OWL-independent way, whereas the ontologies under evaluation are OWL. This is due to the fact that OQuaRE has been designed to be applicable to ontologies in different formats and with different level of formalization. In this way, the experts miss some metrics relevant for OWL ontologies. They suggested new metrics, like relations in use or new associations between metrics and subcharacteristics, like ANOnto for Learnability. In some cases, they recommended having a limited number of metrics per subcharacteristic. This is consistent with the result obtained about the low relevance and usefulness of many pairs (metric, subcharacteristic).

Evolution of OQuaRE

According to the results, OQuaRE should be improved in several ways before being a candidate for supporting ontology evaluation processes. On the one hand, OQuaRE should be extended with the quality requirements module, which would allow for determining potential contexts of use that would be useful for the human evaluators. Given its relation with SQuaRE, OQuaRE provides a family of evaluation methods. New methods can be defined by just adapting the number of subcharacteristics associated with each characteristic, the number of metrics associated with each subcharacteristics, and the corresponding weights. This would then satisfy the expert suggestions of evaluation profiles, since different combinations could be associated with the different contexts of use of the ontologies. The set of subcharacteristics should also be refined, which can also be done given the philosophy of SQuaRE-based approaches.

It should be noted that our goal differ from other approaches like [START_REF] Oh | A comprehensive framework for the evaluation of ontology modularization[END_REF], which focus on the evaluation of ontology modularization, although we plan to see how their results could be used to improve our framework. We are currently extending the home-made tool that has been used in this experiment which would allow the definition of different evaluation methods based on the OQuaRE framework, including profiles for OWL ontologies. This seems a sensible way of proceeding according to the experts suggestions, and this would require to select not only generic metrics that could be applied to ontologies but also metrics specifically designed for OWL ontologies, like the ones provided by http://owl.cs.manchester.ac.uk/metrics/ or the Ontology Pitfall Scanner [START_REF] Poveda | Common pitfalls in ontology development[END_REF]. In addition to this, the work developed by the Ontology Usage Framework http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySum-mit2011_ApplicationFramework_Synthesis is certainly of interest for OQuaRE since they attempt to define a set of ontology usage characteristics and ontology value metrics that could be mapped onto ours.

  Fig. 1. Boxplots representing a basic description of the answers of experts in both methods and 95 percent confidence intervals for the means.

Table 1

 1 Some subcharacteristics of OQuaRE.

Table 2

 2 Definition of the metrics used in OQuaRE.

Table 3

 3 Basic statistics for the scores of the three methods (1 = lowest; 5 = highest).

	Method Mean StDev Minimum Q1	Median Q3	Maximum
	M1	2.76	1.19	1.00	2.00	2.50	3.50 5.00
	M2	3.08	0.68	2.00	2.66 3.00	3.66 5.00
	M3	3.33	1.16	1.00	2.66 3.00	4.50 5.00

Fig. 2. Boxplot representing basic statistics of scores methods.

Table 4

 4 Sample correlation of M1, M2 and M3.

		M1	M2
	M2	0.649
	P-value	0.000
	M3	À0.095	0.001
	P-value	0.624	0.994

Table 5

 5 The correlations between the original and new variables.

	Variable	Cluster centroid		Grand centroid
		Cluster 1	Cluster 2	Cluster 3
	M1	3.66	1.00	2.12	2.68
	M2	3.48	2.00	2.88	3.07
	M3	3.97	5.00	2.61	3.33

Table 6

 6 Cluster Analysis using Difficulty and Appropriateness variables.

	Variable	Cluster centroid		Grand centroid
		Cluster 1	Cluster 2	Cluster 3
	Difficulty	3.34	4.75	1.95	2.90
	Appropriateness	3.84	2.16	4.28	3.87

Table 8

 8 Cluster Analysis using the variables Usefulness and Relevance.

	Variable	Cluster centroid			Grand centroid
		Cluster 1	Cluster 2	Cluster 3
	Relevance	2.83	1.74		3.72	2.47
	Usefulness	2.44	1.32		3.51	2.09
	Table 9
	The associations included in Cluster 1, which can be considered with intermediate
	relevance and usefulness.
	Subcharacteristic		Metric Cluster 1
	Consistent search and query		ANOnto, INROnto, Formalization
	Knowledge acquisition; clustering	RROnto
	and similarity
	Formal relations support, indexing
	and linking
	Results representation		CROnto
	Knowledge reuse		ANOnto, Formalization, LCOMOnto
	Learneability			WMCOnto, LCOMOnto, NOMOnto,
				CBOnto, NOCOnto
	Modularity			WMCOnto
	Reusability			DITOnto, NOCOnto
	Analysability			CBOnto
	Changeability			WMCOnto, DITOnto, LCOMOnto,
				RFCOnto, NOMOnto
	Modification stability		WMCOnto, CBOnto, LCOMOnto,
				RFCOnto, NOCOnto
	Replaceability			NOCOnto, NOMOnto
	Adaptability			WMCOnto, RFCOnto, CBOOnto

Table 10

 10 The associations included in Cluster 2, which can be considered with low relevance and usefulness.

	Subcharacteristic	Metric Cluster 2
	Cohesion; availability	LCOMOnto
	Redundancy	ANOnto
	Schema and value reconciliation;	RROnto, AROnto
	consistent search and query
	Knowledge acquisition	ANOnto, NOMOnto
	Clustering and similarity; indexing and	AROnto
	linking; results representation;
	guidance and decision trees;
	knowledge reuse
	Text analysis	Formalization
	Recoverability	WMCOnto, DITOnto, NOMOnto,
		LCOMOnto
	Learneability	RFCOnto
	Reusability	WMCOnto, RFCOnto, NOMOnto,
		CBOnto
	Analysability	WMCOnto, DITOnto, LCOMOnto,
		RFCOnto, NOMOnto
	Testability	WMCOnto, DITOnto, LCOMOnto,
		RFCOnto, NOMOnto, CBOnto
	Replaceability	WMCOnto, DITOnto
	Adaptability	DITOnto

Table 11

 11 The associations included in Cluster 1, which can be considered relevant and useful.

	Subcharacteristic	Metric Cluster 3
	Tangledness	TMOnto
	Controlled vocabulary	ANOnto
	Schema and value reconciliation	Formalization, Consistency
	Indexing and linking; guidance and decision	INROnto
	trees
	Knowledge reuse	INROnto, NOMOnto,
		Consistency
	Modularity	CBOnto
	Changeability	CBOnto, NOCOnto

Acknowledgements

This work has been possible thanks to the funding of the Spanish Ministry of Science and Innovation through Grant TIN2010-21388-C02-02 and co-funded by the FEDER Programme. Mikel Egaña Aranguren is funded by the Marie Curie Cofund programme (FP7).