Hanane Moummou

Yvonne Kallberg
email: yvonne.kallberg@ki.se

Libert Brice Tonfack
email: libricetonfack@yahoo.fr

Bengt Persson

Benoît Van Der Rest
email: benoit.van-der-rest@ensat.fr

The Plant Short-Chain Dehydrogenase (SDR) superfamily: genome-wide inventory and diversification patterns

Keywords: Short-chain dehydrogenase/reductase (SDRs), SDR Nomenclature Initiative, Hidden Markov Model, Multigenic family, Plant

Background

Short-chain dehydrogenases/reductases (SDRs) form one of the largest and oldest NAD(P)(H) dependent oxidoreductase families. Despite a conserved 'Rossmann-fold' structure, members of the SDR superfamily exhibit low sequence similarities, which constituted a bottleneck in terms of identification. Recent classification methods, relying on hidden-Markov models (HMMs), improved identification and enabled the construction of a nomenclature. However, functional annotations of plant SDRs remain scarce.

Results

Wide-scale analyses were performed on ten plant genomes. The combination of hidden Markov model (HMM) based analyses and similarity searches led to the construction of an exhaustive inventory of plant SDR. With 68 to 315 members found in each analysed genome, the inventory confirmed the over-representation of SDRs in plants compared to animals, fungi and prokaryotes. The plant SDRs were first classified into three major types -'classical', 'extended' and 'divergent' -but a minority (10 % of the predicted SDRs) could not be classified into these general types ('unknown' or 'atypical' types). In a second step, we could categorize the vast majority of land plant SDRs into a set of 49 families. Out of these 49 families, 35 appeared early during evolution since they are commonly found through all the Green Lineage. Yet, some SDR families -tropinone reductase-like proteins (SDR65C), 'ABA2-like'-NAD dehydrogenase (SDR110C), 'salutaridine/menthone-reductase-like' proteins (SDR114C), 'dihydroflavonol 4-reductase'-like proteins (SDR108E) and 'isoflavone-reductase-like' (SDR460A) proteins -have undergone significant functional diversification within vascular plants since they diverged from Bryophytes. Interestingly, these diversified families are either involved in the secondary metabolism routes (terpenoids, alkaloids, phenolics) or participate in developmental processes (hormone biosynthesis or catabolism, flower development), in opposition to SDR families involved in primary metabolism which are poorly diversified.

Conclusion

The application of HMMs to plant genomes enabled us to identify 49 families that encompass all Angiosperms ('higher plants') SDRs, each family being sufficiently conserved to enable simpler analyses based only on overall sequence similarity. The multiplicity of SDRs in plant kingdom is mainly explained by the diversification of large families involved in different secondary metabolism pathways, suggesting that the chemical diversification that accompanied the emergence of vascular plants acted as a driving force for SDR evolution.

Background

Short-chain dehydrogenases/reductases (SDRs) constitute one of the largest and oldest protein superfamilies known to date. This ancient family, found in all domains of life (Archea, Eukaryotes, Prokaryotes and viruses), is characterized by large sequence divergences but several common properties: (i) a conserved 3D structure consisting of 'Rossmann-fold' β-sheet with α-helices on both sides, (ii) an N-terminal dinucleotide cofactor binding motif, (iii) an active site with a catalytical residue motif YxxxK [START_REF] Filling | Critical residues for structure and catalysis in short-chain dehydrogenases/reductases[END_REF][START_REF] Kavanagh | Medium-and short-chain dehydrogenase/reductase gene and protein families: the SDR superfamily: functional and structural diversity within a family of metabolic and regulatory enzymes[END_REF]. With the release of genome sequences of numerous living organisms, the availability of around 300 crystal structures and the identification of many enzymatic functions, much attention has been given to classify the members of the SDR superfamily. A first discrimination was established between five types of SDR: the 'classical' type, consisting of approximately 250 amino acids, the 'extended' type that has an additional 100-residue domain in the C-terminal region, the 'intermediate' type that displays a specific G/AxxGxxG/A cofactor binding motif, the 'divergent' type that comprises enoyl-reductases from plant and bacteria and harbours modifications both in the cofactor binding site and active site motifs and the 'complex' SDR which are usually part of large multi-domain enzymes, such as mammalian fatty acid synthases or bacterial polyketide synthases [START_REF] Kavanagh | Medium-and short-chain dehydrogenase/reductase gene and protein families: the SDR superfamily: functional and structural diversity within a family of metabolic and regulatory enzymes[END_REF][START_REF] Kallberg | Short-chain dehydrogenase/reductase (SDR) relationships: a large family with eight clusters common to human, animal, and plant genomes[END_REF][START_REF] Kallberg | Short-chain dehydrogenases/reductases (SDRs)[END_REF]. Moreover, the discovery of new oxidoreductase structures harbouring the SDR 'Rossmann-fold' motif revealed the existence of uncommon types, often referred to as 'unknown' or 'atypical' types. More recently, the diversity of SDRs, either their amino acid sequences or their functions, led to the development of a second classification effort: the 'SDR Nomenclature Initiative' that aims at being more informative regarding SDRs functions and at establishing a sustainable and expandable nomenclature system based on the use of a large set of hidden Markov models (HMM) [START_REF] Kallberg | Classification of the short-chain dehydrogenase/reductase superfamily using hidden Markov models[END_REF]. Nowadays, 449 families have been listed in this nomenclature [START_REF] Persson | SDR (short-chain dehydrogenases/reductases)[END_REF].

Although mentioned by several authors [START_REF] Kavanagh | Medium-and short-chain dehydrogenase/reductase gene and protein families: the SDR superfamily: functional and structural diversity within a family of metabolic and regulatory enzymes[END_REF][START_REF] Kallberg | Short-chain dehydrogenases/reductases (SDRs)[END_REF], the diversity of SDRs in plants has never been investigated thoroughly. The recent advances in sequencing techniques and the stillincreasing speed of genome releases now facilitate an exhaustive review of complex multigenic families. In the case of SDRs, a second challenge for plant scientific community is to unravel the functions of these oxidoreductases. Indeed, in the TAIR10 annotation of Arabidopsis thaliana genome, a large majority of 'classical' SDRs (two thirds) are merely annotated as NAD(P)-binding Rossmann-fold superfamily protein oxidoreductase [START_REF] Swarbreck | The Arabidopsis Information Resource (TAIR): gene structure and function annotation[END_REF]. This lack of information prompted us to adopt an exhaustive approach on plant SDRs. In a previous paper, we reviewed the involvement of different SDRs in primary and secondary metabolism [START_REF] Tonfack | The plant SDR superfamily: involvement in primary and secondary metabolism[END_REF]. In the present paper, we combined the use of HMMs and phylogenetic analyses on a set of genomes representative of plant diversity, in order to conduct a global inventory of plant SDRs coherent with the current SDR classification and nomenclature. This inventory was integrated into a functional classification of plant SDRs. Since this genomewide inventory confirmed the high diversity of plant SDRs, the distribution and evolution of the different SDR families was examined, notably to investigate the link between SDR diversification and the emergence of secondary metabolism in vascular plants.

Methods

Analysed genomes

Genome analyses were performed on ten distinct genomes comprising four Dicots, three Monocots, the Pteridophytae Selagniella moellendorffii, the moss Physcomitrella patens and the Alga Chlamydomonas reinhardtii. Sequences and most annotations were downloaded from the Joint Genome Institute website. The predicted proteomes analysed [START_REF] Swarbreck | The Arabidopsis Information Resource (TAIR): gene structure and function annotation[END_REF][START_REF] Merchant | The Chlamydomonas genome reveals the evolution of key animal and plant functions[END_REF][START_REF] Rensing | The Physcomitrella genome reveals evolutionary insights into the conquest of land by plants[END_REF][START_REF] Banks | The Selaginella genome identifies genetic changes associated with the evolution of vascular plants[END_REF][START_REF] Tuskan | The genome of black cottonwood, Populus trichocarpa (Torr. & Gray)[END_REF][START_REF] Jaillon | The grapevine genome sequence suggests ancestral hexaploidization in major angiosperm phyla[END_REF][START_REF] Schmutz | Genome sequence of the palaeopolyploid soybean[END_REF][START_REF] Ouyang | The TIGR Rice Genome Annotation Resource: improvements and new features[END_REF][START_REF] Schnable | The B73 maize genome: complexity, diversity, and dynamics[END_REF][START_REF] Paterson | The Sorghum bicolor genome and the diversification of grasses[END_REF] were deduced from the annotations given in Table 1. The number of loci corresponds to the protein coding genes predicted by the annotation

HMM-based analyses of plant genomes

Genomic sets of predicted proteins were challenged with three Pfams HMMs [START_REF] Finn | The Pfam protein families database[END_REF]: PF00106, PF01370 and PF01073 using HMMER3. SDR Nomenclature Initiative HMMs were defined and updated as described previously [START_REF] Kallberg | Classification of the short-chain dehydrogenase/reductase superfamily using hidden Markov models[END_REF]. The five SDR types ('classical', 'extended', 'intermediate', 'divergent', and 'complex') each has an HMM trained to identify sequences of respective type. The HMMs were created using HMMER3, with manually adjusted alignments of representative sequences as seed. Cutoffs are used to decide if a hit is significant or not: 'classical' -138, 'extended' -108, 'intermediate' -162, 'divergent' -160, and 'complex' -140. In addition to the five types, an 'unknown' label is used for sequences with scores lower that these cutoffs but still high enough to safely predict the sequence as an SDR: 'classical' -29, 'extended' -75, and 'divergent' -100. Scores below the cutoffs are considered not positive.

For the PLR/IFR family, an HMM was created and incorporated to the 'SDR Nomenclature Initiative' set of family HMMs. The procedure for training the HMM was the same as previously developed with iterative refinement of the model until no new members were found [START_REF] Kallberg | Classification of the short-chain dehydrogenase/reductase superfamily using hidden Markov models[END_REF].

Decision rules for SDR inventory

For each sequence recognized by a HMM (hit), a score was assigned. Yet, several sequences were only recognized by one or two HMMs (either the Pfam derived HMM or the SDR-type HMM) and sometimes with a very low score. Thus, we defined a series of rules schematized in a decision tree (Figure 1). Sequences that were recognized by SDR nomenclature initiative were directly considered as positive. Sequences identified with both remaining sets of HMMs were also considered as positive. For the remaining sequences recognized only by one HMM, we first looked at the existence of strong homology with positive hits identified in the previous steps in order to include putative 'truncated' proteins (see Inventory refinement, below). Alternatively, we checked individually the existence of structural data in the scientific literature, which allowed either including some hits in our inventory or discarding certain families of enzymes, notably the medium-chain dehydrogenases that display distinct structural motifs. In absence of structural data, the sequences recognized by a single HMM could not be classified and were included in a list of ambiguous sequences (Additional file 1: Table S1) that contains oxido-reductases that still await structural data before confirming or infirming their belonging to the SDR superfamily.

Figure 1 Decision rules used to make an inventory of plant SDRs using three sets of HMM. All the HMM sets were run independently on the 10 predicted proteomes. The complete inventory and the ambiguous predictions are included as supplementary material (Additional file 2: Table S2 and Additional file 1: Table S1)

Inventory refinement

For the gene loci that are associated with several gene models and therefore with different protein predictions, a sole amino acid sequence was selected according two criteria: (1) the maximum HMM score and (2) the maximum alignment score deduced from a BlastP performed on other plant genomes. When the HMM and BlastP analyses led to contradictory predictions, a single protein prediction was manually selected after aligning the different gene models with its closest homologues. To include in the SDR classification the truncated proteins that failed to be recognized by the HMMs, a BlastP sequence search was performed on each genome using as query sequences the complete list of SDRs recognized in the first round of HMM searches. All sequences that displayed a segment of 60 amino acids with more than 50 % identity were classified in the same type or family as its closest homologue.

Distance matrices and phylogenetic analyses

Phylogenetic analyses and distance matrices were built using the Mega5 package [START_REF] Tamura | MEGA5: molecular evolutionary genetics analysis using maximum likelihood, evolutionary distance, and maximum parsimony methods[END_REF]. Full length amino acid sequences were aligned using the ClustalW algorithm. Distance matrices evaluating the percentage of sequence identity were calculated on the basis of p-distance with the pairwise deletion option. Unless stated differently, phylogenetic trees were built using the Neighbor-Joining method. The percentage of replicate trees in which the associated taxa clustered together was calculated in the bootstrap test (500 replicates). Trees were drawn to scale, with branch lengths in the same units as those of the evolutionary distances used to infer the phylogenetic tree. The evolutionary distances were computed using the Poisson correction method and were expressed as the units of the number of amino acid substitutions per site.

Statistical analyses

Principal Component Analysis was performed on the distribution matrix given in Figure 2 using the R 2.14.1 software [START_REF] Core | R: A language and environment for statistical computing[END_REF] with in-house developed scripts (Elie Maza, personnal communication). The robustness of the conclusions was checked by carrying the same analysis after removal of the individual exhibiting extreme values (SDR108E). S2. The blue to red color gradient reflects the number of SDR listed in each family in the different genomes; the absence of family is indicated with a white square. The names of the families were deduced from the 'SDR Nomenclature Initiative' HMMs or by a representative gene accession for orphan families not recognized by a specific HMM

Results and discussion

HMM-driven inventory of plant SDR

Initial HMM analyses were performed on ten complete genomes: 4 Eudicots (Arabidopsis thaliana, Populus trichocarpa, Vitis vinifera, Glycine max), 3 Monocots (Zea mays, Oryza sativa and Sorghum bicolor), the lycophyte Selaginella moellendorffii, the moss Physcomitrella patens and the unicellular green algae Chlamydomonas reinhardtii (Table 1). The predicted 'proteomes' deduced from the genome annotations were searched against three distinct sets of HMMs: the Pfam HMMs considered to encompass most SDR (PF00106, PF01370, PF01073), HMMs developed in the framework of the SDR nomenclature initiative [START_REF] Kallberg | Classification of the short-chain dehydrogenase/reductase superfamily using hidden Markov models[END_REF][START_REF] Persson | The SDR (short-chain dehydrogenase/reductase and related enzymes) nomenclature initiative[END_REF] and a set of HMMs developed to predict the type ('classical', 'extended', 'intermediary', 'divergent' and 'complex') of SDR (see Methods).

This first analysis led to an exhaustive inventory of plant SDRs presented in supplemental data (Additional file 2: Table S2 and Additional file 1: Table S1). This inventory was divided into a main list (Additional file 2: Table S2), where the HMM scores or the high similarity with known SDRs were sufficient to establish a good prediction, and a complementary list of ambiguous SDR predictions (Additional file 1: Table S1), containing proteins with low HMM scores and absence of structural data (see decision tree in Figure 1 and Methods). Despite its very low HMM scores, we included in the main list a large family that comprises pinoresinol reductase (PLR), isoflavone reductase (IFR), vestitone reductase, phenylcoumaran benzylic ether reductase and eugenol synthase. Indeed, the structures of several members of this family were resolved by crystallography and the data revealed the presence of a SDR-typical Rossmann-fold [START_REF] Min | Crystal structures of pinoresinollariciresinol and phenylcoumaran benzylic ether reductases and their relationship to isoflavone reductases[END_REF][START_REF] Shao | Crystal structure of vestitone reductase from alfalfa (Medicago sativa L.)[END_REF][START_REF] Wang | Crystal structure of isoflavone reductase from alfalfa (Medicago sativa L.)[END_REF][START_REF] Louie | Structure and reaction mechanism of basil eugenol synthase[END_REF]. Subsequently, an HMM was created and incorporated to the 'SDR Nomenclature Initiative' set of HMMs. The PLR/IFR family was named SDR460A, where the 'A' stands for 'atypical'.

Distribution of plant SDRs

The number and the distribution of plant SDRs of different types are summarized in Table 2.

As in other Eukaryotes, the major types consist of 'classical' and 'extended' SDRs. 'Divergent' SDRs are limited to one conserved family: an enoyl-ACP reductase (ENR) involved in lipid biosynthesis (AT2G05990.1 in Arabidopsis) [START_REF] Tonfack | The plant SDR superfamily: involvement in primary and secondary metabolism[END_REF][START_REF] Rafferty | Common themes in redox chemistry emerge from the X-ray structure of oilseed rape (Brassica napus) enoyl acyl carrier protein reductase[END_REF]. While neither 'intermediate' nor 'complex' types are found in plants, we can notice a high number of 'unknown' types, meaning that the sequence patterns clearly differ from the other types. As previously noticed [START_REF] Kavanagh | Medium-and short-chain dehydrogenase/reductase gene and protein families: the SDR superfamily: functional and structural diversity within a family of metabolic and regulatory enzymes[END_REF], the SDR family is highly represented in the plant kingdom: while 73 SDRs were numbered in the human genome [START_REF] Bray | The human short-chain dehydrogenase/reductase (SDR) superfamily: a bioinformatics summary[END_REF] and 39 in the cyanobacteria Synechocystis sp. PCC 6803 [START_REF] Kramm | Short-chain dehydrogenases/reductases in cyanobacteria[END_REF], the number of SDRs in land plants vary from 126 in the moss P. patens to 315 in soybean (G. max). Even if we consider the variations due to the genome sizes (Table 1), SDRs are more represented in the Angiosperms than in the algae C. reinhardtii or in P. patens, suggesting a relationship between the emergence of vascular plant and the apparent multiplicity of plant SDRs.

Sub-classification of plant SDR

The HMMs developed by the SDR nomenclature initiative [START_REF] Kallberg | Classification of the short-chain dehydrogenase/reductase superfamily using hidden Markov models[END_REF] aim at classifying the SDR superfamily into a large number of families, at a level where this classification would be informative regarding the functions of SDRs. In a first analysis, the HMMs defined in the frame of the SDR nomenclature initiative directly recognized 74 % of plant SDRs. After performing similarity searches and associating truncated proteins with its closest homologues (see Methods), the proportion of non-classified SDRs dropped markedly since 94.5 % of plant SDRs were categorized into 49 families. While the majority of these families are found in most Tracheophytes (Table 3), seven (SDR58C, 59C, 74C, 86C, 90C, 103U, 107C; Additional file 2: Table S2) are found only in P. patens or in C. reinhardtii. The occurrence of different family sequences in the analysed genomes was represented as a heat map (Figure 2). -ViridP 56,5 Each family was associated with a representative gene and, when possible, with a specific SDR nomenclature initiative HMM. Information on the occurrence of SDRs in different genomes are reported by the taxon name (ViridP: Viridiplantae; LandP: Embryophytae; TracheoP: Tracheophytae; FlowerP: Magnioliophyta). Average pairwise identities were calculated from the sequences of plant genomes. Ambiguously predicted SDRs and families absent in flowering plants were omitted. *: occurrence in Viriplantae was deduced from the presence of homologues in other Green Algae genomes On the opposite, 5.5 % of plant SDRs (from 4 % in Angiosperms to 29 % in C. reinhardtii) remained unclassified. The existence of these orphan SDRs lays in the conception of the 'SDR nomenclature initiative' HMMs. In order to achieve robust HMMs, the authors considered only families with sufficient number of representative and non-redundant sequences [START_REF] Kallberg | Classification of the short-chain dehydrogenase/reductase superfamily using hidden Markov models[END_REF], thus excluding SDR families with too few members. To circumvent this difficulty, we examined the possibility to define new families on the sole basis of amino-acid sequence conservation. Therefore, all the unclassified SDRs from the main inventory (Additional file 2: Table S2) were associated to its closest homologues using BlastP searches and sequence alignments. Interestingly, all the unclassified sequences from Angiosperms clearly matched with at least one Arabidopsis SDR, the e value obtained from a BlastP against Arabidopsis predicted proteome never exceeding 1 e-40. Thus, seven new clusters were defined on the basis of sequence conservation, four being common to all the Viridiplantae genomes while three were found only in land plants (Figure 2 and Table 3). Within these clusters, the average pairwise sequence identities ranged from 48 % to 62 %. These conservation rates are consistent with the average pairwise identities observed for the families defined by a 'SDR nomenclature initiative'-HMM, that ranges from 37 % to 82 % identity (Table 3). All these clusters were represented by a limited number of sequences in each genome, supporting the explanation that the lack of 'SDR-nomenclature-initiative'-HMMs is simply the consequence of an insufficient set of sequences and that these families might be defined in the future, with the release of new sequences in the UNIPROT database. To complete the plant SDR classification, each new cluster was assigned a representative gene, based on an Arabidopsis thaliana identifier. While all angiosperms SDRs could be categorized in a family, defined either by a specific HMM or by primary structure conservation, 15 sequences from C. reinhardtii, 4 sequences from P. patens and one sequence from S. moellendorffii were too distant to other SDR sequences and remained unclassified.

By extension, the ambiguous SDR sequences were also clustered on the basis of sequence homologies, allowing the definition of nine potential families (Additional file 1: Table S1). Yet, in absence of structural data confirming the existence of typical SDR structures, these sequences were not analysed further.

In a last step, plant SDR classification was combined with functional information. Taking advantage of our previous bibliographic research [START_REF] Tonfack | The plant SDR superfamily: involvement in primary and secondary metabolism[END_REF] and of the annotations found for Arabidopsis (TAIR10), we completed the classification by mentioning all the known functions described in the scientific literature in Table 3. Also, to each family, a representative gene was chosen according to three criteria: (1) favour Arabidopsis accessions with respect to the quality of TAIR annotations and its pertinence as a model plant; (2) when possible, opt for genes that have been functionally characterised; otherwise (3), priority was given to the accession that displayed the lowest average distance with other members of its family.

Evolution and diversification of plant SDR as a potential trait of land plant emergence?

The distribution of the different families in the different taxa was further examined to understand the evolution of the plant SDR superfamily. We first addressed the question of potential origins of the different SDR families. Out of the 49 families listed in Table 3, 32 were found both in the algae C. reinhardtii and in the majority of land plants, suggesting that most plant SDRs families emerged prior to land plant radiation that started -460 Myear ago, in the Ordovician period [START_REF] Steemans | Origin and radiation of the earliest vascular land plants[END_REF]. For three additional families (SDR40C, SDR42E and SDR358U), the absence of a member in C. reinhardtii or even in P. patens predicted proteomes masked the occurrence of these families in other genus of green algae (Volvox, Micromonas, Chlorella and Ostreococcus), suggesting that the families were ancestral, but that the genes might have been lost in some taxa. In addition, 10 families absent in green algae are common to all land plants (Figure 2 and Table 3), indicating that 45 families are shared among land plants (embryophytes). 48 families were common to vascular plants as 3 additional families were specific to S. moellendorffii and Angiosperms. At last, a sole family, SDR115E, was found only in Angiosperms. The origins of some families may be very ancient: SDR1E, 2E, 6E and 7C families are found in all domains of life (Archea, Eukaryote, Prokaryote) while the SDR12C, 17C, 25C, 34C, 35C, 22E and 31E families are common to the majority of Eukaryotes [START_REF] Kallberg | Classification of the short-chain dehydrogenase/reductase superfamily using hidden Markov models[END_REF]. Besides, several ancestral SDR families are close to Prokaryotic 'homologues'. For example, the origin of the plastids is illustrated by the presence of chloroplastic SDRs similar to its cyanobacterial homologue. In a recent paper, Kramm et al. [START_REF] Kramm | Short-chain dehydrogenases/reductases in cyanobacteria[END_REF] listed 39 SDRs in the genome of the cyanobacteria Synechocystis sp. PCC 6803. 20 of these SDRs show clear homologies (>35 % identity) with plant SDRs (data not shown). The SDRs clusters present both in cyanobacteria and plant genomes include the very ancient families (SDR1E, 2E, 3E, 6E) and several plastidial proteins involved in primary metabolism, such as sulfolipid biosynthesis protein (SDR52E), protochlorophyllide oxidoreductase (SDR73C), 3,8-divinyl protochlorophyllide a 8-vinyl reductase (SDR98U) or the members of the fatty acid synthase (FasII) complex (SDR152C and SDR87D).

The origin of these taxon-specificities probably results from three evolutionary mechanisms: horizontal gene transfers, differentiation of a novel family from a pre-existing SDR family and loss of genes. Indeed, Tarrio et al. [START_REF] Tarrío | The Vein Patterning 1 (VEP1) gene family laterally spread through an ecological network[END_REF] established that the Vein Patterning 1 (SDR75U) gene family had undergone five lateral gene transfer events, one occurring from bacteria to an ancestor of land plants. Conversely, extensive search of SDR homologues in the Genbank database revealed clear homologies between independent taxa, such as the similarities between the Tracheophyte SDR68C members and its Proteobacteria homologues or the close relationship between plant PLR-IFR family and Bacteria or Ascomycete isoflavone reductase-like proteins (data not shown), thus illustrating the possible importance of horizontal gene transfers. An original example of SDR differentiation is illustrated by the emergence of the Angiosperm-specific HC-toxin reductase (SDR115E) family, involved in the pathogen Helminthosporium carbonum (HC) toxin reduction [START_REF] Meeley | A biochemical phenotype for a disease resistance gene of maize[END_REF]. Since previous phylogenetic analyses [START_REF] Sindhu | A guardian of grasses: specific origin and conservation of a unique disease-resistance gene in the grass lineage[END_REF] showed the existence of significant homologies between HC-toxin reductase (SDR115E) and the large dihydroflavonol 4-reductase (4-DFR, SDR108E) family, we integrated SDR108E and SDR115E amino acids sequences in the same alignment and phylogenetic analysis (Figure 3 and Additional file 3: Figure S1A). The topology of the deduced tree (Figure 3 and Additional file 3: Figure S1A) suggests that the SDR115E branch belongs to a larger clade that includes 4-DFR (AT5G42800.1 cluster, [START_REF] Shirley | Effects of ionizing radiation on a plant genome: analysis of two Arabidopsis transparent testa mutations[END_REF]), anthocyanidin reductase (AT1G61720.1 cluster, [START_REF] Devic | The BANYULS gene encodes a DFR-like protein and is a marker of early seed coat development[END_REF][START_REF] Xie | Role of anthocyanidin reductase, encoded by BANYULS in plant flavonoid biosynthesis[END_REF]) and the brassinosteroid related 4-DFR-like protein BEN1 (AT2G45400.1 cluster, [START_REF] Yuan | BEN1, a gene encoding a dihydroflavonol 4-reductase (DFR)-like protein, regulates the levels of brassinosteroids in Arabidopsis thaliana[END_REF]). The robustness of this topology was further checked using different phylogeny algorithms (Neighbour-Joining and Maximum Likelihood) or rooting the tree with external sequences from other SDR families (SDR1E, SDR6E, SDR31E). All trees displayed similar topologies, SDR115E members always clustering with 4-DFR, anthocyanidin reductase and BEN1 (data not shown), thus supporting the view that the HC-toxin reductase (SDR115E) branch evolved from an ancestor belonging to the SDR108E family. The divergences of sequences within the SDR108E-115E 'clade' were sufficient to establish two distinct HMM profiles. At last, two distinct features may illustrate the role of loss of genes in SDR evolution: (i) although found in Monocots, grapevine, poplar and soybean genomes, the SDR115E family is absent in Arabidopsis genome or ESTs database; (ii) some families found in P. patens or in S. moellendorffii genomes (SDR74C, 86C, 103U, Additional file 2: Table S2) are absent in all the Angiosperms genomes, suggesting that genes might have been lost during before flowering plants radiation.

Figure 3 Phylogenetic tree of the SDR108E and SDR115E families. The blue arrow

indicates the node at the origin of the 'AnR, 4-DFR and SDR115E' branch. Amino acid sequences recognized by the SDR108E and SDR115E HMMs were aligned with ClustalW algorithm. The evolutionary history was inferred using the Neighbor-Joining method. The percentages of replicate trees in which the associated taxa clustered together in the bootstrap test (500 replicates) are shown next to the branches. Full references of sequences compressed in different clusters are provided as supplemental data (Additional file 3: figure 1A). Consistent trees were obtained using the Maximum Likelihood method or rooting the tree with other SDR families (SDR1E, SDR6E, SDR31E) as outgroups

The second obvious feature, when observing the distribution of SDR families (Figure 2), is the expansion pattern of the different families. A Principal Component Analysis (PCA) was performed on the distribution matrix used to build the heat map presented in Figure 2. It allowed the individualization of ten families displaying high values on the first axis (Figure 4A). All these families are characterized by a large number of members in contrast to the majority of SDR families represented in plant genomes with a limited set of sequences. Interestingly, the second axis is mainly driven by the vectors formed by P. patens and C. reinhardtii genomes (Figure 4B) and it discriminates two patterns of diversification: families expanded both in the moss P. patens and in vascular plants (SDR1E, SDR2E, SDR6E, SDR7C, SDR50E) and families expanded in vascular and flowering plants (SDR65C, SDR108E, SDR110C, SDR114C and SDR460A). Remarkably, all the five families expanded in vascular plants comprise enzymes involved in secondary metabolism (Table 3): tropinone reductases (SDR65C) are known for their involvement in alkaloids biosynthesis; SDR110C NAD-dehydrogenases oxidize various phenolic or terpeninc compounds, including xanthoxin, a precursor of abscissic acid (ABA); SDR114C menthone and salutaridine reductase, are involved in monterpene and alkaloid metabolism respectively; the large SDR108E family members catalyze the reduction of several phenolic precursors (4-dihydroflavonol, anthocyanidin, cinnamoyl-CoA, phenylacetaldehyde or eutypine) and last, the atypical PLR/IFR family (SDR460A) is also involved in phenolic metabolism. On the opposite, several poorly diversified clusters (SDR52E, 73C, 152C, 87D, 357C) that contain highly conserved sequences participate in primary metabolism such as chlorophyll synthesis or degradation, lipid metabolism or vitamin synthesis.

Identification of functional clusters within SDR families

For multigenic SDR families, the analyses can be conducted further with phylogenetic calculations. To illustrate the importance of this complementary approach, we focused on two large families involved in secondary metabolism: SDR110C (ABA2 xanthoxin dehydrogenase family) and SDR108E (4-DFR) family. For tropinone reductase (SDR65C) and menthone/salutaridine reductase (SDR114C) families, readers are referred respectively to Brock et al. [START_REF] Brock | The functional divergence of short-chain dehydrogenases involved in tropinone reduction[END_REF] and Ziegler et al. [START_REF] Ziegler | Evolution of morphine biosynthesis in opium poppy[END_REF] for complete phylogenetic analyses.

In our previous review [START_REF] Tonfack | The plant SDR superfamily: involvement in primary and secondary metabolism[END_REF], we listed six different functions described for SDR110C in the scientific literature: Arabidopsis ABA2 xanthoxin dehydrogenase (abscissic acid biosynthesis) [START_REF] González-Guzmán | The short-chain alcohol dehydrogenase ABA2 catalyzes the conversion of xanthoxin to abscisic aldehyde[END_REF][START_REF] Cheng | A unique short-chain dehydrogenase/reductase in Arabidopsis glucose signaling and abscisic acid biosynthesis and functions[END_REF], rice diterpenoid momilactone synthase A [START_REF] Shimura | Identification of a biosynthetic gene cluster in rice for momilactones[END_REF], mint (Mintha sativa) isopiperitenol dehydrogenase [START_REF] Ringer | Monoterpene metabolism. Cloning, expression, and characterization of (-)-isopiperitenol/(-)-carveol dehydrogenase of peppermint and spearmint[END_REF], Forsythia intermedia secoisolariciresinol dehydrogenase (lignan biosynthesis), the maize or rice feminization gene TASSELSEED2 [START_REF] Delong | Sex determination gene TASSELSEED2 of maize encodes a short-chain alcohol dehydrogenase required for stage-specific floral organ abortion[END_REF] and the Arabidopsis AtATA1 gene, involved in pollen and anther tapetal cells development [START_REF] Lebel-Hardenack | Conserved expression of a TASSELSEED2 homolog in the tapetum of the dioecious Silene latifolia and Arabidopsis thaliana[END_REF]. A phylogenetic tree was established from the analysed genome sequences and completed with mint isopiperitenol dehydrogenase and forsythia secoisolariciresinol dehydrogenase sequences (Figure 5 and Additional file 3: Figure S1B). Remarkably, the six functions described in the literature were distributed on five different clades, thus giving valuable hypotheses regarding the putative function of the orthologues or paralogues in other Angiosperm species. On the different clades, we also observe that highly homologous SDRs are often clustered in specific chromosomal regions, illustrating the importance of gene duplication events in the diversification process. At last, the different accessions of Selaginella are distributed in three distinct clades. However, the bootstrap values of the different nodes were too low to clearly establish a clear relationship between lycophytes and Angiosperms SDR110C sequences.

Figure 5

Phylogenetic tree of the SDR110C family. Amino acids sequences recognized by the SDR110C HMM were aligned with ClustalW algorithm. The evolutionary history and the bootstrap test (500 replicates) were computed as described for SDR108E (Figure 3). Full references of sequences compressed in different clusters are provided as supplemental data (Additional file 3: Figure S1B) For the highly variable SDR108E family, we included the SDR115E family in the analysis as both family are closely related (see above). As reported for the SDR110C family, several branches can be associated with functions described in the literature [START_REF] Tonfack | The plant SDR superfamily: involvement in primary and secondary metabolism[END_REF]: 4-DFR [START_REF] Shirley | Effects of ionizing radiation on a plant genome: analysis of two Arabidopsis transparent testa mutations[END_REF], anthocyanidin reductase (AnR) [START_REF] Devic | The BANYULS gene encodes a DFR-like protein and is a marker of early seed coat development[END_REF][START_REF] Xie | Role of anthocyanidin reductase, encoded by BANYULS in plant flavonoid biosynthesis[END_REF], HC-toxin reductase [START_REF] Meeley | A biochemical phenotype for a disease resistance gene of maize[END_REF], phenylacetaldehyde reductase [START_REF] Tieman | Tomato phenylacetaldehyde reductases catalyze the last step in the synthesis of the aroma volatile 2-phenylethanol[END_REF], cinnamoyl-CoA reductase (CCR) [START_REF] Lacombe | Cinnamoyl CoA reductase, the first committed enzyme of the lignin branch biosynthetic pathway: cloning, expression and phylogenetic relationships[END_REF] or eutypine reductase [START_REF] Guillén | A novel NADPH-dependent aldehyde reductase gene from Vigna radiata confers resistance to the grapevine fungal toxin eutypine[END_REF] (Figure 3). In contrast to SDR110C, the tree is also informative concerning the evolution of SDRs among land plant since distinct sequences from S. moellendorffii and P. patens are clearly associated with independent clades. These associations are of special interest for certain classes of enzymes such as the CCR catalysing the first irreversible oxidation step leading to monolignol synthesis. Indeed, several enzymes involved in the lignin biosynthesis pathway appeared early in land plant evolution and the moss are believed to accumulate uncondensed monolignols [START_REF] Xu | Comparative genome analysis of lignin biosynthesis gene families across the plant kingdom[END_REF]. Thus, the association on the same branch of sequences from P. patens and S. moellendorffii with Angiosperms bona fide CCR suggests that the enzyme anciently acquired its specificity and diverged rapidly from other SDR108E members. Last, as observed for SDR110C, several highly similar genes are clustered in specific chromosomal regions. Hence, with numerous members and a low conservation rate of amino acid sequences, the SDR108E family and its daughter branch SDR115E constitute a good example of a gradual and fast evolution of a multigenic family. Since the majority of the described enzymes reduce phenolic compounds, we may hypothesize that the SDR108E evolution accompanied tightly the complexification of phenolic and phenylpropanoid metabolism during land plant radiation.

Although essential for the functional study of large SDR families, phylogenetic analyses may be very informative for smaller families as well. This is the case of the Non-Yellow-Coloring 1 (NYC1) chlorophyllase b family, where the phylogenetic analyses clearly divide the family in two distinct clades: NYC1 and NOL (Non-yellow-coloring-Like) that diverged from a common ancestor (Figure 6). It was suggested that during evolution, the divergence led to the emergence of a functional hetero-oligomer, since both genes are necessary for chlorophyll b degradation [START_REF] Kusaba | Rice NON-YELLOW COLORING1 is involved in light-harvesting complex II and grana degradation during leaf senescence[END_REF][START_REF] Sato | Two short-chain dehydrogenase/reductases, NON-YELLOW COLORING 1 and NYC1-LIKE, are required for chlorophyll b and light-harvesting complex II degradation during senescence in rice[END_REF].

Figure 6

Phylogenetic trees of the chlorophyllase b (NYC1/NOL) family. Amino acids sequences from the AT4G13250 family were aligned with ClustalW algorithm. The evolutionary history and the bootstrap test (500 replicates) were computed as described for SDR108E (Figure 3) At last, we carried analyses on three SDR families involved in lipids primary metabolism: fatty-acid synthase (FAS-II)-β-ketoacyl synthase (SDR152C) [START_REF] Slabas | Molecular cloning of higher-plant 3-oxoacyl-(acyl carrier protein) reductase. Sequence identities with the nodG-gene product of the nitrogen-fixing soil bacterium Rhizobium meliloti[END_REF], (FAS-II)-enoyl-ACP reductase (SDR87D) [START_REF] Kater | cDNA cloning and expression of Brassica napus enoyl-acyl carrier protein reductase in Escherichia coli[END_REF] and the UDP-sulfoquinovose synthase (SDR52E) involved in sulfolipids biosynthesis [START_REF] Sanda | Recombinant Arabidopsis SQD1 converts udp-glucose and sulfite to the sulfolipid head group precursor UDPsulfoquinovose in vitro[END_REF]. By contrast with the families involved in secondary metabolism discussed above, the average sequence identity is high (Table 3), ranging from 68 % to 75 %. When the N-terminal chloroplast peptide signals are removed from sequence alignments, the average identities reach the scores of 79 % (SDR152C) and 84 % (SDR87D and SDR 52E). Phylogenetic trees were deduced from sequence alignments (Figure 7). Despite the presence of some duplication events observed for SDR152C (Figure 7A) and SDR87D (Figure 7B), the tree topologies are in good agreement with plant taxonomy for all the three SDR families, thus suggesting that the primary structure has been conserved under a high pressure of selection. (C) SDR52E UDP-sulfoquinovose synthase (SQD1). Amino acids sequences recognized by the SDR152C, SDR87D and SDR52E HMMs were aligned with ClustalW algorithm. The evolutionary history and the bootstrap test (500 replicates) were computed as described for SDR108E (Figure 3)

Conclusions

Work presented in this paper aimed at providing a full picture of plant SDRs using the current classification, especially the recent SDR nomenclature initiative. The combination of HMM models and similarity searches enabled us to classify most of the plant SDRs into a core of 49 families. Of these 49 families, 42 could be associated to an HMM, while the other 7 families being only defined on the basis of amino acids sequence conservation. Remarkably, all predicted SDRs from Angiosperms or S. Moellendorffii (corresponding to the so-called 'higher plants') could be categorized within these families. As all families exhibit a high degree of primary structure conservation, the average amino acid identities ranging from 37 % to 87 % among plant genomes, all SDRs sequences from Angiosperms can be analysed easily on the sole basis of sequence alignment, using very classical software (Blast, Multialin, ClustalW). For moss P. patens and green algae C. reinhardtii sequences, the predictions are less accurate, 3 % and 20 % of predicted SDRs remain unclassified. This limitation probably results from the under-representation of bryophyte and chlorophyte sequences compared to Angiosperms. In addition, the development of genome sequencing on more distant taxa (for example charophytes, liverworts or hornworts) should increase the number of UNIPROT sequences with sufficient divergences, thus improving the quality of HMM and allowing, in a mid-term, the definition of HMMs for the orphan SDR families.

Strikingly, the number of families found in Angiosperms [START_REF] Kusaba | Rice NON-YELLOW COLORING1 is involved in light-harvesting complex II and grana degradation during leaf senescence[END_REF] does not differ much from the 47 SDR families listed in the human genome [START_REF] Kallberg | Classification of the short-chain dehydrogenase/reductase superfamily using hidden Markov models[END_REF]. The large proportion of families (35 out of 49) found in all Viridiplantae, from Algae to Angiosperms, is consistent with the view that most SDR sub-branches diverged early during evolution [START_REF] Jörnvall | Superfamilies SDR and MDR: from early ancestry to present forms. Emergence of three lines, a Znmetalloenzyme, and distinct variabilities[END_REF]. Plants possess either SDRs common to all Eukaryotes or SDRs of bacterial origin, in particular SDRs deriving from the plastidial endosymbiosis. However, the major difference between plants and other eukaryotes, that explains the high number of SDRs in 'higher plants', lies in the existence of large multigenic families. These families expanded much later during evolution, as attested by their under-representation in moss and algae. Because of their involvement in secondary metabolism routes (including hormone biosynthesis), they can be considered as an adaptative character that emerged during land colonization and emergence of the vascular apparatus.

SDR initiative HMMs

Pfam HMMs

SDR type HMMs

Figure 2

 2 Figure 2 Distribution of the SDR families in the analyzed plant genomes represented as a heat map. The heat map was built on a distribution matrix deduced from the inventory classification shown in Additional file 2: TableS2. The blue to red color gradient reflects the number of SDR listed in each family in the different genomes; the absence of family is

Figure 4

 4 Figure 4 Diversification patterns of plant SDR families deduced from Principal Component Analysis. PCA was calculated on the distribution matrix shown in Figure 2. A) Scatter plot deduced from the two first components: the first and second axes respectively participate for 79 % and 9 % of the diversity. B) Contribution of different genomes (expressed as vectors) in the first and second axes values. Angiosperms genomes follow the

Figure 7

 7 Figure 7 Phylogenetic trees of three families involved in lipid primary metabolism. (A) SDR152C-FasII-β-keto-reductase (β-KR); (B): SDR87D-FasII-Enoyl-ACP-reductase (ENR);(C) SDR52E UDP-sulfoquinovose synthase (SQD1). Amino acids sequences recognized by the SDR152C, SDR87D and SDR52E HMMs were aligned with ClustalW algorithm. The evolutionary history and the bootstrap test (500 replicates) were computed as described for SDR108E (Figure3)

Figure 1 Figure 2

 12 Figure1

Figure 4 Figure 7

 47 Figure3

Table 1 Reference and size of the analyzed genomes

 1

	Species	Taxa	Annotation used	Number of	Reference
				loci
	Chlamydomonas reinhardtii Chlorophyte Chlre4.1_Augustus9	15935	[9]
	Physcomitrella patens	Moss	proteins. Phypa1_1.FilteredModels	35938	[10]
	Selaginella moellendorffii	Lycophyte	Selmo1_GeneModels_FilteredModels3	22285	[11]
	Arabidopsis thaliana	Eudicot	TAIR9	27379	[7]
	Populus trichocarpo	Eudicot	Populus.trichocarpa.v2.0	41377	[12]
	Vitis vinifera	Eudicot	12X March 2010 release	26346	[13]
	Glycine max	Eudicot	Glyma1_pacId	46367	[14]
	Oryza sativa	Monocot	MSU Rice Genome Annotation (Osa1) Release 6.1 40577	[15]
	Zea mays	Monocot	ZmB73_4a.53_working_translations	102202	[16]
	Sorghum bicolor	Monocot	Sorbi1_GeneModels_Sbi1_4_aa	34496	[17]

Table 2 Distribution of SDRs in different plants Total SDR Types of SDR Classical Divergent Extended Atypical (PLR-IFR) Unknown

 2 Families with low scores and no structural data (listed in TableS2) were omitted. *The presence of divergent SDRs in Sorghum bicolor was deduced from the Sbicolor_79_peptide annotation

	Arabidopsis	178	90	1	72	8	7
	Poplar	268	122	4	106	16	20
	Grapevine	205	95	2	88	14	6
	Soybean	315	145	4	138	15	13
	Rice	227	110	2	95	10	10
	Maize	230	97	3	113	7	10
	Sorghum	237	106	2*	114	7	8
	Selaginella	142	64	1	55	5	17
	Physcomitrella	126	59	2	55	0	10
	Chlamydomonas	68	41	1	21	1	4

Table 3 Classification of plant SDRs Representative gene SDR nomenclature initiative

 3

			Known functions	Occurence Average
				identity (%)
	AT4G23420	SDR7C	Pisum sativum Tic32 (chloroplast protein	ViridP	49,4
			import translocon)
	AT1G67730	SDR12C	β-ketoacyl reductase (fatty acids elongation) LandP	48,4
	AT3G12800	SDR17C	-	ViridP	64,1
	AT4G05530	SDR25C	SDRA-IBR1 (indole-3-butyric acid	ViridP	67,7
			response 1)
	AT3G03330	SDR34C	-	ViridP	56,1
	AT3G06060	SDR35C	-	ViridP	47,9
	AT4G09750	SDR40C	-	ViridP*	70,8
	AT1G54870	SDR57C	-	ViridP	58,0
	AT5G06060	SDR65C	Tropinone Reductase	ViridP	53,3
	AT3G03980	SDR68C		TracheoP	57,0

Acknowledgements

Hanane Moummou has received a doctoral fellowship from EGIDE within the frame of a French-Morocco Volubilis project (MA-06-155) and Libert Brice Tonfack from "Service de Coopération et d'Action Culturelle" of the French Embassy (Cameroon). The authors are grateful to Jean-Claude Pech (INP-ENSAT), Mohamed Benichou (CADI AYYAD-Marrakech University) and Emmanuel Youmbi (University of Yaoundé) for the coordination of the exchange programs and their remarks during the manuscript preparation and acknowledge Elie Maza (INP-ENSAT) and Christine Rousseau (INP-ENSAT) for their help in statistical and bioinformatic analyses. Bioinformatic analyses benefited from the Bioinfo-GenoToul facilities and those of Bioinformatics Infrastructure for Life Sciences (BILS).

Additional files provided with this submission:

Additional file 1: 6226069457362749_add1.xls, 27K http://www.biomedcentral.com/imedia/1710856356853177/supp1.xls Additional file 2: 6226069457362749_add2.xls, 478K http://www.biomedcentral.com/imedia/1953275481853177/supp2.xls Additional file 3: 6226069457362749_add3.ppt, 170K http://www.biomedcentral.com/imedia/1683065970853177/supp3.ppt

Abbreviations

AnR, Anthocyanidin reductase; A. thaliana, Arabidopsis thaliana; C. reinhardtii, Chlamydomonas reinhardtii; 4-DFR, Dihydroflavonol 4-reductase; G. max, Glycine max; HMM, Hidden Markov model; IFR, Isoflavone reductase; O. sativa, Oryza sativa; P. patens, Physcomitrella patens; P. trichocarpa, Populus trichocarpa; PLR, Pinoresinol reductase; S. bicolor, Sorghum bicolor; S. moellendorffii, Selaginella moellendorffii; SDR, Short-chain dehydrogenase/reductase; V. vinifera, Vitis vinifera; Z. mays, Zea mays.

Competing interests

The authors declare that they have no competing interests.

Author' contributions

HM participated in sequence alignments, phylogenetic analyses and collection of functional annotations. LB initiated the SDR inventory and participated in sequence alignments. BVDR conceived the study, participated in its design and coordination and drafted the manuscript. YK and BP led the HMM analyses and helped to draft the manuscript. All authors read and approved the final manuscript.

Additional files

Additional_file_1 as XLS

Additional file 1: Table S1 List of ambiguous predictions. Proteins were only recognized with one set of HMMs with a low score.

Additional_file_2 as XLS

Additional file 2: Table S2 Exhaustive inventory of SDR.

Additional_file_3 as PPT

Additional file 3: Figure S1 Full phylogenetic trees of SDR108E (A) and SDR110C (B) families. The evolutionary history was inferred using the Neighbor-Joining method. The percentage of replicate trees in which the associated taxa clustered together in the bootstrap test (500 replicates) are shown next to the branches.


Heiner Stuckenschmidt

Keywords: ontology matching, ontology alignment, evaluation, benchmarks, efficiency measure

Based on the results of the first evaluation campaign (T12.3), and taking into account the technical progress of the SEALS platform, we deliver an updated and extended evaluation and test data design for our second evaluation campaign. This campaign is planned to take place in the context of the OAEI at the ISWC 2011.

Executive Summary

The lessons learnt from the experiences gathered in the first evaluation campaign [START_REF] Trojahn | Results of the first evaluation of matching tools[END_REF] as well as the development progress of the SEALS platform allows us to improve and extend the design of the second evaluation campaign. This campaign is planned to take place in the context of OAEI, next in the summer of 2011. This deliverable reports the design of this campaign, highlighting the main extensions and novelties:

• new test data (Chapter 4): in the first campaign, only three OAEI test cases (anatomy, benchmark and conference) have been used and we plan to add more datasets from different resources, namely directory, multilingual conference and UMLS meta-thesaurus.

• automatic test generation (Chapter 4): the benchmark test case is not discriminant enough between systems and we plan to introduce a controlled automatic test generator for creating tests of increasing difficulty.

• new evaluation criterion (Chapter 3): in the first campaign, due to the lack of a controlled execution environment, the evaluation criterion was mostly conformance to reference alignments. Now that core parts of the SEALS platform have been delivered, we plan to evaluate efficiency, mainly in terms of runtime.

We also present the evaluation workflows that represent the interaction between evaluation components in an evaluation scenario (Chapter 2). We have identified three different evaluation workflows. Finally, we have selected a set of potential targets, based on maturity and availability (Chapter 5): Aflood, AgreementMaker, Aroma, Eff2Match, Falcon, Lily, Rimon, Sobom and Taxomap. We briefly report on our tests of running these tools on the express runtime version of the platform. The design of this coordinated campaign, in terms of methodology, evaluation criteria, datasets and targeted tools, was reported in [START_REF] Trojahn Dos Santos | D12.1 Evaluation Design and Collection of Test Data for Matching Tools[END_REF]. The crucial impact was the introduction of a technology for automating the evaluation process, which has affected both tool developers and campaign organizers [START_REF] Trojahn | Results of the first evaluation of matching tools[END_REF].

The progressing maturity of the SEALS platform enables us now to use an extended approach. In particular, we will switch from a web based evaluation scenario [START_REF] Trojahn | Automating oaei campaigns (first report)[END_REF] to a scenario where we execute the systems on top of the platform. This, as well as the lessons learned from OAEI 2010, has an influence on several aspects related to evaluation design, criteria to be measured, test data, and required preparation.

The deliverable is structured as follows. In Chapter 2, we present an extended version of the evaluation workflow we have designed for the first campaign, mainly in terms of new evaluation criteria and inputs for the matching process. Chapter 3 presents the evaluation criteria to be taken into account for evaluating systems. In Chapter 4, we present the evaluation test cases that we will use, highlighting potential new test cases and the automatic generator of tests. A list of potential evaluation targets and a short report on testing them is presented in Chapter 5. Finally, we conclude the deliverable in Chapter 6.

FP7 -238975

Deliverable 12.4

Evaluation Workflows

The goal of this chapter is to present the evaluation workflows that represent the sequence of activities carried out in different evaluation experiments. In [START_REF] Trojahn Dos Santos | D12.1 Evaluation Design and Collection of Test Data for Matching Tools[END_REF], we have presented a basic evaluation workflow and its extensions, in order to consider the lack of reference alignments or the intervention of the user in the evaluation loop. In this deliverable, we focus on the specific workflows that will be considered in the second campaign. First, in Section 2.1, we present the basic workflow, which has been already used in the first campaign and next (Section 2.2) we present its extensions.

Basic workflow

An evaluation workflow represents the interaction between several components in an evaluation experiment (matchers, test providers, evaluators, etc.). The basic workflow restricts the experiment to the evaluation of one matcher using a set of test cases (Figure 2.1), using conformance as evaluation criterion.

As illustrated in Figure 2.1, the first step is to retrieve, from a repository of tests, the ontologies to be matched and the corresponding reference alignments. These are the components of the test cases to be considered in such an evaluation. Next, the matcher performs the matching task, taking the two ontologies as input parameters. Then, the resulting alignment is evaluated against the reference alignment, by an evaluator. Finally, each raw result (alignment A) and its interpretations i, i.e., precision and recall, are stored into the result repository. In Appendix A, an executable BPEL workflow, that corresponds to the workflow of Figure 2.1, can be found.

Extended workflows

Due to the variability of the alignment evaluation, different scenarios can be specified, by adding new components to the process presented in Figure 2.1. We have identified two main extensions, by taking into account:

Use of an input alignment. In some test cases (e.g., subtask #4 of the OAEI anatomy track), an initial alignment is required by the matching process. Thus, three input parameters must be taken: the two ontologies and an initial alignment.

Test generator. Test cases can be generated from a description of the kind of evaluation to be executed (for example, removing n% of the properties of the ontologies). A description of the desired test case must be provided and the output of the test generation process is then used as input to the matching process. Regarding the generation of tests, Figure 2.3 illustrates a more elaborated workflow where the test cases are automatically generated by a test generator, according to some description provided by the user and based on some reference ontology. As output, the generation process creates a set of alternative ontologies, from a reference ontology, for instance, by removing its properties or individuals or adding new subclasses into the ontology hierarchy.

Criteria and Measures

In the following, we give a concise overview on the criteria that we plan to use in the context of the second evaluation campaign. Note that we gave a comprehensive overview on different criteria and metrics already in [START_REF] Trojahn Dos Santos | D12.1 Evaluation Design and Collection of Test Data for Matching Tools[END_REF]. We used most of these metrics, except effiency and scalability related metrics, already for the OAEI 2010.

Interoperability

Despite efforts on composing matchers [START_REF] Lee | etuner: tuning schema matching software using synthetic scenarios[END_REF] and on defining an Alignment API [START_REF] Euzenat | An API for ontology alignment[END_REF], ontology matching lacks interoperability benchmarks between tools. The first attempt to evaluate interoperability1 between ontology matching systems is to measure their compliance to standards such as RDF(S) and OWL. In accordance to this, we will test whether systems are able to correctly work on ontologies specified in the language standard RDF(S) and OWL.

We do not use a specific way to measure non-conformance to these standards, nor do we use a specific dataset for this purpose. Non-conformance can be detected indirectly through different criteria. For instance, the inability to identify class names in a certain language will lead to a decrease of recall. The inability to parse a certain ontology might results in throwing an error and the matching tool will eventually terminate without returning any results. We have to take care of these issues, both in the definition of the workflow (appropriate error-handling) and when we finally generate a report from the evaluation results. In particular, we will count the number of testcases that resulted in an internal error thrown by the matching system as well as the number of testcases that resulted in an empty alignment. In the latter case we have to distinguish between those cases where the matching systems failed to read the relevant information for syntactic reason and the cases where the matching system generated an empty alignment on purpose.

Compliance against a reference alignment

There are many ways to qualitatively evaluate returned results [START_REF] Do | Comparison of schema matching evaluations[END_REF]. One possibility is to propose a reference alignment R that is the one that the participants must find (sometimes also called a gold standard ). The alignment A generated by the evaluated alignment algorithm can then be compared to that reference alignment.

The most commonly used measures are precision (true positive/retrieved) and recall (true positive/expected) which have been adopted for ontology alignment. They are commonplace measures in information retrieval.

Definition 1 (Precision) Given an alignment A and a reference alignment R, the precision of A is given by

P (A, R) = |R ∩ A| |A| . FP7 -238975 Deliverable 12.4
the recall of A is given by

R(A, R) = |R ∩ A| |R| .
In addition to precision and recall, we will also use the f-measure, which is defined as the weighted harmonic mean of precision and recall. The libraries that compute precision, recall, and f-measure are already available as part of the Alignment API [START_REF] Euzenat | An API for ontology alignment[END_REF].

Other measures are available withing the Alignment API, such as relaxed and semantic versions of precision and recall as well as confidence weighted precision and recall which have been used in OAEI 2010.

Efficiency and scalability

Metrics such as execution time (speed) and amount of required memory are usually considered to measure efficiency. For the first campaign we planned to measure these aspects of a matching process, however, the platform was not yet available. We had to use our web-based evaluation service to conduct an automated evaluation campaign. In this setting, the measurement of runtimes or required memory was not possible, since all tools have been executed on different machines.

Still none of these measurements (related to both runtime and memory) are implemented in the platform. Note that these kinds of measurements cannot be implemented as work-package specific services. They belong to a generic type of functionality that is required by each workpackage in the same way.

Without evaluating the efficiency of matching systems we loose a unique selling point, especially in comparison to the web-based evaluation approach of 2010. We had many requests by tool vendors related to this functionality. In particular, the measurement of runtimes was requested. Thus, we have to find a way to implement these metrics in time for using the SEALS platform in the context of the OAEI 2011 (or 2012 at the latest).

Scalability is a metric that deals with the dependency between size of the input problem (e.g., size of the ontologies to be aligned) and the resources needed to perform the matching operation. It is clear that we can only measure scalability of a tool, if we can measure its efficiency for a specific test case. Implementing the missing functionality is thus also crucial for scalability.

Test Data for Evaluation

This chapter describes the potential datasets we plan to use in the second campaign. In the first three sections we briefly describe the datasets we already used in 2010. In the remaining sections we describe potential new datasets. For these datasets, we have not yet decided whether we can already use them for the OAEI 2011. In addition, we describe a new test data generator.

Benchmark

The systematic benchmark test set is built around one reference ontology and many variations of it. The ontologies are described in OWL-DL and serialized in the RDF/XML format. The reference ontology is that of test #101. It contains 33 named classes, 24 object properties, 40 data properties, 56 named individuals and 20 anonymous individuals. Participants have to match this reference ontology with the variations. Variations are focused on the characterization of the behavior of the tools rather than having them compete on real-life problems. They are organized in three groups:

Simple tests (1xx) such as comparing the reference ontology with itself, with another irrelevant ontology (the wine ontology used in the OWL primer) or the same ontology in its restriction to OWL-Lite; Systematic tests (2xx) obtained by discarding features from some reference ontology. It aims at evaluating how an algorithm behaves when a particular type of information is lacking. The considered features were: Four real-life ontologies of bibliographic references (3xx) found on the web and left mostly untouched (there were added xmlns and xml:base attributes).

The dataset has been used since several years in the OAEI with only minor modifications. We gave a more detailed description in Deliverable D12.1 [START_REF] Trojahn Dos Santos | D12.1 Evaluation Design and Collection of Test Data for Matching Tools[END_REF]. We supported this track in 2010. In 2011, we plan to run this track on the SEALS platform.

Anatomy

The ontologies of the anatomy track are the NCI Thesaurus describing the human anatomy, published by the National Cancer Institute (NCI) 1 , and the Adult Mouse Anatomical Dictionary2 , which has been developed as part of the Mouse Gene Expression Database project. Both resources are part of the Open Biomedical Ontologies (OBO). They are typical examples of large, carefully designed ontologies that are described in technical terms.

We gave a more detailed description of the dataset in Deliverable D12.1 [START_REF] Trojahn Dos Santos | D12.1 Evaluation Design and Collection of Test Data for Matching Tools[END_REF]. The anatomy track is the only track that makes use of an additional input alignment. While in 2010 we abstained from including this aspect within the SEALS approach, we plan in 2011 to include this in the automated evaluation supported by SEALS.

Conference

The conference dataset consists of a collection of ontologies that describe the same domain, namely the domain of conference organization. This dataset has been developed by a group of researchers from the University of Economics, Prague. Its origin is described in [START_REF] Svab | Ontofarm: Towards an experimental collection of parallel ontologies[END_REF]. The characteristics of the dataset, which consists of 15 ontologies, are listed in Table 4.1. In 2010, we already supported the evaluations related to the conference dataset. We plan to do this again in 2011.

Directory

So far we supported the evaluations related to three important OAEI datasets in 2010 and we will do this again for the OAEI 2011. The next OAEI dataset that we plan to support is the dataset of the directory track. The focus of this track is to evaluate precision and recall in a real world taxonomy integration scenario. Within this track the organizers analyze whether ontology matching tools can effectively be applied to the integration of shallow ontologies as we find them for web directories. The concrete datasets were extracted from Google, Yahoo and Looksmart web directories. The specific characteristics of the datasets are simple relationships as well as vague terminology and modeling principles. The dataset, thus, seems to be a perfect complement compared to the benchmark, conference, and anatomy datasets, which consist of (mainly) well-modeled ontologies with higher expressivity. We are currently in contact with the organizers of the directory track and discuss in how far the evaluations centered around this dataset can be supported by SEALS.

Multilingual Conference

All of our datasets are annotated with English names and labels (or technical terms from a specific domain) 3 . This neglects the fact that there are many ontologies, which are not described by / built from an English terminology. In the following we report on our attempts to create a comprehensive dataset, which deals with the important issue of multilingual ontology matching. Our approach is based on an existing monolingual dataset, which comprises both a set of different ontologies as well as reference alignments between them. We have chosen the conference dataset, namely the subset of ontologies that comes with a reference alignment. Then we translated the ontologies in different languages. In particular, we planned to generate for each ontology a Spanish, German, French, Russian, Portuguese, Czech, Dutch, and Chinese version.

As illustrated in Figure 4.1, which depicts a small subset of the dataset, this dataset allows to derive a comprehensive set of non-trivial multilingual matching tasks. In particular, we can generate for each pair of languages (9 languages, 36 pairs of languages) a reference alignment between all ordered pairs of ontologies (7 ontologies, 2 × 21 = 42 unordered pairs). Overall we have thus 1512 non-trivial multilingual matching tasks based on the high quality reference alignments of the original dataset.

The current version of the dataset is available in raw-format at http://webrum. uni-mannheim.de/math/lski/matching/trans/. We have been organizing the development of this dataset together with a group of different researchers 4 . Not all translations have been finished yet. We analyse currently in how far we can use a fraction of the dataset for the 2011 campaign. A small sample subset of the dataset is already available as a test suite in the SEALS test repository at http: //seals.sti2.at/tdrs-web/testdata/persistent/ConferenceML+Testsuite.

Linked Open Data Schemas

Currently a group of researchers from the Kno.e.sis Center at the Wright State University (USA) is working on a dataset for matching Linked Open Data (LOD) schemas [START_REF] Jain | Ontology alignment for linked open data[END_REF]. This dataset will probably be used in a new OAEI track in 2011. We have contacted the developer of the dataset and offered to support them in conducting the evaluations for this new track on top of the SEALS platform.

The objective of this dataset is to demonstrate the capabilities of matching systems with respect to effectiveness in identifying subclass and equivalence relationship between classes of LOD schemas. Reference alignments between different LOD dataset schemas have been created manually. Furthermore, the dataset contains also alignments between several LOD datasets and the upper level ontology PROTON.

Further datasets

Aside from these datasets we have identified two additional ones. However, further analysis is required to finally decide whether these datasets are well-suited. A comprehensive dataset was proposed in [START_REF] Jimenez-Ruiz | Towards a umls-based silver standard for matching biomedical ontologies[END_REF]. The authors show that the UMLS Metathesaurus [START_REF] Bodenreider | The unified medical language system (umls): integrating biomedical terminology[END_REF] (UMLS) can be used as a silver standard to align ontologies such as SNOMED CT, FMA or NCI. However, the authors also argue that UMLS contains a significant number of logic errors that have to be taken into account.

Another source of interesting datasets can be found at http://www.obofoundry. org/. This web page is hosted by the OBO foundry, which is an initiative involving developers of science-based ontologies who are establishing a set of principles for ontology development with the goal of creating a suite of orthogonal interoperable reference ontologies in the biomedical domain [START_REF] Smith | The obo foundry: coordinated evolution of ontologies to support biomedical data integration[END_REF]. Based on these efforts, a comprehensive set of correspondences and so called bridges between different important biomedical ontologies are available for download. It has to be clarified in how far these correspondences have been verified by domain experts with respect to correctness and completeness.

Test data generator

In order to generate a systematic benchmark of increasing difficulty in an automatic and parameterized way, a test generator has been designed. It allows for simulating a variety of situations and evaluating how matchers face them.

Basically, the generator receives as input an ontology and a set of parameters related to the kinds of modifications that must be applied over that ontology (see Figure 4.2). It produces as output a set of modified ontologies together with the corresponding reference alignments (i.e., alignments between the initial and modified ontologies).

The following set of input parameters has been defined, which corresponds to the modifications that can be applied over the initial ontology:

• remove classes; The use of such a systematic test generator has several benefits over previous benchmarks. First, one can generate tests from different initial ontologies. Second, the tests can be generated following variable levels of granularity.

From the general scenario, we can have different variations:

• individual test generation (displayed in Figure 4.2): from an initial ontology and a set of input parameters, a test case, i.e., the modified ontology and reference alignment, is generated; • data set generation (based on the former): from an ontology, a full data set (ontologies + reference alignment pairs) is generated. This data set can feature all variations of the parameters following a given stepwise increment or a predefined data set profile, e.g., the test profiles of the previous OAEI benchmarks.

Furthermore, the test generation can happen:

• offline: the previous benchmark data set is replaced by a newly generated dataset (this is what will be used for the next campaign);

• online: tests are generated on the fly (this requires a fine calibration of the test difficulty).

FP7 -238975

Deliverable 12.4

Evaluation Targets

Planning the first campaign (Deliverable D12.1 [START_REF] Trojahn Dos Santos | D12.1 Evaluation Design and Collection of Test Data for Matching Tools[END_REF]), we focused on five systems as potential evaluation targets. As reported in D12.3 [START_REF] Trojahn | Results of the first evaluation of matching tools[END_REF], finally 13 of 15 participants from the OAEI 2010 participated in one of the SEALS tracks. Among them three systems from the set of five systems that we originally described as potential evaluation targets. We can draw the conclusion that the first campaign was a success in terms of participation. Now we have to ensure that the same will be the case for the second campaign.

In the following, we introduce the modifications of the infrastructure that willfrom the tool developers point of view -have an impact on participating in the second campaign (Section 5.1). We report on our efforts to ensure that these changes might not result in a barrier for participation and list the systems for which we already tested that technical hurdles will not be a problem (Section 5.2).

Background and infrastructure

For the first campaign many components have not yet been available in an integrated platform. For that reason we had to find a way to offer the services required to perform the automated evaluation in a different way. A detailed description of our approach can be found in Deliverable D12.2 [START_REF] Meilicke | Services for the automatic evaluation of matching tools[END_REF] and in [START_REF] Trojahn | Automating oaei campaigns (first report)[END_REF].

The approach we used for the first campaign was based on a web service interface wrapping the functionality of a tool to be evaluated. This interface allows to execute and evaluate the tool without the need for a runtime environment. Moreover, the tool has been executed on the machine of the tool developer, while the evaluation took place within the SEALS infrastructure. In this approach each tool developer had to implement a web service to make the functionality of his tool available to the SEALS infrastructure.

For the second campaign, the SEALS platform has evolved and can probably be used for running the tool on the platform itself. This requires the tool developer to implement a different interface, i.e. to follow a completely different approach. In particular, the tool has to be packaged together with all required dependencies. The SEALS platform can then execute the packaged tool taking into account all dependencies that have been made explicit.

All of the development and debugging required for running a tool via the SEALS online service was done by the tool developer. The same is currently not the case with respect to the interface for running the tool on the SEALS platform. In particular, one needs to deploy the SEALS platform or an express version of it to perform the final test for running the tool. This is in principle possible for a tool developer, however, the hurdle for doing so is at the moment still too high.

For that reason we are currently developing a tutorial and some tools/templates to support the interface implementation and tool packaging on side of the tool developer (sometimes briefly referred to as tool wrapping). More details on this will be available in the final version of Deliverable D12.5. While developing these documents and libraries, we tested them with a comprehensive set of tools. This process has been Comment AFlood [START_REF] Seddiqui | Anchor-flood: results for OAEI 2009[END_REF] 6.1 MB unknown -AgreementMaker [START_REF] Cruz | Using agreementmaker to align ontologies for oaei[END_REF] [2] 48.4 MB yes -Falcon [START_REF] Hu | Falcon-ao: A practical ontology matching system[END_REF] 12.5 MB unknown -Lily [START_REF] Wang | Lily: The results for the ontology alignment contest oaei[END_REF] 16.9 MB yes runs only on WIN, change language options to English(USA)

Rimom [START_REF] Wang | RiMOM results for oaei 2010[END_REF] 11.2 MB yes -Sobom [START_REF] Xu | Alignment results of sobom for oaei 2010[END_REF] 12.6 MB unknown -Taxomap [START_REF] Hamdi | Taxomap alignment and refinement modules: Results for oaei 2010[END_REF] 57.3 MB yes requires additional perl installation on windows conducted partially in consultation with the developer of the specific tool. We report on our experiences briefly in the following section.

Wrapped Matching Systems

The results of our attempts to wrap some of the most important ontology matching tools are listed in Table 5.1. All of the systems listed there have been wrapped successfully. Most of them run on both Linux and Windows systems (only Lily runs on Windows only). Most systems require in addition to the included libraries some additional resources. For all of them it was sufficient to make the relevant files available from the working directory where the matching process is executed by the SEALS platform.

We have also asked some of the tool vendors whether they plan to participate in the OAEI 2011, and thus also in the second evaluation campaign. So far we have a positive answer from 5 systems. Some of the other systems we will eventually run on our own. Especially AFood and Aroma are known to be very efficient with respect to their runtime [START_REF] Euzenat | Preliminary results of the ontology alignment evaluation initiative[END_REF]. Their runtime results will thus be an important reference value.

With this set of systems we have reached a critical mass. Moreover, we used our experiences to improve documentation/material that supports tool vendors to wrap the tools on their own. During the next months we have to show that all components can be combined successfully by running a final integration test. In this test we will simulate an evaluation campaign on top of the platform using the set of matching systems in Table 5.1. Moreover, we have to finish our work on the material that supports the process of wrapping a tool. More details have to be made available in the final version of D12.5.

FP7 -238975

Deliverable 12.4

Conclusions

In this deliverable we described the current status of planning the SEALS support for OAEI 2011. The OAEI 2011 will be the second ontology matching evaluation campaign that is supported by SEALS technology. Our second campaign will thus take place earlier than it was planned originally. This decision is based on the importance and acceptance of the OAEI by the ontology matching community. We presented the approach taken for OAEI 2010 in a concise format and focused then on the modifications, extensions and differences relevant for the OAEI 2011, which are summarized in the following.

Criteria and Metrics. For the second evaluation campaign a limited set of criteria will be used that can be tested using the workflows as described in this deliverable. Criteria and measures to be considered are:

• Efficiency: runtime; • Interoperability: compliance to the standard language RDFS and OWL-DL;

• Compliance with reference alignment: precision and recall;

Datasets. The datasets were selected based on the existence of reliable reference alignments and experiences with using the datasets in evaluation campaigns. Furthermore, we list a potential set of new datasets that exploit complementary features from those in the previous datasets. However, further analysis is required to ensure, for example, the correctness and completeness of the reference alignments for some of these testsets.

Tools. For the second campaign, we have selected a subset of the systems that have been involved in past OAEI campaigns. Tools have been selected based on maturity and availability. Based on these criteria, we have identified the following tools as potential candidates to participate in the second campaign:

• AFlood, • AgreementMaker, • Aroma, • Eff2Match, • Falcon, • Lily • Rimom • Sobom, • Taxomap.
We have as well explained the technical differences between the SEALS support for OAEI 2010 and our plans for 2011. We focused on the consequences for participants.

In particular, we reported on our successful tests of running nine matching tools on the SEALS platform1 . The lessons learned from this effort will finally be summarized in a tutorial to support tool developers to deploy and execute their tools on the SEALS platform autonomous. After receiving the input parameters (receiveInput), the result composer metadata is specified (assignMetadata and addMetadaResults), the parameters for loading a test suite are initialised (setParamsToLoadTestSuite) and the test suite is loaded (loadTestSuite).

A. BPEL workflow

In ontology matching, one test suite is composed of a set of ontology pairs and their reference alignments. Each test case in the test suite contains three data items: ontology source, ontology target and reference alignment. The getOntoSource and getOntoTarget activities retrieve the URLs of the corresponding data items from the Test Data repository.

In order to invoke the tool (align), the URLs of the two ontologies are used as input parameters. The result of this invocation is the URL of the file containing the generated alignment. This URL is then sent to the results composer (addRawResultItem), which will later use this information to generate the ZIP file containing all generated raw results. The output of the tool together with the reference alignment (getRefAlign) are sent to the interpreter (interpret), which computes precision and recall. Next, the interpretations are added to the results composer bundle (addInterpretationsItem). This process is repeated for each test case. After that, ZIP files containing the raw results (createBundleRawResults) and interpretations (createBundleIntepretations) are generated by the results composer service and these bundles are submitted to the results repository (addRawResults and addInterpretations, respectively).

Particular fragments of the BPEL workflow (Figure A.1) are dedicated to fault handling. Service interfaces specify the faults that can be returned by each operation. A generic business fault, SEALSFault, has been specified for wrapping the faults that occur when executing these operations. This generic fault wraps the real faults: tool faults and unsupported platform faults. On the other hand, when implementing the tool interface, for instance, developers have to wrap all the tool exceptions into the SEALSFault.

Three ways to handle faults have been considered within the workflow. For instance, at the invoke level (align activity), it is verified if the execution of the tool raises one of the tool faults, i.e., ToolBridgeFault, related to the tool interface or ToolFault, related with the execution of the tool itself. At the scope level, the execution of activities subsequent to a tool execution fault, can be skipped. Finally, at the process level, a global fault handler is responsible for catching the SEALSFault and verifying if the nested fault is an unsupported platform fault (UnexpectedPlarformFault). In this case, the process instance ends immediately calling the callback service with an unsuccessful response.

Figure 2 . 1 :

 21 Figure 2.1: Basic evaluation workflow.

Figure 2 .

 2 Figure 2.2 illustrates the first case, where an input alignment A is taken into account. This alignment comes from the test repository together with the reference alignment and input ontologies.

Figure 2 . 2 :

 22 Figure 2.2: Extended evaluation workflow with alternative input alignment.

Figure 2 . 3 :

 23 Figure 2.3: Extended evaluation workflow with test generation.

Figure 4 . 1 :

 41 Figure 4.1: Generating a multilingual evaluation dataset

Figure 4 . 2 :

 42 Figure 4.2: Generation process.

Figure A. 1

 1 Figure A.1 illustrates the workflow for evaluating a matching tool. It iterates over a test suite and invokes a tool that generates an alignment between two ontologies. After receiving the input parameters (receiveInput), the result composer metadata is specified (assignMetadata and addMetadaResults), the parameters for loading a test suite are initialised (setParamsToLoadTestSuite) and the test suite is loaded (loadTestSuite).In ontology matching, one test suite is composed of a set of ontology pairs and their reference alignments. Each test case in the test suite contains three data items: ontology source, ontology target and reference alignment. The getOntoSource and getOntoTarget activities retrieve the URLs of the corresponding data items from the Test Data repository.In order to invoke the tool (align), the URLs of the two ontologies are used as input parameters. The result of this invocation is the URL of the file containing the generated alignment. This URL is then sent to the results composer (addRawResultItem), which will later use this information to generate the ZIP file containing all generated raw results. The output of the tool together with the reference alignment (getRefAlign) are sent to the interpreter (interpret), which computes precision and recall. Next, the interpretations are added to the results composer bundle (addInterpretationsItem). This process is repeated for each test case. After that, ZIP files containing the raw results (createBundleRawResults) and interpretations (createBundleIntepretations) are generated by the results composer service and these bundles are submitted to the results repository (addRawResults and addInterpretations, respectively).Particular fragments of the BPEL workflow (Figure A.1) are dedicated to fault handling. Service interfaces specify the faults that can be returned by each operation. A generic business fault, SEALSFault, has been specified for wrapping the faults that occur when executing these operations. This generic fault wraps the real faults: tool faults and unsupported platform faults. On the other hand, when implementing the tool interface, for instance, developers have to wrap all the tool exceptions into the SEALSFault.Three ways to handle faults have been considered within the workflow. For instance, at the invoke level (align activity), it is verified if the execution of the tool raises one of the tool faults, i.e., ToolBridgeFault, related to the tool interface or ToolFault, related with the execution of the tool itself. At the scope level, the execution of activities subsequent to a tool execution fault, can be skipped. Finally, at the process level, a global fault handler is responsible for catching the SEALSFault and verifying if the nested fault is an unsupported platform fault (UnexpectedPlarformFault). In this case, the process instance ends immediately calling the callback service with an unsuccessful response.

Figure A. 1 :

 1 Figure A.1: BPEL workflow for the evaluation of ontology matching tools.

  Basic workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Extended workflows . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

	Table of Contents 1. Introduction
	OAEI and SEALS have been coordinated since 2010 in order to design principled
	List of figures and reproducible evaluation techniques for matching tasks. The first SEALS/OAEI
	evaluation campaign of ontology matching systems has been carried out in autumn
	List of tables 2010.
	1 Introduction
	2 Evaluation Workflows
	2.1 6 Conclusions
	A BPEL workflow
	References

3 Criteria and Measures 3.1 Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Compliance against a reference alignment . . . . . . . . . . . . . . . . . 3.3 Efficiency and scalability . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Test Data for Evaluation 4.1 Benchmark . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Anatomy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Conference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Multilingual Conference . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Linked Open Data Schemas . . . . . . . . . . . . . . . . . . . . . . . . 4.7 Further datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.8 Test data generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Evaluation Targets 5.1 Background and infrastructure . . . . . . . . . . . . . . . . . . . . . . . 5.2 Wrapped Matching Systems . . . . . . . . . . . . . . . . . . . . . . . .

  Ontology Type Concepts Datatype Prop. Object Prop. Expressivity Ref

	FP7 -238975
	Deliverable 12.4
	Ekaw	Insider	77	-	33	SHIN	Yes
	Sofsem	Insider	60	18	46	ALCHIF(D) Yes
	Sigkdd	Web	49	11	17	ALEI(D) Yes
	Iasted	Web	140	3	38	ALCIN (D) Yes
	Micro	Web	32	9	17	ALCOIN (D) -
	Confious	Tool	57	5	52	SHIN (D)	-
	Pcs	Tool	23	14	24	ALCIF(D)	-
	OpenConf Tool	62	21	24	ALCOI(D)	-
	ConfTool Tool	38	23	13	SIN (D)	Yes
	Crs	Tool	14	2	15	ALCIF(D)	-
	Cmt	Tool	36	10	49	ALCIN (D) Yes
	Cocus	Tool	55	-	35	ALCIF	-
	Paperdyne Tool	47	21	61	ALCHIN (D) -
	Edas	Tool	104	20	30	ALCOIN (D) Yes
	MyReview Tool	39	17	49	ALCOIN (D) -

Table 4 .

 4

1: Ontologies of the conference test set.

Table 5 .

 5 1: Matching tools currently executable on top of the SEALS platform.

  Diverse biomedical test cases (available datasets to be analyzed),• Generated test cases (new generator to create datasets).

	• Anatomy (OAEI dataset already supported in 2010),
	• Benchmark (OAEI dataset already supported in 2010),
	• Conference (OAEI dataset already supported in 2010),
	• Directory (OAEI dataset not yet supported)
	• LOD schemas (probably new OAEI track),
	• Multilingual conference (newly generated dataset),
	•

http://knowledgeweb.semanticweb.org/benchmarking_interoperability/owl/index.html

http://www.cancer.gov/cancerinfo/terminologyresources/

http://www.informatics.jax.org/searches/AMA_form.shtml

An exceptions are two test cases from the benchmark dataset.

The following persons have been involved in the development of the dataset: Elena Montiel-Ponsoda, Raúl García Castro, Dominique Ritze, Rim Helaoui, Andrei Tamilin, Fred Freitas, Ryan Ribeiro de Azevedo, Icaro Medeiros, Fernando Lins, Eric Rommel, Roberta Fernandes, Ondrej Zamazal, Vojtech Svatek, Willem Robert van Hage, and Shenghui Wang.

We refer here to the SEALS express runtime version 1.0

.seals-project.eu/ Document URL EU Project Officer Carmela Asero Design and Collection of Test Data for Matching Tools -v2

Project Consortium Information