Jérôme Godard

Mathieu Mangeot
email: mangeot@nii.ac.jp

Frédéric Andrès
email: andres@nii.ac.jp

Data Repository Organization and Recuperation Process for Multilingual Lexical Databases 1

This paper describes the data management in multilingual lexical databases. Since NLP systems are using lexical data, the amount of work to build them is huge. That is the reason why it is important to use rigorous powerful systems and to be able to get data for a minimal cost. As an open source project, Papillon is reusing existing lexical data and wants to make volunteers collaborate. To make it possible, it combines state of the art concepts in the field of linguistic and computing.

Introduction

Many NLP systems are based on lexical data. Creating such kind of data requires an amazing knowledge and a huge amount of work. Furthermore, the existing lexical data has generally been developed for a specific purpose and can't be reused easily in other applications.

In order to overcome these shortcomings, the Papillon project applies some linguistic concepts, tools and methods to develop multipurpose, multilingual lexical data collaboratively on Internet. This data is complete and detailed enough to be finally used either by NLP systems (MT engines for example) or by human users (language learners, translators…).

After presenting the Papillon project and its purpose in section 2, we will describe the management of the data into a multilingual lexical database in section 3. Then, according to that method, we will explain in section 4 how to reuse efficiently existing dictionaries. Finally, section 5 concludes the paper.

The Papillon Project

The Papillon Project is a two-year old project born as a French-Japanese cooperation. It is now an international project building a worldwide open multilingual lexical database.

Motivations

The first motivation of the Papillon Project was to fill the gap in the domain of digital French-Japanese dictionaries and to use very powerful multilingual lexical linguistic concepts. But the real lack of multilingual lexical resources available on Internet proved the need to take other languages in consideration: a lot of free dictionaries are available but very few of them imply more than 2 languages. Moreover, most of these dictionaries include English as one of their languages. Furthermore, the existing dictionaries often lack essential information for beginners or NLP systems. Another point contributing to this lack: the high costs of development of large lexical resources for NLP involves also a high price dissuasive for the end-user.

Description of a multilingual lexical database

Two different ways are possible to develop a multilingual lexical database, either a bilingual or an interlingual approach. The most interesting one is quite obviously the interlingual one because it allows to build only once one dictionary for each language (n languages implies n monolingual dictionaries and one interlingual database); in the case of the bilingual method, it is necessary to create a dictionary for each couple (n languages implies

        ∑ - = 1 1 n i i , i.e.
( )

2 1 - n n
dictionaries). But the interlingual approach requires a strong methodology to become effective [START_REF] Godard | Production of Multilingual Lexical Databases, 1 st workshop on Mlabnet[END_REF].

Some partners of the Papillon project were involved in researches on the definition of structures and tools to handle multilingual lexical databases. They were looking for an opportunity to apply these researches on real scale lexical data.

Structure of the entries

Papillon entries monolingual structure is taken from the DiCo database, based on the research of Igor A. Mel'cuk and Alain Polguère, Université de Montréal, Montréal, Canada (Alain [START_REF] Polguère | Towards a theoretically-motivated general public dictionary of semantic derivations and collocations for French[END_REF] and encoded in XML. Dictionaries' entries are semantic-dependent; indeed, each meaning of one word (vocable) represents an entry. These semantical entities are called lexies. They are used to build monolingual dictionaries and to give those databases a lexical approach. Then, the lexies of various monolingual dictionaries need to be linked. This is done via a pivot architecture that connects the lexies ones with the others; this multilingual architecture is based on the relation between the lexies (Gilles [START_REF] Sérasset | Interlingual Lexical Organization for Multilingual Lexical Databases in NADIA[END_REF] implemented with acceptions called axies.

Collaborative Development over Internet

Most partners were participating, as computer scientists, to the development of open source products. With the democratisation of Internet access in a lot of countries, came the opportunity to apply the open source principles to the development of a multipurpose, multilingual lexical database. Cooperation projects for bilingual dictionaries are already going on such as EDICT, a Japanese-English dictionary lead by Jim [START_REF] Breen | A WWW Dictionary and Word Translator: Threat or Aid to Language Acquisition?[END_REF] for more than 10 years and more recently, SAIKAM, a Japanese-Thai dictionary [START_REF] Vutichai Ampornaramveth | An Internet-Based Collaborative Dictionary Development Project: SAIKAM[END_REF]. Some cooperations have also been launched in order to add Thai (Lexibase), Lao and Vietnamese.

With the Papillon project, the dictionary is extended to a multilingual lexical database. Volunteers will find lexicons developed by others and some tools to complete or correct the Papillon multilingual dictionary. Users will also be able to define their own personal views of the database.

Lexical Data Management

Issues

From our work in the field of lexical data recuperation process, we got some fundamentals principles about storing and manipulating digital dictionaries in order to build a multilingual lexical database.

They are mainly due to the fact that many dictionaries are used. Moreover, those dictionaries evolve from the modification we bring them to make the data relevant to our XML data schema.

There are also many people within the project who have to be able to access quickly and easily the content they are looking for; people who are also evolving, it means some new contributors are continually taking part in the project.

Therefore, we decided to use a strong data repository definition with efficient and effective naming rules.

Data Repository 2

The lexical data repository of the Papillon project is divided into 4 subdirectories (Mathieu Mangeot-Lerebours 2001):

• Administration contains guidelines and administrative files

• Hell (data in original format) • Purgatory (data in XML & UTF-8) • Paradise (data in Papillon format)
An example of that structure is given in the following figure: An illustration of our goal would be to say that any contributor should be able to understand the data organization after typing "ls".

Hell Directory

This directory contains lexical data in their original format. When a dictionary is received, it is first stored there while waiting for an eventual recuperation process. For each dictionary, a metadata file containing all the information available on the dictionary (name, languages covered, creation date, size, authors, domain, etc.) is written. It is then used to evaluate the quality of the dictionary and to begin the recuperation process. Internet users only have access to these dictionaries as whole downloadable files.

Purgatory Directory

The Purgatory directory receives the lexical data once the recuperation process is over. This process consists in converting the lexical data from its original format into XML encoded in UTF-8. To perform this task, we use the RECUPDIC methodology described by Haï Doan-Nguyen (1998) with regular expression tools like Perl scripts.

If a dictionary is already encoded in XML, the recuperation process consists in mapping the elements of information into CDM (Common Dictionary Markup) elements and storing the correspondence into the metadata file.

With this information available, Internet users have access to these dictionaries as classical online dictionaries, retrieving individual entries by way of requests on the Papillon web site.

Paradise Directory

The paradise directory contains only one dictionary often called the "Papillon dictionary".

This dictionary has a particular DML (Dictionary Markup Language) structure [START_REF] Mangeot-Lerebours | Environnements centralisés et distribués pour lexicographes et lexicologues en contexte multilingue[END_REF]. Internet users have access to individual entries of this dictionary through requests to the Papillon web site. They also can add new entries or correct existing ones online.

Other purgatory dictionaries may be integrated into the Papillon dictionary with the help of the CDM elements.

Data Identification

The names of the files and directories are normalised in order to be able to know exactly what a file contains (with an increasing number of files and files' versions, it becomes quickly complicated to remember what makes a version relevant) and to allow people to navigate easily into the repository.

• Directories Each dictionary is installed in a directory which name is built as follow, DirName_la1-la2-la3/, where:

DirName is the dictionary nickname (eg: FeM). The first letter is a capital. The remainder of the name can be either capital or small letters but always ASCII.

la1, la2, la3, etc. are the languages present in the dictionary either source or target language. The languages have to be alphabetically ordered. The 3 letter language codes are used for the name of the languages. They are written in small letters eg: FeM_eng-fra-msa/;

• Files In the directories, each name of the file containing lexical data is written as follow: dictName_sla_tll_tl2-encoding-version.ext where:

dictName is the nickname of the dictionary in small letters (eg: fem). sla is the source language of the volume in 3 letter language code and small letters.

tl1, tl2 are the target languages of the volume written in alphabetical order in 3 letter language code and small letters.

encoding is the encoding type of the file (ISO-8859-[1-9], ASCII, UTF-8, SJIS, etc). If the file is encoded in XML, it is not necessary to indicate it. Furthermore, the default encoding is ISO-8859-1. Mime type encoding names are used 3 .

version is the version number of the file. This number is optional.

ext is the file extension (txt, xml, html, etc.). For example, the version 1 of a volume of the dictionary FeM with French as source language, English and Malay as target language, encoded in mac roman in text format would be named: FeM_en-fr-ml/fem_en_fr-macroman-v1.txt;

• Languages codes We use the 3 letter language codes of the standard ISO 639/2 T (eg: eng for English, fra for French, jpn for Japanese, etc). We also added new language codes for our own purpose: unl for Universal Network Language and axie for Interlingual Acception. A complete list of 3 letter language codes is available online at the same URL.

Data Structure

This major innovative structure of the lexical data repository presented in the previous part can be used for any kind of dictionary. Indeed, the precise classification into a logical organization allows building reliable and coherent multilingual lexical databases.

In order to handle heterogeneous structures with the same tools, we defined a subset of DML elements and attributes that are used to identify which part of the different structures represent the same lexical information. This is CDM. This set is in constant evolution. If the same kind of information is found in several dictionaries then a new element representing this piece of information is added to the CDM set. It allows tools to have access to common information in heterogeneous dictionaries by using pointers into the structures of the dictionaries.

Preamble

The aim of Papillon as an open source project is of course to make users (who are able to) become contributors. But as mentioned before, the construction of good quality lexical data requires an incredible amount of work.

Since many digital dictionaries already exist, mono-or bilinguals ones, it is obvious that reusing them is very interesting. Unfortunately, their structure does not allow adding easily new languages.

The part tries to describe generic methods to successfully and efficiently manage data recuperation processes.

Problems

It became almost immediately obvious that the recuperation process had to be done with a very strong methodology. In fact, each file requires its own treatment. In spite of that, the various operations that are described below are valid for each dictionary but the way to apply them depends on each file. That is why the scripts that are written to modify the data structure must be respectful of the procedure and well documented. That also allows to other people to reuse and to adapt scripts in order to facilitate and to improve their work on other files. Another important issue is to be able to remake well-structured files from original ones in case of bad manipulation.

Methodology

A recuperation process consists in recuperating lexical data from original format to XML. It also consists in moving the new files from Hell directory to Purgatory directory.

This process can be very complicated. It was very deeply analyzed in Hai Doan Nguyen's Ph.D. Thesis. There is an existing tool called Hgrammar implemented in LISP. It is a powerful compiler that recuperates lexical data. However, errors remain and the code is not fully valid.

It is possible to recuperate existing lexical data with a language implementing regular expressions. We frequently use perl for this purpose. For each main step, a perl script is used. They are named like this: StepNumber-StepName.pl. StepNumber is a number on two digits. The first step is 01.

StepName is a brief explanation of the step. For example, the first Perl script frequently used is 01-convxmlchars.pl. The order of the steps is important because it can be used afterwards to re-recuperate a dictionary or to modify the recuperation process.

If the character encoding of the file is not UTF-8, it can be converted using UNIX/LINUX tools like uniconv.

During the recuperation process, one important rule has to be observed; people manipulating lexical data in order to modify their structure need to keep the following postulate in mind: always keep all the information present in the original file even if it does not seem useful at a first glance (eg typographical marks, etc).

Finally, when the recuperation process is done, it is vital to ensure the update on the database in a proper way; what we use and recommend to use is cvs with the following procedure: make a cvs add and cvs commit for the new files.

Case study

We already worked on various existing dictionaries we were allowed to use. Most of the time, those data are contained into text type files, sometimes with several encoding in one file. We describe here guidelines of the recuperation process that are valid for any kind of text file. The aim of that work is to build a generic methodology; then, almost all the operations can be automatically solved with perl scripts4 .

First two steps have to be done if the file is a ".doc" one:

-conversion of the XML specific characters: « < » by « < » « > » by « > » « & » by « & »; -bold and italic characters: The search of those characters can be easily done using wildcards. Then, they have to be inserted between « <b> » and « </b> » if bold « <i> » and « </i> » if bold; then the .doc file must be saved as a .txt file; -clean the overlapping of the tags <b> and <i>; -replace the characters that present an encoding problem. It is important to point out that this step requires language knowledge. In fact, only people knowing French are able to see if diacritical characters are well represented or not. Then, since a wrong character is identified, it can be automatically replaced by the correct UTF8-encoded character in the whole file. It is important to say that UTF8 does not allow to directly represent all characters. We had problem with a Vietnamese character. In that kind of case, it is possible to build the character with a combination by using existing character codes5 ; -delete empty lines; -add the header and footer that contain relevant information about the file; -add the XML tags according to the schema that defines the data structure. This step is quite critical. In fact, most of time or more precisely almost always, the lexical structure of the dictionaries is far from the Papillon schema. We copy the fields that are coherent with the Papillon schema and copy others into "more-info" tags; -delete spaces besides tags; -save the file as ".xml"; -Here appears a step that needs some observation from the people running those operations. Indeed, the dictionaries always contain some structural errors (even according to there own structure) that create some parsing errors. That is why it is very important to parse the file to check what kinds of error occur. Those errors are most of the time due to missing fields in the original file. The parsing operation can be processed with XML Spy or Netscape 6. The correction of those errors can automatically be done but it implies to understand what causes the errors.

Then, an XML file should be added into the Purgatory directory…

Conclusion

According to its motivations, goals and methodology, the Papillon project is acting in the state of the art within the domain of multilingual lexical database.

As the project progresses, it is confronted with many kinds of problems related to linguistic and computing. Of course, most of those problems are due to NLP complexity. This is why the Papillon project represents a very interesting platform in order to experiment NLP researches in optimal conditions.

We must point out that the benefits of that project will go to the community that will contribute and use it. One very important point in the close future will be to make it attractive to its recipients.

Figure 1 .

 1 Figure 1. Papillon Data Repository.

  3 http://www-clips.imag.fr/geta/services/dml/dml.xsd 4 Recuperation Process

NB all the lexical data stored in the repository are free of rights or protected by a GPL-like licence.

NB as said before, the steps order must be respected; mixing the steps could modify the lexical content.

http://www.unicode.org/unicode/faq/char_combmar k.html

Acknowledgement

We would like to thank NII, GETA, and Kasetsart University/NAIST for their support and help in the Papillon project.

1 This research has been partially supported by National Institute of Informatics (NII) under the leadership framework, by the Multimedia Annotation Consortium, and by the JSPS


Mathieu Mangeot-Lerebours
email: mangeot@nii.ac.jp

Proposal Changes for the Papillon Monolingual XML Schema

The Papillon project aims at building a multilingual lexical database for extracting dictionaries. This paper describes the Papillon monolingual lexie structure with an example and propose some changes for spotted problems.

Introduction

The Papillon XML entries structure are described with XML schemata.

The basic schema is the DML schema. This schema describe general elements and attributes used to encode lexical databases in XML. It allows one to encode users, groups, APIs, dictionaries, modifications histories and entries. The DML is an XML representation and extension of the SUBLIM languages from Gilles Serasset [START_REF] Gilles | SUBLIM: un Syst me Universel de Bases Lexicales Multilingues et NADIA: sa sp cialisation aux bases lexicales interlingues par acceptions[END_REF].

The Papillon schema redefines the DML entry element with the Papillon lexies and axies structures. It then contains the monolingual eixe structure common to all languages.

The language specific shemata define language specific elements like the list of partsof-speech or the language levels.

The DML and Papillon XML schemata and XML examples are available online at http://www-clips.imag.fr/geta/services/dml/dml.xsd.

To write an XML entry, you can use any text editor. Nevertheless, we recommend to use an XML editor like XML Spy to avoid XML syntax errors. XML Spy for PC available at http://www.xmlspy.com for a 30 days free evaluation version.

The best tool is an XML schema validating editor. The editor will check online the validity of your documents thanks to the URL indicated at the root element of the XML documents with the xsi:schemaLocation XML attribute :

Detailed Example of a French Lexie

A lexie is an entry of a Papillon monolingual dictionary. The structure of the entries, that is microstructure of the monolingual dictionaries, is based on the structure used for the formal lexical database DiCo of the OLST laboratory at Universite de Montreal.

The encoding methodology is directly borrowed from the explanatory and combinatorial lexicology, which is part of the meaning-text theory elaborated by Igor Melc'uk and his colleagues.

The attribute id of the lexie element is an internal unique id. It is hidden from the users. If the lexie is deleted, its id remains. It can't be reused. It is metalinguistic information. The id is built by adding a . to the headword and then a unique integer.

The attribute basic is a boolean. It indicates if this lexie is the basic lexical unit of the vocable. The information isd taken from the DiCo database: if one vocable has more than one lexie, the basic lexie ie basic= true is a hidden lexie containing all the common elements of the vocable.

< lexie d:id ="meurtre.1" basic ="true">

The headword element is a common DML element declared in the DML schema. It represents the headword of the lexie. The hn attribute is the homograph number of the headword. It is not necessary linked to the lexie id.

< headword hn ="1">meurtre</ headword >

The reading element is not useful for languages that not use kanjis. It is used in Japanese to write the headword in kanas.

< reading >meurtre</ reading >

The pronunciation element is used to indicate the pronunciation of the headword. As several encodings can be used (GETA simplified encoding for French, romaji for Japanese, IPA, etc), the encding must be indicated.

< pronunciation encoding ="GETA">meu+rtr(e)</pronunciation>

The part-of-speech element is used to indicate the part-of-speech of the lexie. The list of parts-of-speech for one language has to be described in language-specific schemata.

<pos>n.m.</pos>

The language-levels element is used to indicate the various language levels of the lexie. The list of language levels for one language has to be described in language-specific schemata.

<language-levels> <politeness/> <usage/> <reference/> </language-levels> The semantic formula comes from the meaning-text theory. It is a formal definition of the lexie. It begins with a semantic label which list should be closed and properly defined for each language. Then, if the lexie is a predicate, it indicates the composition of the arguments of the predicate.

<semantic-formula>

The element sem-label tags the semantic label of the formula.

<sem-label>action de tuer</sem-label>: ~ PAR L'

The element actor tags an actor of the semantic formula. The element sem-variable tags a semantic variable of the formula.

<sem-variable>X</sem-variable> </actor> DE L'<actor> <sem-label>individu</sem-label> <sem-variable>Y</sem-variable> </actor> </semantic-formula>

The government pattern comes also from the meaning-text theory. It is the syntactical realization of the arguments of the predicate defined in the semantic formula.

<government-pattern>

There might be more than one government pattern (called "modifications") for the same lexical unit. The nb attribute of the mod elements encode this. Lists some full idioms containing the headword of the lexie.

<full-idioms> <idiom d:id="i1" xlink:href="papillon-axi.xml#axii0004">_appel au meurtre_</idiom> </full-idioms>

The <more-info> element is used to store additional XML information for the lexie. It contains typically, information from recuperated resources. The content of the element is skipped for the validation. It just need to be valid XML.

<more-info> <fem>information taken from the French-English-Malay dictionary</fem> </more-info>

The <axies> element encode links between a lexie and various axies.

String encodings

Problem: For the moment, string litterals are encoded as simple strings. But when the user does not know how to read the language, s/he cannot read the strings. For example, it is ver difficult for someone not used to kanjis to read Japanese examples.

Proposal: All strings should be encoded as a text object with string+pronunciation. For Japanese, do we encode both romaji & yomigana ? If we encode all strings with objects, we can delete the <pronunciation> element for the headword.

The Reading element

Problem: The reading element is indicated only for the headword. For Japanese, do we add a reading element for all the strings? Proposal: Leave it as is.

The Id attributes

Problem: The Id attributes and headword are written in capital letters in DiCo database. It has to be discussed if it is necessary to write them in capital letters knowing that it does not exist in Japanese.

Proposal: Encode the id attribute and the headword in small letters for roman scripts.

The basic attribute

< lexie d:id ="meurtre.1" basic ="true"> Problem: This basic was used to indicate that the lexie is the basic lexie of the vocable. This lexie is not a real one, it is used to store all the common information of the vocable to avoid duplication of the info. It had sense in the DiCo database but not in the Papillon dictionary because we duplicate the information.

Alternatives: 1. put of 2. implement a vocable object for the common informations of the vocables Proposal: 1

The hn attribute

Problem: It is not the homograph number attribute, it is the number of the lexie ! Proposal: Call it lexienb.

The pos element

Problem: Now: conflict with a simple <pos> element and a <gram> element with a <pos> inside + more info. How to convert the DiCo Database ? Alternatives: 1. map between DiCo pos and FeM pos, and put the other gram info into the other-info tag 2. enclose the <pos> tag into a gram tag and put the other info in the gram tag too.

Proposal: 1

Axie links

Problem: Now there are double links. Ones from lexies to axies and ones from axies to lexies. It is difficult to maintain the coherence.

Proposal: Keep only the links from axies to lexies and manage all the links from the axies dictionary. The Papillon application is already implemented to take this into account.

  <actor> <sem-label>individu</sem-label>

  <headword><text><string> 殺人 </ string >< pron >satsujin</ pron ></ text > </ headword >

href="#assassinat.1">assassinat</reflexie>

  This lists the lexical functions and their results applicable to the lexie headword.The lftype attribute is used to indicate if the sense of the lexical function is included in the lexical function lftype='merged' or not. It is equivalent to the // sign in the DiCo format.Lists some usage examples of the headword of the lexie.

	<valgroup lftype="merged">
	<mod nb="1"> <value>
	<actor> <reflexie xlink:href="#meurtrier.2">meurtrier-n</reflexie>
	<sem-actant>X</sem-actant> = </value>
	<synt-actant>I</synt-actant> = </valgroup>
	<surface-group> </function>
	<surface> </lexical-functions>
	<reflexie xlink:href="#de">de</reflexie>
	N</surface>,
	<surface>A-poss</surface> <examples> </surface-group> <example d:id="e1">C'est ici que le double meurtre a été </actor> commis.</example> <actor> <sem-actant>Y</sem-actant> = </examples>
	<synt-actant>II</synt-actant> =
	<surface-group>
	<surface>
	<reflexie xlink:href="#de">de</reflexie> N
	</surface>,
	<surface>A-poss</surface>
	</surface-group>
	</actor>
	</mod>
	</government-pattern>
	<lexical-functions>
	<function name="Qsyn">
	<valgroup>
	<value>
	Reflexie reference to another lexie with an xlink href attribute. It is used to build a
	monolingual network.
	</value>
	</valgroup>
	</function>
	<function name="S1">
	<valgroup>
	<comment> Nom pour X </comment>
	<value>
	<reflexie xlink:href="#auteur.1">auteur</reflexie> [de ART
	~]</value>
	</valgroup>//

<reflexie xlink:

Conclusion

If everybody agree on te changes, we declare this structure as the version 1.0 of the Papillon common monolingual structure.