Nancy Ide
email: ide@cs.vassar.edu

Laurent Romary
email: romary@loria.fr

Standards for Language Resources

This paper presents an abstract data model for linguistic annotations and its implementation using XML, RDF and related standards; and to outline the work of a newly formed committee of the International Standards Organization (ISO), ISO/TC 37/SC 4 Language Resource Management, which will use this work as its starting point. The primary motive for presenting the latter is to solicit the participation of members of the research community to contribute to the work of the committee.

Introduction

The goal of this paper is two-fold: to present an abstract data model for linguistic annotations and its implementation using XML, RDF and related standards; and to outline the work of a newly formed committee of the International Standards Organization (ISO), ISO/TC 37/SC 4 Language Resource Management, which will use this work as its starting point. The primary motive for presenting the latter is to solicit the participation of members of the research community to contribute to the work of the committee.

The objective of ISO/TC 37/SC 4 is to prepare international standards and guidelines for effective language resource management in applications in the multilingual information society. To this end, the committee will develop principles and methods for creating, coding, processing and managing language resources, such as written corpora, lexical corpora, speech corpora, dictionary compiling and classification schemes. The focus of the work is on data modeling, markup, data exchange and the evaluation of language resources other than terminologies (which have already been treated in ISO/TC 37). The worldwide use of ISO/TC 37/SC 4 standards should improve information management within industrial, technical and scientific environments, and increase efficiency in computer-supported language communication.

Motivation

The standardization of principles and methods for the collection, processing and presentation of language resources requires a distinct type of activity. Basic standards must be produced with wide-ranging applications in view. In the area of language resources, these standards should provide various technical committees of ISO, IEC and other standardizing bodies with the groundwork for building more precise standards for language resource management.

The need for harmonization of representation formats for different kinds of linguistic information is critical, as resources and information are more and more frequently merged, compared, or otherwise utilized in common systems. This is perhaps most obvious for processing multi-modal information, which must support the fusion of multimodal inputs and represent the combined and integrated contributions of different types of input (e.g., a spoken utterance combined with gesture and facial expression), and enable multimodal output (see, for example, [START_REF] Bunt | Towards Multimodal Content Representation[END_REF]. However, language processing applications of any kind require the integration of varieties of linguistic information, which, in today's environment, come from potentially diverse sources. We can therefore expect use and integration of, for example, syntactic, morphological, discourse, etc. information for multiple languages, as well as information structures like domain models and ontologies.

We are aware that standardization is a difficult business, and that many members of the targeted communities are skeptical about imposing any sort of standards at all. There are two major arguments against the idea of standardization for language resources. First, the diversity of theoretical approaches to, in particular, the annotation of various linguistic phenomena suggests that standardization is at least impractical, if not impossible. Second, it is feared that vast amounts of existing data and processing software, which may have taken years of effort and considerable funding to develop, will be rendered obsolete by the acceptance of new standards by the community. To answer both of these concerns, we stress that the efforts of the committee are geared toward defining abstract models and general frameworks for creation and representation of language resources, rather than specific formats. These models should, in principle, be abstract enough to accommodate diverse theoretical approaches. The model so far developed in ISO TC/37 for terminology, which has informed and been informed by work on representation schemes for dictionaries and other lexical data [START_REF] Ide | A Formal Model of Dictionary Structure and Content[END_REF] and syntactic annotation [START_REF] Ide | Standards for Language Resources, IRCS Workshop on Linguistic Databases[END_REF] demonstrates that this is not an unrealizable goal. Also, by situating all of the standards development squarely in the framework of XML and related standards such as RDF, DAML+OIL, etc., we hope to ensure not only that the standards developed by the committee provide for compatibility with established and widely accepted web-based technologies, but also that transduction from legacy formats into XML formats conformant to the new standards is feasible.

ISO/TC 37/SC 4 will liaison with ISLE (International Standards for Language Engineering), which has implemented various recent efforts to integrate EC and US efforts for language resources. Where possible, these and other standards set up in EAGLES will be incorporated into the ISO standards. ISO/TC 37/SC 4 will also broaden the work of EAGLES/ISLE by including languages (e.g. Asian languages) that are not currently covered by EAGLES/ISLE standards.

At present, language professionals and standardization experts are not sufficiently aware of the standardization efforts being undertaken by ISO/TC 37/SC 4. Promoting awareness of future activities and rising problems, therefore, will be a crucial factor in the success of the committee, and will be required to ensure widespread adoption of the standards it develops. An even more critical factor for the success of the committee's work is to involve, from the outset, as many and as broad a range of potential users of the standards as possible. This presentation serves as a call for participation to the linguistics and computational linguistics research communities.

Objectives

ISO TC37/SC 4's goal is to develop a platform for the design and implementation of linguistic resource formats and processes in order to facilitate the exchange of information between language processing modules. This will be accomplished by defining a common interface format capable of representing multiple kinds of linguistic information. The interface format must support the communication among all modules in the system, and be adequate for representing not only the end result of interpretation, but also intermediate results.

A well-defined representational framework for linguistic information will also provide for the specification and comparison of existing applicationspecific representations and the definition of new ones, while ensuring a level of interoperability between them.

Requirements

Very generally, a linguistic representation framework must meet the following requirements:

• Expressive adequacy: the framework should be expressive enough to represent all varieties of linguistic information;

• Semantic adequacy: the representation structures should have a formal semantics, i.e., their definition should provide a rigorous basis for further processing (e.g., deductive reasoning, statistical analysis, generation, etc.).

Providing interface formats within a system architecture demands that "incremental" construction of intermediate and partial representations be supported. In addition, if the construction of a final representation does not succeed, the representation must capture the information required to enable appropriate system action. This dictates additional requirements:

• Incrementality: support for the various stages of input interpretation and output generation, allowing both early and late fusion and fission.

• Uniformity: the representation of various types of input and output should utilize the same "building blocks" and the same methods for combining complex structures composed of these building blocks.

• Underspecification and partiality: support for the representation of partial and intermediate results, including the capture of unresolved ambiguities.

Finally, the representational framework must be accommodate the developing field of language processing system design by satisfying these further requirements:

• Openness: the framework should not depend on a single linguistic theory, but should enable representations based on different theories and approaches;

• Extensibilty. The framework should be compatible with alternative methods for designing representation schemas (e.g., XML) rather than being tied to a specific schema.

Methodology

A working group of SC 4 (WG1/WI-1) has been charged with the task of defining a linguistic annotation framework, which will be used by other SC 4 working groups to develop more precise specifications for particular annotation types. The full list of SC 4 working groups is as follows: We focus here on the work of WG1/WI-1, which will serve as the starting point for that of most of the others. This group will propose a data architecture consisting of basic mechanisms and data structures for linguistic annotation and representation, comprised of the following:

•
• Basic components: the basic constructs for building representations of linguistic information; specifically, identification of types of building blocks and ways to connect them. • General mechanisms: representation techniques that make the annotations more compact and flexible and enable linking them to external sources of information; for example, sub-structure labeling, argument under-specification, restrictions on label values and/or disjunctions or lists to represent ambiguity or partiality, structure sharing; linking to domain models, linking to other levels of annotation, etc. • Contextual data categories: administrative (meta-) data relevant for processing, such as environment data (e.g., time stamps, spatial information); processing information (e.g., module that produced the representation; confidence level); interaction information (speaker, audience, etc.).

The following section outlines a linguistic framework which will serve as the starting point for development within SC 4. The current model is based on work on development of annotation formats for lexicons [START_REF] Ide | Standards for Language Resources, IRCS Workshop on Linguistic Databases[END_REF], morphosyntactic and syntactic annotation (Ide & Romary, 2001a;[START_REF] Ide | A Common Framework for Syntactic Annotation[END_REF]Ide & Romary, forthcoming), and which has been further developed within TC37/SC4 for the definition of TMF (Terminological Markup Framework; ISO 16642, under DIS ballot).

A Framework for Linguistic Annotation

Our fundamental assumption is that representation formats for linguistic data and its annotations can be modeled by combining a structural meta-model, that is, an abstract structure shared by all documents of a given type (e.g. syntactic annotation), with a set of data categories that are associated with the various components of the meta-model. Our work in SC4 is concerned, first, with identification of a reduced set of meta-models that can be used for any type of linguistic data and its annotations. Data categories, on the other hand, are defined by the implementer; interoperability among formats is ensured by providing a Data Category Registry in which the categories and relations required for a particular type of annotation are precisely defined.

The model for linguistic annotation must satisfy two general criteria: 1. It must be possible to instantiate it using a standard representational format; 2. It must be designed so as to serve as a pivot format into and out of which proprietary formats can be transduced, in order to enable comparison and merging, as well as operation on the data by common tools.

Abstract model for annotation

At its highest level of abstraction, an annotation is a set of data or information (in our case, linguistic information) that is associated with some other data. The latter is what could be called "primary" data (e.g., a part of a text or speech signal, etc.), but this need not be the case; consider, for example, the alignment of parallel translations, where the "annotation" is a link between two primary data objects (the aligned texts). Typically, primary data objects are represented by "locations" in an electronic file, for example, the span of characters comprising a sentence or word, or a point at which a given temporal event begins or ends (as in speech annotation). As such, at the base primary data objects are relatively simple in their structure; more complex data objects may consist of a list or set of contiguous or non-contiguous locations. Annotation objects, on the other hand, often have a more complex internal structure: syntactic annotation, for example, may be expressed as a tree structure, and may include more elemental annotations such as dependency relations (which is itself an annotation relating two objects, where the relation is directional (dependent-to-head)).

Thus, we can conceive of an annotation as a one-or two-way link between an annotation object and a point (or a list/set of points) or span (or a list/set of spans) within a base data set. Links may or may not have a semantics-i.e., a type--associated with them. Points and spans in the base data may themselves be objects, or sets or lists of objects. This abstract formulation can serve as the basis for defining a general model for linguistic annotation that can be realized in a standard representational format. In fact, this model is consistent with well-established data modeling concepts used in diverse areas, including knowledge representation (KR), object-oriented design, and database systems, and which inform fundamental data structures in computer science (trees, graphs, etc.) and database design (notably, the Entity-Relationship (ER) model). As such, the model provides us with established means to describe our data objects (in terms of composition, attributes, class membership, applicable procedures, etc.) and relations among them, independent of their instantiation in any particular form. It also ensures that standardized representation formats exist that can instantiate the model.

One way to represent linguistic annotation in terms of the abstract model is as a graph of elementary structural nodes to which one or more information units are attached. The distinction between the structure of annotations and the informational units of which it is comprised is, we feel, critical to the design of a truly general model for annotations. Annotations may be structured in several ways; perhaps the most common structure is hierarchical. For example, phrase structure analyses of syntax are structured as trees; in addition, hierarchy is often used to break annotation information into sub-components, as in the case of lexical and terminological information.

There are several special relations among annotations that must be represented in the model, including the following:

• Parallelism: two or more annotations refer to the same data object;

• Alternatives: two or more annotations comprise a set of mutually exclusive alternatives (e.g., two possible part-of-speech assignments, before disambiguation);

• Aggregation: two or more annotations comprise a list or set that should be taken as a unit.

Information units or data categories provide the semantics of the annotation. Data categories are the most theory and application-specific part of an annotation scheme. We do not attempt to define the relevant data categories for given types of annotation. Rather, we propose the development of a Data Category Registry to provide a framework in which the research community can formally define data categories for reference and use in annotation. To make them maximally interoperable and consistent with existing standards, data categories can be defined using RDF schemas to formalize the properties and relations associated with each. Note that RDF descriptions function much like class definitions in an object-oriented programming language: they provide, effectively, templates that describe how objects may be instantiated, but do not constitute the objects themselves. Thus, in a document containing an actual annotation, several objects with the same type may be instantiated, each with a different value. The RDF schema ensures that each instantiation is recognized as a sub-class of more general classes and inherits the appropriate properties.

A formally defined set of categories will have several functions: (1) it will provide a precise semantics for annotation categories that can be either used "off the shelf" by annotators or modified to serve specific needs;

(2) it will provide a set of reference categories onto which scheme-specific names can be mapped; and (3) it will provide a point of departure for definition of variant or more precise categories. Thus the overall goal of the Data Category Registry is not to impose a specific set of categories, but rather to ensure that the semantics of data categories included in annotations (whether they exist in the Registry or not) are well-defined and understood.

An Example

We illustrate a simple application of the framework presented above for the domain of morpho-syntactic annotation. For the purposes of illustration, it is necessary to make technical choices concerning the representation format. XML and related standards developed by the World Wide Web consortium appear at present to provide the best means to represent information structures intended to be transmitted across a network. For the purposes of linguistic resource representation, XML provides several important features:

• it is both Unicode and ISO 10646 compatible;

• XML namespaces provide the options of combining element definitions from multiple sources in an XML document, thereby fostering modularity and reuse;

• XML schemas provide a powerful means to define, constrain, and extend definitions of the structure and contents of classes of XML documents and document sub-parts;

• W3C has defined accompanying standards for interand intra-document linkage (XPath, XPointer, and Xlink) as well as document traversal and transformation (XSLT);

• XML is fully integrated with emerging standards such as the Resource Definition Framework (RDF) and DAML+OIL, which can be "layered" on top of XML documents to provide a formal semantics defining XML-instantiated objects and relations.

We have defined an XML format for representing linguistic annotations called the Generic Mapping Tool (GMT). The GMT defines XML elements for encoding annotation structure (primarily, a nestable <struct> element) and data categories (a nestable <feat> tag). A <seg> element provides a pointer to the annotated data using XPointers. Relations among objects can be specified explicitly using a <rel> element or may be implicit in the hierarchical nesting of <struct> elements. The GMT is described in detail in [START_REF] Ide | A Common Framework for Syntactic Annotation[END_REF]. We stress, however, that the details of the XML format-in particular, element names-is arbitrary; the only requirement is that the underlying data model can be expressed using the format.

Morpho-syntactic annotation

Morpho-syntactic annotation provides a good example of how the data model instantiated in the GMT is applied, and demonstrates some of the mechanisms required for representing annotations in general. Morpho-syntactic annotation involves the identification of word classes over a continuous stream of word tokens. The annotations may refer to the segmentation of the input stream into word tokens, but may also involve grouping together sequences of tokens or identifying sub-token units (or morphemes), depending on the language under consideration and, in particular, the definitions of "word" and "morpheme" as applied to this language. The description of word classes may include one or several features such as syntactic category, lemma, gender, number etc., which is again dependent on the language being analyzed.

Morpho-syntactic annotation can be represented by a single type of structural node (named W-level) representing a word-level structure unit. One or several information units are associated with each structural node.

For the purposes of illustration, we identify the following data categories (in practice these would be defined in reference to categories in the Data Category Registry):

• /lemma/: contains or points to a reference word form for the token or sequence of tokens being described; • /part of speech/: a reference to a morpho-syntactic category; • /confidence/: a confidence level assigned by the manual or automatic annotator in ambiguous cases. • /gender/: the grammatical gender information associated with a word token or a sequence of word tokens; • /number/: the grammatical gender information associated with a word token or a sequence of word tokens; • /tense/: the grammatical tense information associated with a word token or a sequence of word tokens;

• /person/: the grammatical person information associated with a word token or a sequence of word tokens.

The following provides an example of the morphosyntactic annotation of the sentence "Paul aime les croissants" in the GMT format: 1 <struct type="MSAnnot"> <struct type="W-level"> <feat type="lemma">Paul</feat> <feat type="pos">PNOUN</feat> <seg target="#w1"/> </struct> <struct type="W-level"> <feat type="lemma">aimer</feat> <feat type="pos">VERB</feat> <feat type="tense">present</feat> <feat type="person">3</feat> <seg target="#w2"/> </struct> <struct type="W-level"> <feat type="lemma">le</feat> <feat type="pos">DET</feat> <feat type="number">plural</feat> <seg target="#w3"/> </struct> <struct type="W-level"> <feat type="lemma">croissant</feat> <feat type="pos">NOUN</feat> <feat type="number">plural</feat> <seg target="#w4"/> </struct> </struct> Note that there is no limit to the number of information units that may be associated with a given structural node (as opposed to the text based representations that are usually provided by available POS taggers). It is also possible to structure the annotations by embedding <feat> elements to reflect a more complex feature-based annotation, or by pointing to a lexical entry providing the information.

In some cases, the morpho-syntactic annotation of a word or sequence of words requires a hierarchy of word level structures (e.g., when a word token results from the combination of several morphemes that must be annotated independently). For example, some occurrences of the token "du" in French can be analyzed as the fusion of the preposition "de" with the determiner "le" (as in "la queue du chat"). This is handled by embedding word-level structures as follows:

<struct type="W-level"> <seg target="#w1"/> <struct type="W-level"> <feat type="lemma">de</feat> <feat type="pos">PREP</feat> </struct> <struct type="W-level"> <feat type="lemma">le</feat> <feat type="pos">DET</feat> </struct> </struct> Conversely, annotation of compound words may involve associating a single lemma to a sequence of word tokens at the surface level. In this case, the lemma is attached to the higher level of embedding and reference to the source is given at the leaves of the hierarchy, as in the following representation of the compound "pomme de terre" in French :

<struct type="W-level"> <feat type="lemma"> pomme_de_terre</feat> <feat type="pos">NOUN</feat> <struct type="W-level"> <seg target="#w1"/> <feat type="lemma">pomme</feat> <feat type="pos">NOUN</feat> </struct> <struct type="W-level"> <seg target="#w2"/> <feat type="lemma">de</feat> <feat type="pos">PREP</feat> </struct> <struct type="W-level"> <seg target="#w3"/> <feat type="lemma">terre</feat> <feat type="pos">NOUN</feat> </struct> </struct>

The ability to specify a hierarchical structure where needed enables specification of the level of granularity required. This is especially critical for a representation scheme, since the granularity of the segmentation in (or associated with) the primary data may not directly correspond to the level of granularity required for the annotation.

Alternatives

Morpho-syntactic annotation can be used to illustrate the representation of both structural and informational alternatives, which arises when a given word token is associated with two or more word classes. For example, the French word "bouche" which can be derived both from the verb "boucher" and the noun "bouche", which can be represented as follows:

<struct type="W-level"> <seg target="#w1"/> <alt> <feat type="lemma">boucher</feat> <feat type="pos">VERB</feat> <feat type="tense">present</feat> <feat type="confidence">0.4</feat> </alt> <alt> <feat type="lemma">bouche</feat> <feat type="pos">NOUN</feat> <feat type="confidence">0.6</feat> </alt> </struct>

Relating annotation levels

We assume the use of stand-off annotation; that is, an annotated corpus is represented as a lattice of stand-off annotation documents pointing to a primary source or intermediate annotation levels. However, depending on the point of view, the relations between various annotation levels can be more or less explicit. It is possible to identify three major ways to relate different levels of annotation: temporal anchoring, event-based anchoring, and objectbased anchoring.

Temporal anchoring associates positional information to each structural level. This positional information is typically represented as a pair of numbers expressing the starting point and ending point of the segment being described. To do so in our framework, we introduce two attributes for the <seg> element: • /startPosition/: the temporal or offset position of the beginning of the current structural node;

• /endPosition/: the temporal or offset position of the end of the current structural node. For example, the following associates a phonetic transcription with a given portion of a primary text: <struct type="phonetic"> <seg startsAt="2300" endsAt="3200"/> <feat type="phone">iy</feat> </struct>

We also define an event-based anchoring, which effectively introduces a structural node to represent a location in the text, to which all annotations for the object at that location can refer. This strategy is useful in two cases:

• Situations where it is not possible or desirable to modify the primary data by inserting markup to identify specific objects or points in the data (e.g., speech annotation, associated with a speech signal, or in general any "read-only" data).

• Primary data marked with "milestones", such as time stamps in speech data, where spans across the various milestones must be identified. In this case, the < s t r u c t > elements represent the markup for segmentation (e.g., segmentation into words, sentences, etc.). 2

To represent this, we introduce a specific type of structural node, named landmark, which is referred to by annotations for the defined span, as follows:

<struct type="landmark"> <seg startsAt="2300" endsAt="3200"/> </struct>

The third mechanism, object-based anchoring, enables pointing from a given level to one or several structural nodes at another level. This mechanism is particularly useful to make dependencies between two or more annotation levels explicit. For example, syntactic annotation can refer directly to the relevant nodes in a morpho-syntactically annotated corpus, in order, for example, to identify the correct NP "le chat" in "la queue du chat", as shown below:

<!--Morphosyntactic level --> <struct type="W-level"> <seg target="#w3"> <struct type="W-level"> <seg target="#w3.1"> <feat type="lemma">de</feat> <feat type="pos">PREP</feat> </struct> <struct type="W-level"> <seg target="#w3.2"> <feat type="lemma">le</feat> <feat type="pos">DET</feat> <feat type="gender">masc</feat> </struct> </struct> <struct type="W-level"> <seg target="#w4"> <feat type="lemma>chat</feat> <feat type="pos">NOUN</feat> </struct> </struct> <!--Syntactic level (simplified) --> <struct> <feat type="synCat">NP</feat> <seg targets="w3.2 w4"/> </struct> 2 The annotation graph (AG) formalism [START_REF] Bird | A formal framework for linguistic annotation[END_REF] was explicitly designed to deal with time-stamped data. However, we feel the AG is not sufficiently general because (1) AG reifies the "arc" and distinguishes it from identification of spans via, e.g., XML tags; and (2) AG requires ad hoc mechanisms to deal with hierarchically organized annotations. In both cases, AG requires different mechanisms to treat analogous constructs.

Summary

The framework presented here for linguistic annotation is intended to allow for variation in annotation schemes while at the same time enabling comparison and evaluation, merging of different annotations, and development of common tools for creating and using annotated data. We have developed an abstract model for annotations that is capable of representing the necessary information while providing a common encoding format that can be used as a pivot for combining and comparing annotations, as well as an underlying format that can be manipulated and accessed with common tools. The details presented here provide a look "under the hood" in order to show the flexibility and representational power of the abstract scheme. However, the intention is that annotators and users of annotation schemes can continue to use their own or other formats with which they are comfortable; as long as the underlying data model is the same, translation into and out of this or any other instantiation of the abstract format will be automatic.

Our framework for linguistic annotation is built around some relatively straightforward ideas: separation of information conveyed by means of structure and information conveyed directly by specification of content categories; development of an abstract format that puts a layer of abstraction between site-specific annotation schemes and standard specifications; and creation of a Data Category Registry to provide a reference set of annotation categories. The emergence of XML and related standards, such as RDF, provides the enabling technology. We are, therefore, at a point where the creation and use of annotated data and concerns about the way it is represented can be treated separately-that is, researchers can focus on the question of what to encode, independent of the question of how to encode it. The end result should be greater coherence, consistency, and ease of use and access for linguistically annotated data.

Conclusion

ISO TC37/SC4 is just beginning its work, and will use the general framework discussed in the preceding sections as its starting point. However, the work of the committee will not be successful unless it is accepted by the language processing community. To ensure widespread acceptance, it is critical to involve as many representatives of the community in the development of the standards as possible, in order to ensure that all needs are addressed. This paper serves as a call for participation to the language processing community; those interested should contact the TC 37/SC 4 chairman (Laurent Romary: romary@loria.fr).

  WG1/WI-0: Terminology for Language Resources • WG1/WI-1: Linguistic annotation framework • WG1/WI-2: Meta-data for multimodal and multilingual information • WG2/WI-3: Structural content representation (syntax and morphology) • WG2/WI-4: Multimodal content representation • WG2/WI-5: Discourse level representation • WG3/WI-6a: Multilingual text representation • WG4/WI-7: Lexicons • WG5/WI-8: Validation of language resources • WG5/WI-9: Net-based distributed cooperative work for the creation of LRs

For brevity, we use an abbreviated pointer syntax to refer to the primary data in this example.


Harry Bunt
email: harry.bunt@kub.nl

Laurent Romary
email: laurent.romary@loria.frlet.kub.nl

TOWARDS MULTIMODAL CONTENT REPRESENTATION

INTRODUCTION

Multimodal interfaces, combining the use of speech, graphics, gestures, and facial expressions in input and output, promise to provide new possibilities to deal with information in more effective and efficient ways, supporting for instance:

-the understanding of possibly imprecise, partial or ambiguous multimodal input; -the generation of coordinated, cohesive, and coherent multimodal presentations; -the management of multimodal interaction (e.g., task completion, adapting the interface, error prevention) by representing and exploiting models of the user, the domain, the task, the interactive context, and the media (e.g. text, audio, video).

An intelligent multimodal interface requires a number of functionalities concerning media input processing and output rendering, deeper analysis and synthesis drawing at least upon underlying models of media and modalities (language, gesture, facial expression of user or animated agent), fusion and coordination of multimodal input and output at a semantic level, interpretation of multimodal input within the current state of the interaction and the context, and reasoning about and planning of multimodal messages. This implies an architecture with many components and interfaces; a reference architecture of an intelligent multimodal dialogue system was established at the workshop `Coordination and Fusion in Multimodal Interaction' in Dagstuhl, Germany, November 2001 (see Bunt, Kipp, Maybury and Wahlster, forthcoming, and http://www.dfki.de/~wahlster/Dagstuhl_Multi_Modality). The communication between many of the components in a multimodal interactive system rely upon an enabling syntax, semantics and pragmatics. A multimodal meaning representation plays central stage in such a system, supporting both interpretation and generation. Such a representation should support any kind of multimodal input and output, and should, in order to be useful in a field which is still developing, be sufficiently open to support a range of theories and approaches to multimodal communication.

The present document is intended to support the discussion on multimodal content representation, its possible objectives and basic constraints, and how the definition of a generic representation framework for multimodal content representation may be approached. It takes into account the results of the Dagstuhl workshop, in particular those of the informal working group on multimodal meaning representation that was active during the workshop (see http://www.dfki.de/~wahlster/Dagstuhl_Multi_Modality, Working Group 4).

SCOPE

To delineate the task of formulating objectives, constraints and components of multimodal meaning representation, we must first have a shared understanding of what is meant by meaning in multimodal interaction. We propose to define the meaning of a multimodal `utterance' as the specification of how the interpretation of the `utterance' by an understanding system should change the system's information state (taken in a broad sense of the term, including domain model, discourse model, user model, task model, and maybe more -see e.g. [START_REF] Bunt | Dialogue pragmatics and context specification[END_REF]. While formulated with reference to input interpretation only, this definition can also be related to the generation of multimodal outputs, by assuming that an output is generated by the system in order to have an effect on the user through the interpretation of that output by the user. (The generation of appropriate outputs thus depends on the system having an adequate model of what its outputs may mean to the user -which is exactly as it should be.)

A multimodal meaning representation should support the fusion of multimodal inputs and the fission of multimodal outputs at a semantic level, representing the combined and integrated semantic contributions of the different modalities. The interpretation of a multimodal input, such as a spoken utterance combined with a gesture and a certain facial expression, will often have stages of modality-specific processing, resulting in representations of the semantic content of the interactive behavior in each of the separate modalities involved. Other stages of interpretation combine and integrate these representations, and take contextual information into account, such as information from the domain model, the discourse model and the user model. A multimodal meaning representation language should support each of these stages of interpretation, as well as the various stages of multimodal output generation. Since we are considering inputs and outputs from a semantic point of view, the representation of lower-level modality-specific aspects of interactive behavior, like syntactic linguistic information or morphological properties of gestures is not a primary aim, but some such information may percolate as features associated with a meaning representation, especially at intermediate stages of interpretation, where their relevance for semantic interpretation may not have been fully exploited. At the other end of interpretation, where understanding is rooted in information structures like domain models and ontologies, a multimodal meaning representation language should support the connection with frameworks for defining ontologies and specifying domain models, such as DAML + OIL.

While supporting the linking up of meaning representations with ontologies and `low-level' modality-specific information, the design of multimodal meaning representations is to be clearly distinguished from the design of domain model representations, linguistic morphosyntactic representations, representations of facial expressions, etc., which do not fall within this scope. Also, meaning representations should not represent the underlying processes by which they are constructed and manipulated, although it may be important that they are `annotated' with administrative information relating to their processing, such as time stamps.

OBJECTIVES

The main objective of defining multimodal meaning representations is to provide a fundamental interface format to represent a system's understanding of multimodal user inputs, and to represent meanings that the system will express as multimodal outputs to the user. This interface format should thus be adequate for representing the end result of multimodal input interpretation, and for representing the semantic content that the system will present to the user in multimodal form. It should therefore allow dialogue management, planning and reasoning modules to operate on these representations. In order to be useful for this purpose, this interface format should support the interfaces of these as well as other modules that form part of the system, and thus be adequate not only for representing the end result of semantic interpretation but also intermediate results. Something similar holds for generation. This is a second objective that follows almost immediately from the first.

Another objective in defining a well-defined representational framework for multimodal communicative acts is to allow the specification and comparison of existing application-specific representations (e.g. the M3L representation used in the SmartKom project) and the definition of new ones, while ensuring a level of interoperability between these.

Finally, the specification of a multimodal meaning representation should also be useful for the definition of annotation schemes of multimodal semantic content.

BASIC CONSTRAINTS

Given the main objective of defining meaning representations, the first and foremost basic requirements of a semantic representation framework are those that we may call `expressive' and `semantic' adequacy:

 Expressive adequacy: the framework should be expressive enough to correctly represent the meanings of multimodal communicative acts;  Semantic adequacy: the representation structures should themselves have a formal semantics, i.e., their definition should provide a rigorous basis for reasoning (whether deductive, statistical, in the form of plan operators, or otherwise).

The second objective, of providing interface formats within a multimodal dialogue system architecture, means that `incremental' construction should be supported of intermediate and partial representations, leading up to a final representation or, if the construction of a final representation does not succeed, leading to negative feedback or another appropriate system action. This implies three further basic constraints:

 Incrementality, in the sense of supporting various stages of multimodal input interpretation, as well as of multimodal output generation, allowing both early and late fusion and fission;  Uniformity: to make incremental processing feasible, where possible the representation of various types of input and output should be uniform in the sense of using the same kinds of building blocks and the same ways in which complex structures can be composed of these building blocks.  Underspecification and Partiality: to support the representation of partial and intermediate results of semantic interpretation, the framework should allow meaning representations which are underspecified in various ways, and which capture unresolved ambiguities.

Finally, the representational framework should take into account that the design of multimodal human-computer dialogue systems is a developing area in which new research results and new technologies may bring new challenges and new approaches for the representation of multimodal meanings. This means that the representational framework should satisfy the following two constraints:

 Openness: the framework should not depend on a single, particular theory of meaning or meaning representation, but should invite contributions from different semantic theories and approaches to meaning representation;  Extensibilty. The framework should be compatible with alternative methods for designing representation schemas (like XML), rather than support only a single specific schema.

METHODOLOGY

As a first step in the direction of defining a generic multimodal semantic representation form, we have to establish some basic concepts and corresponding terminology.

First, the action-based concept of meaning mentioned above, applicable to multimodal inputs in an interactive situation, means that the meaning of a multimodal `utterance' has two components: one that is often called `propositional' or `referential', and that is concerned with the entities that the utterance refers to and with their properties and relations that may be expressed in propositions, and a `functional' component that expresses a speaker's intention in producing the utterance: what effects does he want to achieve (using `speaker' in a broad, multimodal sense here)? This distinction is familiar from speech act theory, where the two components are called `propositional content' and `illocutionary force', and is also prevalent in other theories of language-based communication (see [START_REF] Bunt | Dialogue pragmatics and context specification[END_REF]; it is often viewed as drawing a border line between semantics and pragmatics. In the analysis of multimodal interaction it is especially important to pay attention to both these aspects of meaning, since different modalities often contribute to each aspect in different ways; for instance, in spoken interaction the referential and propositional aspects of meaning are often expressed verbally, while gestures and facial expression contribute primarily to the functional aspects. The term `multimodal content' should not be confused with `propositional content', and should not make us forget that multimodal messages have meanings with functional aspects that are equally important as their propositional and referential aspects. In this document we use `multimodal content' as synonymous with `multimodal meaning', including functional aspects, and we use `semantic representation' as synonymous with `representation of meaning'.

A convenient term that has become popular in the literature on human-computer dialogue is `dialogue act'. This term is mostly used in an informal, intuitive way, or as a variant of `speech act; it has a formal definition in terms of the effects that a `speaker' intends to achieve through its understanding by the addressee (see [START_REF] Bunt | Dialogue pragmatics and context specification[END_REF], which makes it suitably precise for use in the analysis of the meaning of multimodal inputs and outputs. Without further going into definitions here, we will use the term `dialogue act' in the rest of this document. Definitions of other useful concepts can be found in [START_REF] Romary | MMIL requirements specification. Project MIAMM -Multidimensional Information Access using Multiple Modalities[END_REF].

As a second methodological step, we propose to distinguish the following three basic types of ingredients that would seem to go into any multimodal meaning representation framework. Each of these ingredients is discussed further in subsequent sections 1. Basic components: the basic constructs for building representations of the meaning of multimodal dialogue acts: types of building blocks and ways to connect them. 2. General mechanisms: representation techniques like substructure labeling and linking, that make the representations more compact and flexible. 3. Contextual data categories: types of administrative (meta-)data that do not, strictly speaking, contribute to the meanings of semantic representations, but that may nonetheless be relevant for their processing.

BASIC COMPONENTS

Initially, the following basic components can be identified to represent the general organization of any semantic structure:

1. temporal structures (`events'), to represent, for instance:  spoken utterances (input or output dialogue acts);  gestures (same);  noncommunicative action (like searching for information, making a calculation);  events, states, processes,.. in the discourse domain, representing meanings of verbs and possibly other linguistic expressions; 2. referential structures (`participants'), to represent, for instance:

 the speaker of an input utterance, or the person performing a gesture;  the addressee of a system output dialogue act;  individuals and objects participating in a semantic event 3. restrictions on temporal and referential structures, to represent, for instance:

 the type(s) of dialogue, act associated with an utterance;  a gesture type, assigned to a gesture token 4. dependency structures, representing semantic relations between temporal and/or referential structures, for instance:

 participant roles (like SPEAKER, ADDRESSEE, AGENT, THEME, SOURCE, GOAL,..)  discourse/rhetorical relations  temporal relations.

It may be noted that linguistic semantic phenomena that have been studied extensively in relation to the needs of underspecific representation, such as quantification and modification, can also be represented with these basic components. For instance, a quantified statement like `Three men moved the piano' can be represented as a move-event involving a group of three men and a piano, where the collectiveness and the group size of the set of men that form the agent of the event are represented by means of restrictions on the event.

GENERAL MECHANISMS

In addition to these basic components, certain general mechanisms are important to make meaning representations suitable for representing partial and underspecified meanings, to give the representations a more manageable form, and to relate them to external sources of information. Examples of such mechanisms are:

1. substructure labeling: assigning labels to subexpressions and allowing the use of these labels, instead of the substructures that they label, as arguments in other subexpressions; 2. argument underspecification: partial or underspecified representations can be constructed using labels in argument positions; restrictions on labels can represent limitations on the ways in which such variables can be instantiated by labels of substructures elsewhere in the representation; 3. restrictions on label values: see previous mechanism. Alternatively, disjunctions, or lists of labels can be used to represent ambiguity or partiality; 4. structure sharing, as in typed feature structures, makes it possible to represent that a certain part of the representation plays more than one role, e.g. a participant may be both agent and theme in a semantic event, or may be the speaker of an utterance and the performer of a gesture, as well as the agent in a semantic event expressed by the multimodal dialogue act; 5. linking to domain models (types and instances) to anchor meaning representations in the domain of discourse; 6. linking to lower levels, such as syntactic structure, prosodic cues, gestural trajectories,.. is useful for tying a purely semantic representation to lower-level information that has given rise to it, and that may not yet have been fully interpreted.

CONTEXTUAL DATA CATEGORIES

Finally, meaning representations will need to be annotated with general categories of administrative information, both globally and also at the level of subexpressions, to capture certain information which is not found inside the elements of interactive behaviour, but which is potentially relevant for their interpretation and generation, such as:

1. Environment data, for instance:  time stamps and spatial information (when and where was this input received, etc.) 2. Processing information, such as:

 which module has produced this representation; what is its level of confidence, etc.

Interactional information:

 who is the speaker; what addressees are there,..

Technical Backgound: XML

At this stage, we should say a word about what appear to be the unavoidable technical choices for the definition of a multimodal content representation format that would be used, among other possibilities, to exchange information between processing modules within a man-machine dialogue system. As a matter of fact, f, as defined by the World Wide Web Consortium, appears to be the best candidate so far (and probably for quite a long time) to represent information structures intended to be transmitted across a network. In the following section, we give a very brief overview of XML, which we will then use to illustrate some of the principles mentioned above by means of a concrete example. XML (eXtended Markup Language) is a simplified (but also in some respects enhanced) version of SGML. It provides a syntax for document markup as well as for the description of the set of tags to be used in classes of documents (a socalled DTD, Document Type Definition). An XML document is made of three parts:

 An XML declaration, which, beyond identifying that the current document is an XML one, allows one to declare the character encoding scheme used in the document (e.g. iso-8859-1, utf-8, etc.);  A document type declaration, which can point to a DTD. This section can be omitted;  An XML instance corresponding to the actual data represented by the document.

XML makes an important distinction between a well-formed document, which only contains the XML declaration and a syntactically conformant instance, and a valid one, where the instance is also checked against the associated DTD.

Among other characteristics, we mention the following important properties of XML:

 XML is both Unicode and ISO 10646 compatible1  XML comes along with a specific mechanism, called namespaces, allowing one to combine, within the same document, markup taken from multiple sources. This very powerful mechanism, which is in particular the basis for XSLT and XML schemas, allows more modularity in the definition of an XML structure and also to reuse components defined in another context;  XML provides a general attribute 'xml:lang' to indicate the language used in a given element (see above).

The W3C also provides three very important recommendations for traversing XML documents, namely:  XPath, which describes a syntax and associated mechanisms to move within a document instance;  XPointer, which allows one to indicate a location within a document and is based upon the XPath recommendation;  XLink, which allows one to combine and qualify a set of pointers to describe a link between them.

These three recommendations are important for instance when one wants to relate some information produced by a given processing level and the information that has been used as input for those processes.

Still, it should be noticed that the existence of such a widely recognized metalanguage as XML does not solve our problems for representing multimodal content. First, XML by itself does not come with a formal semantics for its tags, and thus does not satisfy the requirement of semantic adequancy. Second, the requirements of flexibility and extensibility forbid us to try to standardize once and for all a precise XML format, but rather think of providing concepts and tools for anyone to be able to design his or her own format, while preserving interoperability conditions with someone else's choices. This is the spirit in which work has already been done within TC37/SC4 for the definition of TMF (Terminological Markup Framework; ISO 16642, under DIS ballot), first proposed for lexical information by [START_REF] Ide | A Formal Model of Dictionary Structure and Content[END_REF] and which has recently been taken over to deal with morphosyntactic and syntactic annotation (see (Ide &Romary, 2001a andIde &Romary, 2001b, respectively). The basic assumption that we make is that there exists an entire class of document formats that can be modelled by combining a metamodel, that is an abstract structure shared by all documents of a given type (e.g. syntactic annotation document), with a choice of the data categories that may be associated with the various levels of the metamodel. Such a description can be seen as a specification of the document format, which can be instantiated by providing XML representations for the metamodel and the data categories. In such a view, if a community of researchers and implementers agrees on the definition of a reduced set of metamodels for language resources, the actual choice of data categories is left to the responsibility of a specific application. In this framework, the interoperability between formats is ensured by providing a data category registry which gathers, together with precise reference and definition, the various data categories needed for a particular field.

In the case of multimodal content representation we thus advocate that, beyond agreement on the basic components and mechanisms for instance as described in this paper, which could go into the definition of an actual metamodel for content representation, one should not try to standardize a particular XML format more precisely (though we need to make specific choices to illustrate our approach with concrete examples, see below).

A simple example

In the following, we illustrate the possible combination of basic components, general mechanisms, and contextual data categories into a multimodal meaning representation. This representation exemplifies the general methodology that we suggested here, by taking up a sample semantic representation derived from an initial example expressed in the ULF+ format (ULF+ is a slightly updated version of a semantic representation language that was developed successively in the PLUS dialogue project, see [START_REF] Geurts | Quasi-logical form in PLUS[END_REF] in the multimodal DENK project; see [START_REF] Bunt | Multimodal cooperation with the DENK system[END_REF][START_REF] Kievit | Context-driven Natural Language Interpretation[END_REF].

In the XML excerpt below (corresponding to the sentence "I want to go from Paris to Stuttgart" uttered by a speaker named Peter), we have extended the original ULF+ representation to introduce the notion of dialogue act, whose participants are the speaker and the system. This example is intended to show how we can differentiate between three types of information in such a representation:  The instantiation of the semantic content representation metamodel as an XML outline (shown in underlined characters), which organizes the general information layout of the data to be represented;  The actual information units describing the various levels in the XML outline (shown in gray characters);  The generic mechanisms used to combine events, participants, restrictions and relations (indicated in bold characters).

The specific choices made in this example to represent the metamodel or the data categories as XML objects are only one possibility among many, and this does not affect the formal semantics of the underlying information structure. More precisely, the following explanation may help to clarify the example:

 The <semRep> element corresponds to the semantic representation of one elementary utterance or dialogue act. It is identified uniquely by an id attribute;  The <event> element is used in this example to represent both the dialogue act proper ("e1") and the event expressed by the corresponding linguistic content ("e2");  The <participant> element is used to represent the various entities involved in the events. Events and participants being related to one another by means of <relation> elements (with source and target attributes pointing to the corresponding arguments of the relation.

The various levels are then further described by a number of data categories, chosen here to illustrate the wide variety of possible cases. Notice the use of an <alt> structure to illustrate the case where an ambiguity would remain at a given step of analysis, each possibility being associated with a certainty evaluation ('cert' attribute). In accordance with the methodology developed in TMF, the name of the corresponding XML elements and attributes should not be the object of standardization, data categories being defined by abstract properties.

<semRep id="rep1"> <event id="e0"> <evtCat>utterance</evtCat> <speaker target="Peter"/> <adressee target="System"/> <alt> <dialAct cert="0.8">Order</dialAct> <dialAct cert="0.3">Inform</dialAct> </alt> </event> <participant id="Peter"> <!--A description of the speaker that can be referendum elsewhere in the document -->

ACTION PLAN

The variety of existing theoretical approaches, as well as the wide number of factors to be considered makes it very difficult to devise from scratch a truly generic framework for multimodal content representation. As a consequence it is necessary to involve, beyond the possibilities offered by the definition of a working group on this topic in TC37/SC4, as large a community of experts as possible in the development of such a framework. This is why we suggest that the work shall be initially conducted within a dedicated working group of SIGSEM (Special Interest Group on Computational Semantics of the Association of Computational Linguistics), which would be, right from the beginning, a liaison with TC37/SC4. This group would prepare a working draft, which would then be submitted to ISO. Doing so, it would also be easier to ensure a proper interaction with other interested communities, in particular the people working on multimedia representation (SIGMedia, in complement to the existing liaison between MPEG and TC37/SC4) and on discourse and dialogue (SIGDial).

The agenda would thus be the following:



  Refining the workplan on the basis of the present paper at the TC37/SC4 Preliminary Meeting in Jeju (Korea) in February 2002.  Presenting a position paper at the LREC workshop on "International Standards of Terminology and Language Resources Management" in May 2002.  First working group meeting in conjunction to IWCS-5 (5 th International Workshop on Computational Semantics) in Tilburg, the Netherlands, in January 2003.

The W3C has put pressure on both ISO and the Unicode consortium to make sure that they would not diverge in their parallel work on the definition of a universal character encoding scheme.