Gregor Wiedemann
email: gregor.wiedemann@uni-leipzig.de

Andreas Niekler

Document Retrieval for Large Scale Content Analysis using Contextualized Dictionaries

This paper presents a procedure to retrieve subsets of relevant documents from large text collections for Content Analysis, e.g. in social sciences. Document retrieval for this purpose needs to take account of the fact that analysts often cannot describe their research objective with a small set of key terms, especially when dealing with theoretical or rather abstract research interests. Instead, it is much easier to define a set of paradigmatic documents which reflect topics of interest as well as targeted manner of speech. Thus, in contrast to classic information retrieval tasks we employ manually compiled collections of reference documents to compose large queries of several hundred key terms, called dictionaries. We extract dictionaries via Topic Models and also use co-occurrence data from reference collections. Evaluations show that the procedure improves retrieval results for this purpose compared to alternative methods of key term extraction as well as neglecting co-occurrence data.

Introduction

Due to the vastly growing availability of (retro-)digitized large scale text corpora computer-assisted Content Analysis (CA) is of increasing interest for various disciplines and applications ranging from social sciences to business intelligence. When exploring large corpora analysts are confronted with the problem of selecting relevant documents for qualitative investigation and further quantitative analysis. Standard tasks in Information Retrieval (IR) usually rely on small sets of concrete key terms for querying a collection. Highly abstract research interests in CA often cannot describe research objectives by such small queries. For example, analysis on the war in Iraq certainly can query for Iraq AND war. But, what would be a reasonable query for documents containing neoliberal justifications of politics?

To meet special requirements of CA retrieval tasks we propose a procedure where a query is not (primarily) based on single terms, but on a set of reference documents. Compared to the problem of determining concrete key terms for a query it is rather easy for analysts to manually compile a collection of 'paradigmatic' documents which reflect topics or manner of speech matching their research objective. Retrieval with such a reference collection is then performed in three steps:

1. Extraction of a set of key terms from the reference collection, called dictionary. Terms in the dictionary are ranked by weight to reflect difference in their importance for describing an analysis objective. 2. Extraction of co-occurrence data from the reference collection as well as from an additional generic corpus representative of a given language. 3. Scoring relevancy of each document on the basis of dictionary and co-occurrence data to create a ranked list of documents.

Related work: Using documents as queries is a consequent idea within the Vector Space Model (VSM) of IR where key term queries are modeled as document vectors for comparison with documents in the target collection [START_REF] Salton | A vector space model for automatic indexing[END_REF]. Our approach extends the standard VSM approach by additionally employing of shared meanings of topic defining terms captured by co-occurrence data. Cooccurrence data has been used in standard IR tasks for term weighting as well as query expansion with mixed results (e.g. see [START_REF] Peat | The limitations of term co-occurrence data for query expansion in document retrieval systems[END_REF][START_REF] Billhardt | Using term co-occurrence data for document indexing and retrieval[END_REF][START_REF] Van Rijsbergen | A theoretical basis for the use of co-occurrence data in information retrieval[END_REF][START_REF] Wong | Generalized vector spaces model in information retrieval[END_REF]). These applications differ from our approach as they want to deal with unequal importance of terms in a single query due to term correlations in natural language. The method presented in this paper does not weight semantically dependent query terms by co-occurrence information globally. Instead, in CA analysts are often interested in certain aspects of meaning of specific terms. Following the distributional semantics hypothesis [START_REF] Firth | A synopsis of linguistic theory[END_REF] meaning may be captured by contexts better than just by isolated terms. Therefore, we score relevancy based on similarity of individual contexts of single query terms in sentences of the target documents compared to observed contexts from the reference collection. The paper is organized in 5 sections. After having clarified our motivation the next section presents our approach of semi-supervised dictionary extraction with the help of statistical Topic Models. The third part explains how to utilize ranked dictionaries together with co-occurrence data for document retrieval. Section 4 briefly introduces an example application of this procedure on a target collection of German newspaper articles, followed by an evaluation of the approach.

Dictionary Extraction with Topic Models

The generation and usage of dictionaries is an important part of quantitative CA procedures [START_REF] Krippendorff | Content analysis: An introduction to its methodology[END_REF]. Dictionaries provide the basis of code books and category systems within CA studies. As we want to exploit dictionaries for document retrieval we suggest a procedure of semi-supervised term extraction for dictionary creation. For this we apply a statistical Topic Model on a collection of paradigmatic documents. Reference documents should be selected carefully by the analysts in consideration of representing domain knowledge or specific language use of interest. The resulting list may be compared to domain term extraction based on reference corpus comparison [START_REF] Drouin | Detection of domain specific terminology using corpora comparison[END_REF] or tf/idf weighting of words. These calculate 'keyness' of terms isolated from each other. In contrast to those the Topic Model based approach takes account of the fact that terms do not occur independently of each other.

Statistical Topic Models: Topic Models are a set of statistical models for unsupervised extraction of latent semantic structures from document collections. ( [START_REF] Blei | Latent dirichlet allocation[END_REF] first introduced Latent Dirichlet Allocation). They generate results where underlying latent variables with K dimensions are extracted from a document collection. Those variables represent distributions over words φ •,k = p(w|z k ), (k = 1, . . . , K) representing their alignment to a semantically coherent group which may be interpreted as topic. Words w with a high probability p(w|z k ) in a topic k represent its determining terms and allow for interpretation of the meaning of an underlying thematic composition.

Extracting Dictionaries from Topic Models: Distributions p(w|z k ) represent a simplification of the collection content as a composition of topics. We can assume that highly probable words play an important role in the semantics of the whole corpus. Therefore, those words may be used to compile a dictionary of key terms within that collection. We define the weight of a term in the dictionary by the sum of its probability values in each topic.

In comparison to term frequency counts in a collection the probability weight of a term in a corpus represents a word's contribution to a certain context. Even if a context has relatively low evidence in the data because of a low frequency a term can have high probability p(w n |z k ) within a topic. We don't want to overly bias the ranks in the dictionary with very improbable topics and their wordse.g. the high probability of the top terms in topics of low probability would be ranked almost equal to the top words in highly probable topics. Therefore, we need to normalize the term probabilities p(w|z k ) according to either their topic's probability p(z k ) or their term frequency tf(w n ) within the corpus. We decide to normalize each terms probability in a certain topic with its term frequency, but to use log frequency to dampen the effect of high frequency terms. The final weight of a term in the dictionary is determined by

tw n = log(tf(w n )) K k=1 p(w n |z k ) ( 1
)
where K is the number of topics, tf(•) the term frequency, and w n the dictionary term. Descended sorting of term weights tw results in a list of ranked words which can be cut to a certain length N .

Analyst supervision: Within a Topic Model usually topics with undesired content can be identified. A few topics normally capture rather syntactic information of the collection representing co-occurring functional words in a corpus [START_REF] Alsumait | Topic significance ranking of LDA generative models[END_REF]. Other topics, although capturing semantic structure, may be considered as irrelevant by the analyst with a view to her/his research interest. In contrast to other methods of key term extraction the Topic Model approach allows to exclude those unwanted semantic clusters. Before calculation of the weights of terms one has to identify these topics which do not represent meaningful structures and to exclude them from the set of K topics. This is an important step for the analyst to exercise influence on the so far unsupervised dictionary creation process and a clear advantage over other methods of key term extraction.

Dictionaries can be employed as filters in IR systems reducing general collections to sub collections containing sets of documents of interest for further analysis. Using a dictionary of ranked terms for IR might be translated in the standard Vector Space Model (VSM) approach in combination with 'term boosting'. In this approach prior knowledge of unequal importance of terms is incorporated into query processing via factors for each term. A VSM-based scoring function can be computed for a document d and a dictionary as query q as follows:

score VSM (q, d) = w∈q tf(w, d) • boost(w) • norm(d) (2)
Usually IR weightings consider inverse document frequency of a term as a relevant factor. As the dictionary ranking is derived from Topic Models, information comparable to document frequency has indirectly already been taken into account. We skip the idf factor for each term in favor of our own dictionary weight. Rank information from the dictionary can be translated into a boosting factor for the scoring function. We suggest a factor ranging between 0 and 1 boost

(w) = 1 rank(w) (3)
for each term w which reflects that the most prominent terms in a dictionary of N terms are of high relevancy for the retrieval process while terms located nearer to the end of the list are of more equal importance.

To address the problem of document length normalization and identify relevant documents of all possible length we utilize pivoted unique normalization as introduced in [START_REF] Singhal | Pivoted document length normalization[END_REF]. Pivotal length normalization slightly lowers relevancy scores for shorter documents of a collection D and consequently lifts the score for documents after a pivotal value determined by the average document length. The normalization factor for each document is computed by

norm(d) = 1 (1 -slope) • pivot + slope • |U d | (4)
where U d represents the set of unique terms occurring in document d and pivot is computed by pivot

= 1 |D| • d∈D |U d |.
When evaluation data is available, the value for slope might be optimized for each collection. Lacking a gold standard for our retrieval task we set slope to 0.7 which has proven to be a reasonable choice for retrieval optimization in various document collections [START_REF] Singhal | Pivoted document length normalization[END_REF]. Further, the tf factor should reflect on the importance of an individual term relative to the average frequency of unique terms within a document:

avgtf (d) = 1 |U d | • t∈U d tf (t, d).
Thus, the final scoring formula yields a document ranking for the entire collection:

score dict (q, d) = w∈q 1 + log(tf(w,d)) 1 + log(avgtf(d)) • boost(w) • norm(d) (5)
Contextualizing dictionaries: The approach described above yields useful results when looking for documents which can be described by a larger set of key terms. When it comes to more abstract research interests, however, which aim to identify certain meanings of terms or specific language use, isolated observation of terms may not be sufficient. Fortunately the approach described above can be augmented with co-occurrence data from the reference collection to judge on relevancy of occurrence of a single key term in our target document. This helps not only to disambiguate different actual meanings of a term, but also reflects the specific usage of terms in the reference collection. Therefore we compute patterns of significant co-occurrences of the N terms in our dictionary with each other resulting in an N × N matrix C. Co-occurrences are observed in a sentence window. Statistical significance of a co-occurrence is calculated by the Dice measure which is the fraction of the count of all sentences containing term a and term b over the sum of all sentences containing each single term:

dice(a, b) = 2n ab n a + n b (6)
We decided for this measure instead of more sophisticated co-occurrence significance tests like Log Likelihood [START_REF] Dunning | Accurate methods for the statistics of surprise and coincidence[END_REF] because it also reflects syntagmatic relations of terms in language relatively well [START_REF] Bordag | A comparison of co-occurrence and similarity measures as simulations of context[END_REF], but, more important its values range is between 0 and 1 which makes measurements over different corpora comparable. This is useful for dealing with an unwanted effect we experienced when experimenting with co-occurrence data to improve our retrieval mechanism. Co-occurrences of terms in the sentences of a reference collection may reflect characteristics in language use of the included documents. However, certain co-occurrence patterns may reflect general regularities of language not specific to a collection of a certain domain (e.g. strong correlations between the occurrence of parents and children or multi word units like United States in one sentence). Applying co-occurrence data to IR scoring tends to favor documents where many of those common language regularities can be observed.

Instead of using the co-occurrence matrix C solely based on the reference collection we 'filter' the co-occurrence data by subtracting a co-occurrence matrix based on a second, randomly composed reference corpus. 1 A second N ×N matrix D of co-occurrences is computed from such a generic corpus. The subtraction of D from C delivers a matrix C ′ reflecting the divergence of co-occurrence patterns in the reference collection compared to common language: C ′ = max(C -D, 0). 2Values for common combinations of terms (United States) in C ′ are significantly lowered while combinations specific to the reference collection remain largely constant.

Using sentence co-occurrences:

To exploit co-occurrence data for IR the scoring function in eq. 5 has to be reformulated to incorporate a similarity measure between a co-occurrence vector profile of each term w in the dictionary and each sentence s in the to-be-scored-document d. Instead of using just term frequency we add information on contextual similarity:

tf sim(w, d) = s∈d w∈s tf(w, s) + α • s • C ′ •,w s • C ′ •,w (7)
The frequency of w within a sentence (which usually equals 1) is incremented by the cosine similarity between sentence vector s (sparse vector of length N indicating occurrence of dictionary terms in s) and the dictionary context vector for w out of C ′ . This measure rewards the relevancy score, if the target sentence and the reference term w share common contexts. In case they share no common context tf sim is equal to tf .

Because term frequency and cosine similarity differ widely in their range the influence of the similarity on the scoring needs to be controlled by a parameter α. If α is set to zero tf sim replicates simple term frequency counts. Values for α higher than 0 yield a mixing of unigram matching and contextual information for the relevancy score. Optimal values for α can be calculated by our evaluation method (see section 5). The context-sensitive score is computed as follows:

score context (q, d) = w∈q 1 + log(tfsim(w,d)) 1 + log(avgtf(d)) • boost(w) • norm(d) (8)

Example

The procedure presented above is applied to a political science research task performed as part of a German research project. The project aims at analyzing the influence of neoliberal ideas on domestic politics by studying the discourse in public media. In our example application we identify documents which (potentially) contain neoliberal argumentation in a collection of 101.032 newspaper articles of the German magazine DIE ZEIT (volumes 2000-2009). This example is also used for evaluation of the method in section 5.

To retrieve documents of interest for further analysis political scientists compiled a reference corpus consisting of 36 German books and journal articles written by self-confessed neoliberal theorists (e.g. Milton Friedman). In this reference corpus sentence boundaries are detected and tokens are lemmatized. A topic model based on the Pitman-Yor Process [START_REF] Teh | A hierarchical bayesian language model based on pitman-yor processes[END_REF] is calculated. For this process all paragraphs of books and articles in the collection were treated as single 'documents' for modeling. In the resulting 23 topics of this model we can identify a very large topic containing only English words (originating mostly from bibliographies in the reference collection) which have been clustered by the topic model process. Since this topic does not represent meaningful semantics for the analysis analysts could exclude it for the process of dictionary extraction. Using the process described in Section 2 a dictionary of 500 key terms is extracted from the reference collection. Co-occurrence patterns of these 500 terms are extracted according to Section 3.

This contextualized dictionary is then used to query the target corpus of DIE ZEIT newspaper articles. The 2,000 highest ranked articles are retrieved and used as starting point for further qualitative and quantitative analysis procedures by the analysts.

Evaluation

Determining a large set of key terms from a reference collection and extracting its co-occurrence profiles to compose a "query" is an essential step in the proposed retrieval mechanism to meet requirements of content analysts. Due to this, standard approaches of IR evaluation [START_REF] Clough | Evaluating the performance of information retrieval systems using test collections[END_REF] are not applicable. There are no test collections like the TREC datasets [START_REF] Voorhees | TREC: experiment and evaluation in information retrieval[END_REF] regarding such type of retrieval task. In order to evaluate our method we therefore follow two approaches:

1. Generating a quasi-gold standard of pseudorelevant documents to show performance improvements through the use of co-occurrence data as well as key term extraction via topic models, 2. Judging on the overall validity with precision at k evaluation in our example retrieval performed by domain experts.

Generating pseudorels: Due to the lack of proper gold standards for our special retrieval task we create a custom evaluation set to evaluate our approach. Therefore, we use a strategy of data fusion. This strategy merges results, namely ranked lists of documents, of multiple retrieval systems to a set of pseudorelevant documents. These pseudorels can be used as an automatically generated quasigold standard. The approach proposed by [START_REF] Nuray | Automatic ranking of information retrieval systems using data fusion[END_REF] shows that merging results of different retrieval systems by the Condorcet method generates pseudorels which produce evaluation results of IR systems that highly correlate with the TREC testset rankings of IR systems. Although it is hard to judge on the overall absolute performance of a system, the procedure allows for relative judgment between tested systems. We employ this strategy in a two-fold manner:

1. Optimizing parameter α for the best mix of unigram / co-occurrence matches 2. Deciding whether using topic models improves retrieval results over a simple tf/idf measure for key term extraction.

For the latter a second dictionary is created on the basis of the highest 500 tf/idf measures for each term in the reference collection. In [START_REF] Drouin | Detection of domain specific terminology using corpora comparison[END_REF] we set the influence of the context similarity within a sentence by the parameter α. For the evaluation we will treat every setting for this parameter as a different retrieval system in order to artificially create a large set of different systems. We vary the parameter α by steps of 2 in a range of [0, 30]. Furthermore, a system is added where just context similarities contribute to the ranking by setting α = 1 and leaving out tf (w, s) in [START_REF] Drouin | Detection of domain specific terminology using corpora comparison[END_REF]. Each of the 17 'systems' produces a ranked list of 2000 highestscored documents using the contextualized dictionary created by the topic model approach. Another set of retrieval systems is created in the same manner, but using the dictionary which was extracted with the tf/idf measure. Both sets together provide 34 lists of ranked retrieval results allowing for a comparison between the represented systems. For this, results of the systems need to be merged to a set of pseudorelevant documents by the following procedure taken from [START_REF] Nuray | Automatic ranking of information retrieval systems using data fusion[END_REF]:

1. Selecting a set of the most biased systems: As we want to compare results in two dimensions (i. use of co-occurrence data, ii. key term extraction procedure) we select 4 from our 34 'systems': The system which neglects cooccurrence data (α = 0) and the system which solely relies on co-occurrence data-both combined with a dictionary generated by Topic Model or tf/idf. 2. Select the top documents of each of the most biased systems as candidates for pseudorels. For each retrieved document in each system a norm weight can be computed and summed up over all systems: n d = S s=1 m s /i d,s , with system s, m i number of ranks in system s and i d,s rank of document d in system s. Figure 1 shows the sorted values of norm weights n d for the best 100 documents in our example. The first documents rank very high in most of the tested systems. Documents with lower n d ranks are retrieved only by a few of the tested systems and, thus, are considered not to be good candidates for pseudorels. We select the top 50 documents of each system. 3. Rank the candidates for pseudorels by using the Condorcet method which relies on counting wins and losses of direct comparison of document rankings within the most biased systems. 4. The selection of pseudorels should reflect only those documents which yield a robust ranking within the Condorcet method. Figure 2 shows that roughly half of the documents of our example dataset have distinguishable values. Others produce more equal amounts of wins. Following [START_REF] Nuray | Automatic ranking of information retrieval systems using data fusion[END_REF] we define the top 50% of the Condorcet ranked list as pseudorels.

Mean average precision: With this procedure a list of 54 documents is selected as pseudorels for evaluating the example application. Performance of each of the 34 retrieval systems is measured by utilizing the mean average precision (MAP).

Since we only have one result e.g. one query for each system MAP is used as

M AP = 1 R R k=1 P (R[k]
), with the number of relevant documents R and P (R[k]) as precision within the ranking of a system up to the document R[k].

The best performance of all tested systems is achieved by the system which mixes unigram and co-occurrence matching with parameter α = 14. Table 1 displays MAP values for the two retrieval results based solely on unigram or context matching and the best performing mixed approach. Furthermore, it contrasts systems based on the tf/idf dictionary with systems based on the Topic Model dictionary. The results indicate that systems based on a Topic Model approach perform indeed better than systems using tf/idf for term extraction if co-occurrence data is used. This is not surprising considering the fact that Topic Models are based on co-occurrences as well. Our dictionary extraction method preserves this information in contrast to the independence assumption underlying tf/idf. Overall, the evaluation shows that using context information outperforms the base line approach without context information for our retrieval purpose. Nonetheless, because the evaluation method uses pseudorels care has to be taken in interpreting absolute MAP values of the systems.

Precision at k: A second evaluation target is to test how dense the relevant documents on different ranges in the ranks are. The precision at k measure can be utilized to determine the quality of the process by manually assessing the first 10 documents downwards from the ranks 1, 101, 501, 1001, 1501, 1991. Documents were marked as relevant in case a domain expert was able to annotate text snippets therein regarding arguments, topics or claims representing a discourse framed in neoliberal terminology. The results in table 2 confirm the usefulness of our approach. Density of positively evaluated results in the upper ranks is very high and gets lower towards the bottom of the list. Precision in the best performing system remains high also in lower ranks while it drops off in comparison with the system which solely exploits unigram matchings.

We presented an extension of the VSM approach of IR by exploiting automatically extracted dictionaries and co-occurrence data from manually compiled reference collections. This method has proven to produce valuable results for Content Analysis studies to extract collections specific to a certain research interest from large unspecific corpora. The use of co-occurrence data improves upon merely taking into account raw frequencies. Results could be enhanced further by creating dictionaries with the help of Topic Models. The retrieval mechanism allows domain experts, such as social scientists, to build collections for further analyses based on paradigmatic example documents or theory texts representing abstract domain knowledge.

Fig. 1 .

 1 Fig. 1. Plot of the accumulated values n d of all documents from each retrieval system (first 100 ranks)

Table 1 .

 1 MAP evaluation of compared retrieval systems.

	Wins
	0
	1	10	20	30	40	50	60	70	80	90	100
						Rank
	Fig. 2. Plot of the wins of the documents
	found by the Condorcet method (values
	have been sorted).

Table 2 .

 2 Precision at 10 evaluated at different ranks of the unigram and α-mix retrieval results

Suitable corpora for this purpose, such as the ones provided by the "Leipzig Corpora Collection"[START_REF] Biemann | The leipzig corpora collection. monolingual corpora of standard size[END_REF] which is carefully maintained by computational linguists, may be seen as representative of common language characteristics not specific to a certain domain or topic.

max asserts that all negative values in C ′ (representing terms occurring less together in sentences of the reference collection than in sentences of common language) are set to zero.


Nancy Ide
email: ide@cs.vassar.edu

Laurent Romary
email: romary@loria.fr

Standards for Language Resources

The goal of this paper is two-fold: to present an abstract data model for linguistic annotations and its implementation using XML, RDF and related standards; and to outline the work of a newly formed committee of the International Standards Organization (ISO), ISO/TC 37/SC 4 Language Resource Management, which will use this work as its starting point. The primary motive for presenting the latter is to solicit the participation of members of the research community to contribute to the work of the committee.

Introduction

The goal of this paper is two-fold: to present an abstract data model for linguistic annotations and its implementation using XML, RDF and related standards; and to outline the work of a newly formed committee of the International Standards Organization (ISO), ISO/TC 37/SC 4 Language Resource Management, which will use this work as its starting point. The primary motive for presenting the latter is to solicit the participation of members of the research community to contribute to the work of the committee.

The objective of ISO/TC 37/SC 4 is to prepare international standards and guidelines for effective language resource management in applications in the multilingual information society. To this end, the committee will develop principles and methods for creating, coding, processing and managing language resources, such as written corpora, lexical corpora, speech corpora, dictionary compiling and classification schemes. The focus of the work is on data modeling, markup, data exchange and the evaluation of language resources other than terminologies (which have already been treated in ISO/TC 37). The worldwide use of ISO/TC 37/SC 4 standards should improve information management within industrial, technical and scientific environments, and increase efficiency in computer-supported language communication.

The standardization of principles and methods for the collection, processing and presentation of language resources requires a distinct type of standardization activity. Basic standards should be produced with wide-ranging applications in view. In the area of language resources, for instance, these standards should provide various technical committees of ISO, IEC and other standardizing bodies with the groundwork for building more precise standards for language resource management. ISO/TC 37/SC 4 will liaison with ISLE (International Standards for Language Engineering), which has implemented various recent efforts to integrate EC and US efforts for language resources. Where possible, these and other standards set up in EAGLES will be incorporated into the ISO standards. ISO/TC 37/SC 4 will also broaden the work of EAGLES/ISLE by including languages (e.g. Asian languages) that are not currently covered by EAGLES/ISLE standards.

We are aware that standardization is a difficult business, and that many members of the targeted communities are skeptical about imposing any sort of standards at all. There are two major arguments against the idea of standardization for language resources. First, the diversity of theoretical approaches to, in particular, the annotation of various linguistic phenomena suggests that standardization is at least impractical, if not impossible. Second, it is feared that vast amounts of existing data and processing software, which may have taken years of effort and considerable funding to develop, will be rendered obsolete by the acceptance of new standards by the community.

To answer both of these concerns, we stress that the efforts of the committee are geared toward defining abstract models and general frameworks for creation and representation of language resources that should, in principle, be abstract enough to accommodate diverse theoretical approaches. The model so far developed in ISO TC/37 for terminology, which has informed and been informed by work on representation schemes for dictionaries and other lexical data [START_REF] Ide | A Formal Model of Dictionary Structure and Content[END_REF] and syntactic annotation [START_REF] Ide | A Common Framework for Syntactic Annotation[END_REF] demonstrates that this is not an unrealizable goal. Also, by situating all of the standards development squarely in the framework of XML and related standards such as RDF, we hope to ensure not only that the standards developed by the committee provide for compatibility with established and widely accepted web-based technologies, but also that transduction from legacy formats into XML formats conformant to the new standards is feasible.

At present, we feel that language professionals and standardization experts are not sufficiently aware of the standardization efforts being undertaken by ISO/TC 37/SC 4. Promoting awareness of future activities and rising problems, therefore, will be a crucial factor in the future success of the committee, and will be required to ensure widespread adoption of the standards it develops. An even more critical factor for the success of the committee's work is to involve, from the outset, as many and as broad a range of potential users of the standards as possible. This paper serves in part as a call for participation to the linguistics and computational linguistics research communities.

In the following sections, we describe the principles and architecture of a general framework for annotations that can serve to define an abstract format capable of representing the range of annotation types. We then provide examples in which the format is applied to specific annotations types.

Representing linguistic annotation

The goal of our work is to define a model for linguistic annotation that can (a) be instantiated in a standard representational format; and (b) can serve as a pivot format into and out of which proprietary formats can be transduced, in order to enable comparison and merging, as well as operation on the data by common tools, as shown in Figure 1.

Figure 1. Overall architecture for handling multiple formats ("A", "B", and "C")

To accomplish this, it is necessary to identify a consistent underlying data model for data and its annotations. A data model is a formalized description of the data objects (in terms of composition, attributes, class membership, applicable procedures, etc.) and relations among them, independent of their instantiation in any particular form. A data model capable of capturing the structure and relations in diverse types of data and annotations is a pre-requisite for developing a common corpus-handling environment: it impacts the design of annotation schema, encoding formats and data architectures, and tool architectures.

Abstract model for annotation

At its highest level of abstraction, an annotation is a set of data or information (in our case, linguistic information) that is associated with some other data. The latter is what could be called "primary" data (e.g., a part of a text or speech signal, etc.), but this need not be the case; consider, for example, the alignment of

A B C Abstract Format
Operation via common tools, merging, etc parallel translations, where the "annotation" is a link between two primary data objects (the aligned texts). Typically, primary data objects are represented by "locations" in an electronic file, for example, the span of characters comprising a sentence or word, or a point at which a given temporal event begins or ends (as in speech annotation). As such, at the base primary data objects are relatively simple in their structure; more complex data objects may consist of a list or set of contiguous or noncontiguous locations. Annotation objects, on the other hand, often have a more complex internal structure: syntactic annotation, for example, may be expressed as a tree structure, and may include more elemental annotations such as dependency relations (which is itself an annotation relating two objects, where the relation is directional (dependent-to-head)).

Thus, we can conceive of, an annotation as a one-or two-way link between an annotation object and a point (or a list/set of points) or span (or a list/set of spans) within a base data set. Links may or may not have a semantics--i.e., a type--associated with them. Points and spans in the base data may themselves be objects, or sets or lists of objects. This abstract formulation can serve as the basis for defining a general model for linguistic annotation that can be realized in some representational format. However, we first make several observations:  the model assumes a fundamental linearity of objects in the base,1 e.g., as a time line (speech); a sequence of characters, words, sentences, etc.; or pixel data representing images;  the granularity of the data representation and encoding is critical: it must be possible to uniquely point to the smallest possible component (e.g., character, phonetic component, pitch signal, morpheme, word, etc.);

 an annotation scheme must be mappable to the structures defined for annotation objects in the model;  an encoding scheme must be able to capture the object structure and relations expressed in the model, including class membership and inheritance, therefore requiring a sophisticated means to specify linkage within and between documents;  it is necessary to consider the logistics of identifying spans by enclosing them in start and end tags (thus enabling hierarchical grouping of objects in the data itself), vs. explicit addressing of start and end points;  it must be possible to represent objects and relations in some (fairly straightforward) form that is both usable by a variety of tools and prevents information loss;  ideally, it should be possible to represent the objects and relations in a variety of formats suitable to different tools and applications.

A closer look at annotations

As noted above, annotation objects may be relatively complex. In order to define a generic model for linguistic annotation, it is necessary to consider the representational needs for annotations themselves.

Linguistic annotation can be represented as a graph of elementary structural nodes to which one or more information units are attached. The distinction between the structure of annotations and the informational units of which it is comprised is, we feel, critical to the design of a truly general model for annotations. Annotations may be structured in several ways; perhaps the most common structure is hierarchical. For example, phrase structure analyses of syntax are structured as trees; in addition, hierarchy is often used to break annotation information into subcomponents, as in the case of lexical and terminological information, There are several special relations among annotations that must be represented in the model, including the following:

1 Parallelism: two or more annotations refer to the same data object;

Alternatives: two or more annotations comprise a set of mutually exclusive alternatives (e.g., two possible part-ofspeech assignments, before disambiguation); 3

Aggregation: two or more annotations comprise a list or set that should be taken as a unit.

Information units or data categories provide the semantics of the annotation. Data categories are the most theory and application-specific part of an annotation scheme. We do not attempt to define the relevant data categories for given types of annotation. Rather, we propose the development of a Data Category Registry to provide a framework in which the research community can formally define data categories for reference and use in annotation, Data categories would be defined using RDF schemas to formalize the properties and relations associated with each. Note that RDF descriptions function much like class definitions in an object-oriented programming language: they provide, effectively, templates that describe how objects may be instantiated, but do not constitute the objects themselves. Thus, in a document containing an actual annotation, several objects with the same type may be instantiated, each with a different value. The RDF schema ensures that each instantiation is recognized as a sub-class of more general classes and inherits the appropriate properties.

A formally defined set of categories will have several functions: (1) it will provide a precise semantics for annotation categories that can be either used "off the shelf " by annotators;

(2) it will provide a set of reference categories onto which scheme-specific names can be mapped; and (3) it will provide a point of departure for definition of variant or more precise categories.

Thus the overall goal of the Data Category Registry is not to impose a specific set of categories, but rather to ensure that the semantics of data categories included in annotations (whether they exist in the Registry or not) are well-defined and understood.

The Generic Mapping Tool

We instantiate the abstract format for annotations using the eXtensible Markup Language (XML). XML is the emerging standard for data representation and exchange on the World Wide Web [START_REF] Bray | Extensible Markup Language (XML) Version 1.0. W3C Recommendation[END_REF].

Although at its most basic level XML is a document markup language directly derived from SGML (i.e., allowing tagged text (elements), element nesting, and element references), various features and extensions of XML (including XSLT, RDF, etc.) make it a powerful tool for data representation and access.

The model is instantiated by the Generic Mapping Tool (GMT), which includes the following tags:  <struct> represents a structural node in the annotation. <struct> elements may be recursively nested at any level to reflect a tree structure for the annotation. Attributes include  type : annotation type (e.g., "syntax"), where necessary or desirable;  ID: unique identifier for the node  ref : node this <struct> node represents (for implicit structures)  <feat> (feature) is used to provide information attached to the node represented by the enclosing <struct> element. A type attribute on the <feat> element identifies the data category of the feature. The tag may contain a string that provides an appropriate value for the data category (e.g., for type=CAT the value might be "NP") or <feat> can be recursively refined to describe complex structures. Alternatively, it may point via a target attribute to an object in another document that provides the value. Note that this allows the possibility for generating a single instantiation of an annotation value in a separate document that can be referenced as needed within the annotation document.

 <alt> is used to provide one or more alternative annotations, where necessary.  <rel> is used to point to a non-contiguous related element, e.g., to identify dependencies explicitly by pointing to the related <struct> node.  <seg> points to the data to which the annotation applies. We assume the use of stand-off annotation2 -i.e., annotation that is maintained in a document separate from the primary (annotated) data-as first defined in the Corpus Encoding Standard (CES) (Ide, 1998a, b) and subsequently adopted by the research community. A target attribute on the <seg> element uses XML Pointers (Xpointer) [START_REF] Daniel | XML Pointer Language (XPointer) Version 1.0. W3C Recommendation[END_REF] to specify the location of the relevant data.  <brack> is used to group information to be regarded as a unit.

The GMT is sufficiently powerful to represent the information across all annotation types. We have already demonstrated its applicability to terminological and lexical information [START_REF] Ide | A Formal Model of Dictionary Structure and Content[END_REF] and syntactic annotation [START_REF] Ide | A Common Framework for Syntactic Annotation[END_REF]; we provide additional examples below. Existing formats (whether or not in XML) can be mapped to the GMT, in order to enable merging, comparison, and manipulation via common tools. The GMT version can then be re-mapped to the original formats for use in in-house tools and applications. etc.

Examples

Morpho-syntactic annotation

We illustrate a simple application of the framework presented above for the domain of morpho-syntactic annotation. Morpho-syntactic annotation involves the identification of word classes over a continuous stream of word tokens.

The annotations may refer to the segmentation of the input stream into word tokens, but may also involve grouping together sequences of tokens or identifying sub-token units (or morphemes), depending on the language under consideration and, in particular, the definitions of "word" and "morpheme" as applied to this language. The description of word classes may include one or several features such as syntactic category, lemma, gender, number etc., which is again dependent on the language being analyzed.

Morpho-syntactic annotation can be represented by a single type of structural node (named Wlevel) representing a word-level structure unit organized hierarchically. One or several information units are associated with each structural node.

For the purposes of illustration, we identify the following data categories (in practice these would be defined in reference to the Data Category Registry):

 /lemma/: contains or points to a reference word form for the token or sequence of tokens being described;  /part of speech/: a reference to a morphosyntactic category;  /confidence/: a confidence level assigned by the manual or automatic annotator in ambiguous cases.  /gender/: the grammatical gender information associated with a word token or a sequence of word tokens;  /number/: the grammatical gender information associated with a word token or a sequence of word tokens;  /tense/: the grammatical tense information associated with a word token or a sequence of word tokens;  /person/: the grammatical person information associated with a word token or a sequence of word tokens.

The following provides an example of the morpho-syntactic annotation of the sentence "Paul aime les croissants" in the GMT format: 3 <struct type="MSAnnot"> <struct type="W-level"> <feat type="lemma">Paul</feat> <feat type="pos">PNOUN</feat> <seg target="#w1"/> </struct> <struct type="W-level"> <feat type="lemma">aimer</feat> <feat type="pos">VERB</feat> <feat type="tense">present</feat> <feat type="person">3</feat> <seg target="#w2"/> </struct> <struct type="W-level"> <feat type="lemma">le</feat> <feat type="pos">DET</feat> <feat type="number">plural</feat> <seg target="#w3"/> </struct> <struct type="W-level"> <feat type="lemma">croissant</feat> <feat type="pos">NOUN</feat> <feat type="number">plural</feat> <seg target="#w4"/> </struct> </struct> Note that there is no limit to the number of information units that may be associated with a given structural node (as opposed to the text based representations that are usually provided by available POS taggers). It is also possible to structure the annotations by embedding <feat> elements to reflect a more complex featurebased annotation, or by pointing to a lexical entry providing the information, In some cases, the morpho-syntactic annotation of a word or sequence of words requires a hierarchy of word level structures (e.g., when a word token results from the combination of several morphemes that must be annotated independently). For example, some occurrences of the token "du" in French can be analyzed as the fusion of the preposition "de" with the determiner "le" (as in "la queue du chat"). This is handled by embedding word-level structures as follows:

<struct type="W-level"> <seg target="#w1"/> <struct type="W-level"> <feat type="lemma">de</feat> <feat type="pos">PREP</feat> </struct> <struct type="W-level"> <feat type="lemma">le</feat> <feat type="pos">DET</feat> </struct> </struct> Conversely, annotation of compound words may involve associating a simgle lemma to a sequence of word tokens at the surface level. In this case, the lemma is attached to the higher level of embedding and reference to the source is given at the leaves of the hierarchy, as in the following representation of the compound "pomme de terre" in French :

<struct type="W-level"> <feat type="lemma"> pomme_de_terre</feat> <feat type="pos">NOUN</feat> <struct type="W-level"> <seg target="#w1"/> <feat type="lemma">pomme</feat> <feat type="pos">NOUN</feat> </struct> <struct type="W-level"> <seg target="#w2"/> <feat type="lemma">de</feat> <feat type="pos">PREP</feat> </struct> <struct type="W-level"> <seg target="#w3"/> <feat type="lemma">terre</feat> <feat type="pos">NOUN</feat> </struct> </struct>

The ability to specify a hierarchical structure where needed enables specification of the level of granularity required. This is especailly critical for a representation scheme, since the granularity of the segmentation in (or associated with) the primary data may not directly correspond to the level of granularity required for the annotation.

Alternatives

Morpho-syntactic annotation can be used to illustrate the representation of both structural and informational alternatives, which arises when a given word token is associated with two or more word classes. For example, the French word "bouche" which can be derived both from the verb "boucher" and the noun "bouche", which can be represented as follows:

<struct type="W-level"> <seg target="#w1"/> <alt> <feat type="lemma">boucher</feat> <feat type="pos">VERB</feat> <feat type="tense">present</feat> <feat type="confidence">0.4</feat> </alt> <alt> <feat type="lemma">bouche</feat> <feat type="pos">NOUN</feat> <feat type="confidence">0.6</feat> </alt> </struct>

Relating annotation levels

As noted above, we assume the use of stand-off annotation; that is, an annotated corpus is represented as a lattice of stand-off annotation documents pointing to a primary source or intermediate annotation levels.

However, depending on the point of view, the relations between various annotation levels can be more or less explicit. It is possible to identify three major ways to relate different levels of annotation: temporal anchoring, event-based anchoring, and object-based anchoring.

Temporal anchoring associates positional information to each structural level. This positional information is typically represented as a pair of numbers expressing the starting point and ending point of the segment being described. To do so in our framework, we introduce two attributes for the <seg> element:  /startPosition/: the temporal or offset position of the beginning of the current structural node;  /endPosition/: the temporal or offset position of the end of the current structural node. For example, the following associates a phonetic transcription with a given portion of a primary text:

<struct type="phonetic"> <seg startsAt="2300" endsAt="3200"/> <feat type="phone">iy</feat> </struct>

We also define an event-based anchoring, which effectively introduces a structural node to represent a location in the text, to which all annotations for the object at that location can refer. This strategy is useful in two cases:  Situations where it is not possible or desirable to modify the primary data by inserting markup to identify specific objects or points in the data (e.g., speech annotation, associated with a speech signal, or in general any "read-only" data).  Primary data marked with "milestones", such as time stamps in speech data, where spans across the various milestones must be identified. In this case, the <struct> elements represent the markup for segmentation (e.g., segmentation into words, sentences, etc.). To represent this, we introduce a specific type of structural node, named landmark, which is referred to by annotations for the defined span, as follows:

<struct type="landmark"> <seg startsAt="2300" endsAt="3200"/> </struct>

The annotation graph (AG) formalism [START_REF] Bird | A formal framework for linguistic annotation[END_REF] was explicitly designed to deal with time-stamped data. We can represent annotation graphs in the GMT as shown in Figure 2. We feel that the GMT representation is more general, for the following reasons:

 The AG formalism reifies the "arc" and treats it as a special <annotation> <arc><source id="0" offset="0"/><label att_1="P" att_2="h#"/><target id="1" offset="2360"/></arc> <arc><source id="1" offset="2360"/><label att_1="P" att_2="sh"/><target id="2" offset="3270"/></arc> <arc><source id="2" offset="3270"/><label att_1="P" att_2="iy"/><target id="3" offset="5200"/></arc> <arc><source id="1" offset="2360"/><label att_1="W" att_2="she"/><target id="3" offset="5200"/></arc> <arc><source id="3" offset="5200"/><label att_1="P" att_2="hv"/><target id="4" offset="6160"/></arc> <arc><source id="4" offset="6160"/><label att_1="P" att_2="ae"/><target id="5" offset="8720"/></arc> <arc><source id="5" offset="8720"/><label att_1="P" att_2="dcl"/><target id="6" offset="9680"/></arc> <arc><source id="3" offset="5200"/><label att_1="W" att_2="had"/><target id="6" offset="9680"/></arc> <arc><source id="6" offset="9680"/><label att_1="P" att_2="y"/><target id="7" offset="10173"/></arc> <arc><source id="7" offset="10173"/><label att_1="P" att_2="axr"/><target id="8" offset="11077"/></arc> <arc><source id="6" offset="9680"/><label att_1="W" att_2="your"/><target id="8" offset="11077"/></arc> </annotation>

Figure 2a. XML instantiation of an annotation graph

<struct type="landmarkDesc> <struct type="landmark" id="0"> <position>0</position> </struct> <struct type="landmark" id="1"> <position>2360</position> </struct> <struct type="landmark" id="2"> <position>5200</position> </struct> … </struct> <struct type="phoneticAnnot"> <struct type="phone"> <startsAt target="#0"/> <endsAt target="#1"/> <phone>h#</phone> </struct> <struct type="phone"> <startsAt target="#1"/> <endsAt target="#2"/> <phone>sh</phone> </struct> <struct type="phone"> <startsAt target="#2"/> <endsAt target="#3"/> <phone>iy</phone> </struct> … </struct> <struct type="morphAnnot"> <struct type="w"> <startsAt target="#0"/> <endsAt target="#3"/> <source>she</source> </struct> … </struct>

Figure 2b. Annotation graph representation in GMT

The third mechanism, object-based anchoring, enables pointing from a given level to one or several structural nodes at another level. This mechanism is particularly useful to make dependencies between two or more annotation levels explicit. For example, syntactic annotation can refer directly to the relevant nodes in a morpho-syntactically annotated corpus, in order, for example, to identify the correct NP "le chat" in "la queue du chat", as shown below:

<!--Morphosyntactic level --> <struct type="W-level"> <seg target="#w3"> <struct type="W-level"> <seg target="#w3.1"> <feat type="lemma">de</feat> <feat type="pos">PREP</feat> </struct> <struct type="W-level"> <seg target="#w3.2"> <feat type="lemma">le</feat> <feat type="pos">DET</feat> <feat type="gender">masc</feat> </struct> </struct> <struct type="W-level"> <seg target="#w4"> <feat type="lemma>chat</feat> <feat type="pos">NOUN</feat> </struct> </struct> <!--Syntactic level (simplified) --> <struct> <feat type="synCat">NP</feat> <seg targets="w3.2 w4"/> </struct>

Conclusion

The framework presented here for linguistic annotation is intended to allow for variation in annotation schemes while at the same time enabling comparison and evaluation, merging of different annotations, and development of common tools for creating and using annotated data. We have developed an abstract model for annotations that is capable of representing the necessary information while providing a common encoding format that can be used as a pivot for combining and comparing annotations, as well as an underlying format that can be manipulated and accessed with common tools. The details presented here provide a look "under the hood" in order to show the flexibility and representational power of the abstract scheme; however, the intention is that annotators and users of syntactic annotation schemes can continue to use their own or other formats with which they are comfortable, and translation into and out of the abstract format will be automatic.

Our framework for linguistic annotation is built around some relatively straightforward ideas: separation of information conveyed by means of structure and information conveyed directly by specification of content categories; development of an abstract format that puts a layer of abstraction between site-specific annotation schemes and standard specifications; and creation of a Data Category Registry to provide a reference set of annotation categories. The emergence of XML and related standards, together with RDF, provides the enabling technology. We are, therefore, at a point where the creation and use of annotated data and concerns about the way it is represented can be treated separately-that is, researchers can focus on the question of what to encode, independent of the question of how to encode it. The end result should be greater coherence, consistency, and ease of use and access for linguistic annotated data.

Note that this observation applies to the fundamental structure of stored data. Because the targets of a relation may be either individual objects, or sets or lists of objects, information with more than one dimension is accommodated.

In the original CES definition, the term remote markup was introduced (see http://www.cs.vassar.edu/CES/CES1-5.html#ToCOview), but has subsequently been replaced by the term stand-off annotation.

For brevity, we use an abbreviated pointer syntax to refer to the primary data in this example.


Nancy Ide
email: ide@cs.vassar.edu

Laurent Romary
email: romary@loria.fr

Tomaž Erjavec
email: tomaz.erjavec@ijs.si

A Common XML-based Framework for Syntactic Annotation

It is widely recognized that the proliferation of annotation schemes runs counter to the need to re-use language resources, and that standards for linguistic annotation are becoming increasingly mandatory. To answer this need, we have developed a framework comprised of an abstract model for a variety of different annotation types (e.g., morpho-syntactic tagging, syntactic annotation, co-reference annotation, etc.), which can be instantiated in different ways depending on the annotator's approach and goals. In this paper we provide an overview of the framework, demonstrate its applicability to syntactic annotation, and show how it can contribute to comparative evaluation of parser output and diverse syntactic annotation schemes.

Introduction

It is widely recognized that the proliferation of annotation schemes runs counter to the need to re-use language resources, and that standards for linguistic annotation are becoming increasingly mandatory. In particular, there is a need for a general framework for linguistic annotation that is flexible and extensible enough to accommodate different annotation types and different theoretical and practical approaches, while at the same time enabling their representation in a "pivot" format that can serve as the basis for comparative evaluation of parser output, such as PARSEVAL (Harrison, et al., 1991), as well as the development of reusable editing and processing tools.

To answer this need, we have developed such a framework comprised of an abstract model for a variety of different annotation types (e.g., morpho-syntactic tagging, syntactic annotation, co-reference annotation, etc.), which can be instantiated in different ways depending on the annotator's approach and goals. We have implemented both the abstract model and various instantiations using XML schemas [START_REF] Thompson | XML Schema Part 1: Structures[END_REF], the Resource Definition Framework (RDF) (Lassila and Swick, 2000) and RDF schemas [START_REF] Brickley | Resource Description Framework (RDF) Schema Specification 1[END_REF], which enable description and definition of abstract data models together with means to interpret, via the model, information encoded according to different conventions. The results have been incorporated into XCES (Ide, et al., 2000a), part of the EAGLES Guidelines developed by the Expert Advisory Group on Language Engineering Standards (EAGLES), 1 which provides a ready-made, standard encoding format together with a data architecture designed specifically for linguistically annotated corpora.

In this paper we provide an overview of our encoding framework and demonstrate its applicability to syntactic annotation. The framework has also been applied to terminology (Terminological Markup Framework 2 , ISO project n.16642) and computational lexicons (Ide, et al., 2000b), thus demonstrating its general applicability for a variety of linguistic annotation types. We also show how the framework can contribute to comparative evaluation of diverse syntactic annotation schemes and enable assessment of the relative strengths and weaknesses of different parsers, an area of increasing importance for the field.

2

Current practice

Syntactic annotation schemes provide at least one, and often both, of the following kinds of information:

(1) category : bracketing and labeling of strings based on syntactic category (e.g., noun phrase, prepositional phrase); (2) dependency: relations among bracketed elements reflecting their syntactic roles (e.g., subject, object). For example, the annotation in Figure 1, drawn from the Penn Treebank II 3 (hereafter, PTB), uses LISP-like list structures to bracket strings and provide category information. Hierarchical nesting and left-to-right adjacency are used to reflect dependency relations; however, "subject" is explicitly (and redundantly, given that the relation is implicit in the structure) labeled along with the category information, although other relations (e.g., "object") are left implicit in the structure. Relations among non-contiguous elements demand a special numbering mechanism to enable reference to a non-contiguous elements, which enables specifying the NP-SBJ of the embedded sentence by reference to the earlier NP-SBJ-1 node.

1 http://www.ilc.pi.cnr.it/EAGLES/home.html 2 http://www.loria.fr/projects/TMF 3 http://www.cis.upenn.edu/treebank ((S (NP-SBJ-1 Jones) (VP followed) (NP him) (PP-DIR into (NP the front room)) , (S-ADV (NP-SBJ *-1) (VP closing (NP the door) (PP behind (NP him))))) .))

Figure 1 : PTB annotation example

Although they differ in the labels and in some cases the function of various nodes in the tree, most syntactic annotation schemes rely on hierarchical nesting to indicate component relations among elements, together with linear, left-to-right ordering at any level of the tree to indicate grammatical role. In the PTB example, the nesting of the NP "the front room" implies that the NP is the object of the prepositional phrase, whereas the position of the NP "him" following and at the same level as the VP node implies that this NP is the grammatical object. When it is necessary to specify a relation among non-contiguous elements, any of a variety of ad hoc mechanisms may be employed (such as the node numbering scheme in the PTB example). In contrast, pure dependency annotation schemes, such as the scheme described in Carroll, Minnen and Briscoe (forthcoming), specify grammatical relations among elements explicitly; for example, the sentence "Paul intends to leave IBM" could be represented as shown in Figure 2, where the predicate is the relation type, the first argument is the head, the second the dependent, and additional arguments may provide category-specific information (e.g., introducer for prepositional phrases, etc.). Although dependency schemes do not rely on hierarchical nesting to indicate relations, an independently specified relation hierarchy is typically defined which enables construction of a syntax tree from the dependency annotation.

A model for syntactic annotation

The goal in the XCES is to provide a framework for annotation that is theory and tagset independent. We accomplish this by treating the description of any specific syntactic annotation scheme as a process involving several knowledge sources that interact at various levels. The process allows one to specify, on the one hand, the informational properties of the scheme (i.e., its capacity to represent a given piece of information), and, on the other, the way the scheme can be instantiated (e.g., as an XML document).

Figure 3 shows the overall architecture of the XCES framework for syntactic annotation.

Non-XML Encoding

Universal Resources Project Specific Resources

Recommendations for Syntactic Annotation of Corpora [START_REF] Leech | EAGLES Recommendations for the Syntactic Annotation of Corpora[END_REF]. Data categories are defined using RDF schema that formalize the properties associated with each. The categories are organized in a hierarchy, from general to specific. For example, a general dependent relation may be defined, which may be defined to have one of the possible values argument or modifier; argument in turn may have the possible values subject, object, or complement; etc.4 Note that RDF schemas function much like class definitions in an object-oriented programming language: they provide, effectively, templates that describe how objects may be instantiated, but do not constitute the objects themselves. Thus, in a document containing an actual annotation, several objects with the type argument may be instantiated, each with a different value. The RDF schema ensures recognition that each instantiation of argument is a sub-class of dependent and inherits the appropriate properties.

The project-specific format for the annotation scheme, in terms of its expressive power and its instantiation in XML is defined in the two other knowledge sources: Data Category Specification (DCS): describes the set of data categories that can be used within a given annotation scheme, again using RDF schema. The DCS defines constraints on each category, including restrictions on the values they can take (e.g., "text with markup"; a "picklist" for grammatical gender, or any of the data types defined for XML), restrictions on where a particular data category can appear (level in the structural hierarchy). The DCS may include a subset of categories from the DCR together with application-specific categories additionally defined in the DCS. The DCS also indicates a level of granularity based on the DCR hierarchy.

Dialect specification: defines, using XML schemas, XSLT scripts, and XSL style sheets, the project-specific XML format and labels for syntactic annotations. The specifications may include:

Data category instantiation styles: Data categories may be realized in a project-specific scheme in any of a variety of formats. For example, if NounPhrase is the value of a complex data category Cat, Cat may be realized as an element (<cat>nom</cat>), an attribute (cat="NP" attached to an anchoring element), a typed element (e.g., <gram type="cat">NP</gram>), a valued element (<cat value="NP"/>, etc. Data category vocabulary styles: Project-specific formats can utilize names different from those in the Data Category Registry; for instance, a DCR specification for NounPhrase can be expressed as "NP" or "SN" ("syntagme nominal", in French) in the project-specific format, if desired. Expansion structures: A project-specific format may alter the structure of the annotation as expressed using the structural skeleton. For example, it may be desirable for processing or other reasons to create two sub-nodes under a given <struct> node, one to group features and one to group relations (see section 3.1).

The combination of the structural skeleton and the DCS defines a virtual annotation markup language (AML). Any information structure that corresponds to a virtual AML has a canonical expression as an XML document; therefore, the inter-operability of different AMLs is dependent only on their compatibility at the virtual level. As such, virtual AML is the hub of the annotation framework: it defines the pivot format for syntactic annotations that can be used to compare annotation schemes as well as enable the design of generic tools for visualization, editing, extraction, etc.

The combination of a virtual AML with the Dialect specification provides the information necessary to automatically generate a concrete AML representation of the annotation scheme, which conforms to the project-specific specification provide in the Dialect specification. XSLT filters translate between the representations of the annotation in concrete and virtual AML, as well as between non-XML formats (such as the LISP-like PTB notation) and concrete AML.

The structural skeleton

For syntactic annotation, we can identify a general, underlying model that informs current practice: specification of dependency relations (with some set of application-specific names and properties) among a set of grammatical constituents (also with a set of application-specific names and properties), whether this is modeled with a tree structure or the relations are given explicitly.

The most common representation for syntactic annotation in both treebanks and parser output based on manually-developed generative grammars is the phrase-structure tree. Because documents marked in XML are also hierarchically-structured, we provide a structural skeleton for syntactic markup that follows this approach. The skeleton consists of the following elements:

<struct> represents a node (level) in the tree. <struct> elements may be recursively nested at any level to reflect the structure of the corresponding syntax tree. In the virtual AML, <struct> elements are not typed (i.e., associated via attributes with specific data categories such as Sentence, NounPhrase, etc.), although project-specific XML schema can provide this information. <feat> (feature) is used to provide information attached to the node in the tree represented by the enclosing <struct> element. A type attribute on the <feat> element identifies the data category of the feature. The tag may contain a string that provides an appropriate value for the data category (e.g., for type=CAT the value might be "NP") or may point via a target attribute to an object in another document that provides the value. Note that this allows the possibility for including not only simple values but also complex data items as annotations. It also enables generating a single instantiation of an annotation value in a separate document that can be referenced as needed within the annotation document. <brack> is a general-purpose bracketing element to group associated features; it can be recursively nested to describe complex structures.

<alt> provides for encoding one or more alternative annotations (e.g., attachment of a PP to the verb and to the preceding object, where ambiguous or undetermined). Together, <feat>, <brack>, and <alt> can be used to encode simple or complex feature structures. <rel> is used to identify dependencies explicitly, by pointing to a <struct> element to which the current node is related. The <rel> element has the attributes type, head, dependent, introducer, which reference the relevant strings or structures, and initial, which specifies a thematic or semantic role. <seg> points to the data to which the annotation applies. In XCES, we recommend the use of stand-off annotation, i.e., annotation that is maintained in a document separate from the primary (annotated) data. The <seg> element would use the xlink attribute, which would use the XML Path Language, XPath [START_REF] Clark | XML Path Language (XPath)[END_REF] to specify the location of the relevant data. Figure 4 shows the annotation from the PTB example (Figure 1) rendered in the abstract XML format. Our encoding makes explicit relations that were implicit in the original version; however, implicit relations could be made explicit via an XML schema rather than encoding them explicitly. A strict dependency annotation can be encoded using a flat hierarchy (i.e., a single enclosing <struct> element) and specifying all relations explicitly with the <rel> tag, as shown in Figure 5. For the sake of readability, this encoding assumes that the sentence is marked up as: <s id= "s1"> <w id="w1">Paul</w> <w id="w2">intends</w> <w id="w3">to</w> <w id ="w4">leave</w> <w id="w5">IBM</w> </s>.

Note that we encode explicitly here much information that will ultimately be specified using RDF, which provides built-in support for linkage with labeled relations (links). In particular, information currently specified using the <feat> element can be represented as a 3-tuple (resource, property, value) that indicates that the resource (identified by ID=s1 in the encoding above) has the property CAT with value NP, etc. Similarly, information represented in the encoding above using <rel> elements and the ref attribute could also be expressed in RDF. Because universal web-based support for RDF is still under development, we provide alternative XML formats for the interim.

<struct xml:base="http://www.loria.fr/doc.xml#"> <struct id="s0"> <feat type="CAT">S</feat> <struct id="s1"> <feat type="CAT">NP</feat> <rel type="SBJ" head="s2"/> <seg xlink:href="xptr(substring(/p/s[1]/text(),1,5))"/></struct> <struct id="s2"> <feat type="CAT">VP</feat> <seg xlink:href="xptr(substring(/p/s[1]/text(),7,8))"/> <struct> <feat type="CAT">NP</feat> <rel type="OBJ" head="s2"/> <!--implicit in original --> <seg xlink:href="xptr(substring(/p/s[1]/text(),16,3))"/></struct> <struct> <feat type="CAT">PP</feat> <rel type="DIR" head="s2"/> <seg xlink:href="xptr(substring(/p/s[1]/text(),20,4))"/> <struct> <feat type="CAT">NP</feat> <seg xlink:href="xptr(substring(/p/s[1]/text(),25,14))"/></struct> <struct> <feat type="CAT">S</feat> <rel type="ADV" head="s2"/> <struct ref="s1"> <feat type="CAT">NP</feat> <rel type="SBJ" head="s3"/></struct> <struct id="s3"> <feat type="CAT">VP</feat><struct>…

Figure 4 : The PTB example encoded according to the structural skeleton <struct> <rel type="subj" head="w2" dependent="w1"/> <rel type="xcomp" head="w2" dependent="w4" introducer="w3"/> <rel type="subj" head="w4" dependent="w1"/> <rel type="dobj" head="w4" dependent="w5"/> </struct>

4

Comparing schemes

The Virtual AML provides a pivot format that enables direct comparison of annotations in different formats-including not only different constituency-based annotations, but also constituency-based and dependency annotations. For example, the PTB annotation corresponding to the dependency annotation in Figure 2 is shown in Figure 6. Figure 7 gives the corresponding encoding in the XCES abstract scheme. Relations are encoded only when they appear explicitly in the original annotation; an XML schema specifies the relations implicit in the embedding (e.g., indicating that the first VP is the head, etc.) From the encoding in Figure 7, an XSLT script5 produces a dependency encoding virtually identical to the one in Figure 5, thus enabling direct comparison of the two annotations. The only differences between the two would be in relation labels (e.g., XCOMP), which would be mapped via the Dialect Specification to the same category in the Data Category Registry and therefore recognized as equivalent.

The ability to generate a common representation for different annotations overcomes several obstacles that have hindered evaluation exercises in the past. For instance, the evaluation technique used in the PARSEVAL exercise is applicable to phrase-structure analyses only, and cannot be applied to dependency-style analyses or "lexical" parsing frameworks such as finite-state constraint parsers. As the example above shows, this problem is eliminated using the XCES framework. It has also been noted that that the PARSEVAL bracket-precision measure penalizes parsers that return more structure than exists in the relatively "flat" treebank structures, even if they are correct (Srinivas, et al., 1995). XSLT scripts can extract the appropriate information for comparison purposes while retaining links to additional parts of the annotation in the original document, thus eliminating the need to "dumb down" parser output in order to participate in the evaluation exercise. Similarly, information lost in the transduction from phrase-structure to a dependency-based analysis (as in the example above), which, as [START_REF] Atwell | Comparative evaluation of grammatical annotation models[END_REF] points out, may eliminate grammatical information potentially required for later processing, can also be retained.

Of course, one of the greatest obstacles to the comparison and evaluation of syntactic annotations remains the difficulty of mapping schemes that are often based on different theories and developed for very different purposes. At the least, our attempt to develop an abstract model that captures the fundamental properties of syntactic annotation schemes will serve, we hope, as a conceptual tool for assessing the coherence and consistency of existing schemes as well as newly developed ones, which is an essential step toward harmonization. The requirement that all relations are made explicit, either in the encoding or via an XML schema that describes them, can lead to the discovery of gaps and inconsistencies in existing schemes. 6 Hand-developed annotation schemes used in treebanks are often described only informally in guidebooks for annotators, often resulting in virtual anarchy; for example, [START_REF] Charniak | Tree-bank grammars[END_REF] notes that the PTB implicitly contains more than 10,000 context-free rules, most of which are used only once. Mappability of schemes becomes virtually impossible under such circumstances. While requiring that annotators make relations explicit and consider the mapping to the XCES abstract format increases overhead, we feel that the exercise will help avoid such problems and can only lead to greater coherence, consistency, and inter-operability among annotation schemes.

The most important contribution to inter-operability of annotation schemes is the Data Category Registry. By mapping site-specific categories onto definitions in the Registry, equivalences (and non-equivalences) are made explicit. Again, the provision of a "standard" set of categories, together with the requirement that scheme-specific categories are mapped to them where possible, can only contribute to greater consistency and commonality for annotation schemes.

Conclusion

The framework presented here for syntactic annotation is intended to allow for variation in annotation schemes while at the same time enabling comparison and evaluation, as well as the development of common tools for creating and using annotated data. Despite its seeming complexity, the framework is also intended to reduce overhead for annotators and users. Part of the work of the XCES is to provide XML support (e.g., development of XSLT scripts, XML schemas, etc.) for use by the research community, thus eliminating the need for XML expertise at each development site. Also, with XML at its core, the XCES framework builds on emerging standards for data interchange for which there is widespread support; common scripts and tools can be developed, reducing the "reinventing of the wheel" that is commonplace in the area of corpus annotation at present. Because XML-encoded annotated corpora are increasingly used for interchange between individual processing and their formats are rendered in the abstract format. analytic tools, for commonly used tools we are developing XSLT scripts for mapping, and extraction of annotated data, import/export of (partially) annotated material, and integration of results of external tools into existing annotated data in XML. Tools for editing annotations in the abstract format and for automatically generating virtual AML from Data Category and Dialect Specifications are already under development in the context of work on the Terminological Markup Language; as XML use becomes more widespread, more and more reusable tools and resources (including web-based applications) will become available.

<struct id="s0"> <feat type="CAT">S</feat> <struct id="s1"> <feat type="CAT">NP</feat> <rel type="SBJ" head="s2"/> <seg target="w1"/></struct> <struct id="s2"> <feat type="CAT">VP</feat> <seg target="w2"/></struct> <struct id="s3"> <feat type="CAT">S</feat> <struct id="s4" ref="s1"> <rel type="SBJ" head="s6"/></struct> <struct id="s5"> <feat type="CAT">VP</feat> <seg target="w3"/> <struct id="s6"> <feat type="CAT">VP</feat> <seg target="w4"/> <struct> id="s7"> <feat type="CAT">NP</feat> <seg target="w5"/></struct> </struct></struct></struct>

Figure 2 :

 2 Figure 2 : Dependency annotation according to Carroll, Minnen, and Briscoe

Figure 3 :

 3 Figure 3 : Overall architecture of the XCES annotation framework Two universal knowledge sources are used define the abstract model: Structural Skeleton: a domain-dependent abstract structural framework for syntactic annotations, capable of capturing all the information in a specific annotation scheme. The structural

Figure 5 :

 5 Figure 5 : Abstract XML encoding for the dependency annotation in Figure 2.

Figure 6 :

 6 Figure 6: PTB annotation of "Paul intends to leave IBM."

Figure 7 :

 7 Figure 7 : PTB encoding of "Paul intends to leave IBM."

Cf. the hierarchy in Figure1.1, Caroll, Minnen, and Briscoe (forthcoming).

The XML schema and XSLT script are not reproduced here due to space constraints.

Indeed, several annotation scheme developers have been surprised to see inconsistencies revealed when