Loïc Maisonnasse
email: loic.maisonnasse@imag.fr

Gilles Sérasset

Jean-Pierre Chevallet

Using Syntactic Dependency and Language Model X-IOTA IR System Used for CLIPS Mono & Bilingual Experiments for CLEF 2005

This document describes the CLIPS experiments done for the CLEF 2005 campaign. We use surface-syntactic parser in order to extract new indexing terms. These terms are syntactic dependencies. Our goal is to evaluate their interest for an information retrieval task. We used them under different forms in different information retrieval models, particularly in a language model.

For the bilingual part we tried two simple tests on Spanish and German to French evaluation, for the translation we use a lemmatization and a dictionary.

Introduction

In the previous participation of the laboratory CLIPS at CLEF [START_REF] Sérasset | Using surface-syntactic parser and Deviation from Randomness. X-IOTA IR system used for CLIPS Mono & Bilingual Experiments for CLEF[END_REF], we promote the use of surface-syntactic parsers in order to extract indexing terms. Last year, we only extracted simple indexing terms, this year we have tried to exploit the structure extracted by the parser. By this, we have made two different evaluations; in the first one, we divided the structure extracted into complex descriptors, which contains a part of the global structure. In the second one, we use the syntactic dependencies between lemmas extracted by the shallow parser in a language model.

Sub-structure training on monolingual run

In this part, we evaluate the use of sub-structures as indexing terms. The shallow parser produces a structure, by using only the lemmas as we did last year, we only use a part of the extracted information. This year we evaluate the interest of the structural information produced by the parser. Different type of parser are available, in this paper we use a dependency parser as that kind of parser seems to be more appropriate for the information retrieval task [START_REF] Koster | Head/Modifier Frames for Information Retrieval[END_REF]. Different works have already been made to use syntactic dependency structures. Some of those works use the dependency structure in order to extract phrases. For example, in [START_REF] Matsumura | The effect of information retrieval method using dependency relationship between words[END_REF] the author produces a structure close from a dependency tree for all documents sentences. After that, he applies some patterns on the structure for extracting phrases then some selected phrases are added to the others descriptors in the document index. At last, the tf-idf weighting schema is adjusted in order to give a higher idf for the extracted phrase. This way, a 20% gain on the average precision is provided on information retrieval results. Nevertheless, this gain cannot be directly linked with the use of a dependency structure.

On the presumption that converting the structures to phrases leads to lose information, others works have tried to use directly the syntactic dependency structure. In [START_REF] Matsumura | The effect of information retrieval method using dependency relationship between words[END_REF], a dependency tree is extracted from each Japanese sentence, mainly on the documents titles. The matching between a query and the documents is provided by a projection matching of the query tree on the documents trees. In addition, to provide, a better matching some cut can be made on the tree. In [START_REF] Metzler | The Constituent Object Parser: Syntactic Structure Matching for Information Retrieval[END_REF], the COP parser (Constituent Object Parser) is used to extract dependency trees. In the query the user has to select important terms and to indicate dependencies between them, the query is then compare to the documents by different types of matching. As the two preceding works provided only one unambiguous structure by sentence, in [START_REF] Smeaton | Using NLP or NLP resources for information retrieval tasks[END_REF] the author incorporates syntactic ambiguity in the extracted structure. His model is apply on phrases, the similarity is provided by tree matching but the IR results are lower than the results obtained by considering only the phrases represented in the tree.

In our evaluation, we consider an intermediary representation level. For this purpose, we use sub-structures, which are composed by one dependency relation. By this representation a sentence is consider as a set of substructures that we call dependencies. In our formalism, the representation of the sentence "the cat eats the mouse" is represented by the set: {DET(the, cat), DET(the, mouse), SUBJ(cat, eat), VMOD(mouse, eat)}. Where "the" is the determiner of "cat", "cat" is the subject of "eat", etc. In this representation we lose only the information that there is two "the" in the sentence.

Experimentation schema

For this experiment, we only used the French corpus. On this corpus, we experiment the use of the dependency descriptor. For this purpose, we use an experimental sequence, described in the following schema: First, the different documents of the collection are analysed with the French parser XIP (Xerox Incremental Parser) [START_REF] Ait-Mokhtar | Robustness beyond shallowness: incremental deep parsing[END_REF] from these documents two descriptors are extracted: the dependencies and the lemmas. In a first experiment, we consider these descriptors separately and create two indexes. One contains the lemmas and the other the dependencies. We query these two indexes separately with the dependencies and the lemmas extracted from the queries by the same parser. We compare the results obtain with the two descriptors for different weighting schemes. In a second experiment, we regroup the two descriptors in the same index and we evaluate the results for different weightings schemes.

For training, we have used the French corpus of CLEF 2003. In this corpus, there are 3 sets of documents. For each set, we have selected the following fields: TITLE and TEXT for "le monde 94", TI KW LD TX ST for "sda 94" and "sda 95". For the queries, we have selected the fields FR-title FR-descr Fr-narr.

Dependencies versus lemmas

We have first compared the result obtain by using dependencies to the result obtain with lemmas. In these experimentation lemmas are used as baseline as they have already show they usability in the CLIPS experiments last year [START_REF] Sérasset | Using surface-syntactic parser and Deviation from Randomness. X-IOTA IR system used for CLIPS Mono & Bilingual Experiments for CLEF[END_REF]. After parsing the documents with XIP, we have transformed the output into a common XML simplified format (Figure 2). From this XML, in one side we have extracted the lemmas: for these descriptors, we have filtered only nouns, proper nouns, verbs, adjectives and numbers. <LUNIT> <NODE num="2" tag="DET" lemma="le" ...>les</NODE> <NODE num="3" tag="NOUN" lemma="manifestation" ... >manifestations</NODE> <NODE num="5" tag="PREP" lemma="contre" ... >contre</NODE> <NODE num="7" tag="DET" lemma="le" ... >le</NODE> <NODE num="8" tag="NOUN" lemma="transport" ... >transport</NODE> <NODE num="10" tag="PREP" lemma="de" ... >de</NODE> <NODE num="12" tag="NOUN" lemma="déchet" ... >déchets</NODE> <NODE num="14" tag="ADJ" lemma="radioactif" ... >radioactifs</NODE> <NODE num="16" tag="PREP" lemma="par" ... >par</NODE> <NODE num="18" tag="NOUN" lemma="conteneur" ... >conteneurs</NODE> <NODE num="23" tag="SENT" lemma="." ... >.</NODE> <DEP name="NMOD" ... w0="déchet" w1="radioactif"/> <DEP name="NMOD" ... w0="manifestation" w1="contre" w2="transport"/> <DEP name="NMOD" ... w0="transport" w1="de" w2="déchet"/> <DEP name="NMOD" ... w0="déchet" w1="par" w2="conteneur"/> <DEP name="DETERM" ... w0="le" w1="manifestation"/> <DEP name="DETERM" ... w0="le" w1="transport"/> </LUNIT> On the other side, we extract the dependencies (Figure 3). As the number of dependencies can be very high, we query separately each documents set and then merge the results. We have compared IR results obtains with these two descriptors for different weighting schema. We used the following weighting schemes on the document and on the query descriptors:

For the documents nnn:

Only the term frequency is used.

lnc: Use a log on term frequency and the cosine is used as the final normalization.

ltc:

The classical tf*idf with a log on the term frequency.

nRn: Derivation from randomness

For the queries nnn:

Only the term frequency is used.

bnn:

The binary model, the terms presents are associated to the value 1, and 0 otherwise.

lnn:

A log is used on the term frequency.

npn: Idf variant used by okapi.

ntn: classical idf.

For more details, see [START_REF] Sérasset | Using surface-syntactic parser and Deviation from Randomness. X-IOTA IR system used for CLIPS Mono & Bilingual Experiments for CLEF[END_REF].

We have first evaluated the c coefficient for the derivation from randomness weighting (nRn) on the document and with an nnn weighting on the queries. Results for the two descriptors are shown on Over all weighting, the dependencies descriptor performs better than the lemmas only for the nnn weighting. The derivation from randomness performs better than the others documents weighting for the two descriptors and the result are stable connecting to the query weighting.

Lemmas and dependencies

As in a first experiment we have used the dependencies and the lemmas separately. In this part, we merged the two descriptors in one index and evaluated the different weighting schemes for that index.

As for the preceding experimentation, we first evaluate the derivation from randomness (Table 4) and then we evaluate the different weighting methods ( The results obtained in this evaluation are better than those obtain for the dependencies alone but they are lower than those obtain for the lemmas. The reason is that the dependencies and the lemmas are considered as equivalent descriptors when these two descriptors are clearly from two different levels as the dependencies contains the lemmas. This particularity was not taken in account in this experimentation. Nevertheless, as we want to evaluate the use of dependencies we have sent a run with nRn nnn weighting with both dependencies and lemmas for the monolingual run and with the coefficient c at 2.25.

Language model

In a second experimentation, we have integrated the structure extracted by XIP in a language model based information retrieval system. Some work have already been made in order to use dependencies between terms in a language model in [START_REF] Gao | Dependence language model for information retrieval[END_REF] [START_REF] Nallapati | Capturing term dependencies using a language model based on sentence trees[END_REF]. These works used statistical based methods in order to get the tree representation of a sentence; here we used a linguistically extracted structure.

In order to use a language model based on dependencies, we have parsed the different documents sets. We have transformed the output into the XML simplified format. From this XML, we have filtered only nouns, proper nouns, verbs, adjectives and numbers and the dependency that connect only these descriptors. For each sentence, we obtain a graph where the nodes are the significant elements of the sentence linked by dependencies (Figure 4). We have used these graphs to apply a language model.

Language model

The language model we used is a simplified version of the model proposed in [START_REF] Gao | Dependence language model for information retrieval[END_REF]. This model assumes that the query generation is formulated as a two-stage process. In a first time a linkage is generated from the document following the distribution P(L|D). In a second time the query is generated following the distribution P(Q|L,D), the query terms are generated at this stage according on the terms linked in L. So in this model the probability of the query P(Q|D) over all possible linkage Ls is :

= = Ls Ls D L Q P D L P D L Q P D Q P ) , ( ) ( ) , ( ) (
The authors assume that the sum

Ls D L Q P ) , (
over all the possible linkages Ls is dominated by a single linkage L, which is the most probable linkage. Here we consider that the most possible linkage L is those extract by our parser. The authors finally obtain:

manifestation transport déchets Radioactif conteneur = ∈ + + = m i L j i j i i D L q q MI D q P D L P D Q P .. 1 ) , ( ) , , ( ) ( )) ( log( ) ( Where ) ( ) ( ) , , ( log ) , , ( D q P D q P D L q q P D L q q MI j i j i j i

=

Here we use simple estimation of the different parameters.

P(L|D)

We estimate P(L|D) as the probability that two terms are linked if they appear in the same sentences in the document. On this estimation, we made an interpolation of the document probability with the collection probability. So we obtain the following formulation:

) , ( ) , , ( ) , ( ) , , ( ) 1 ( ) ( ) ( ) , ( j i j i L j i d j i j i d L l q q C R q q C q q D R q q D D l P D L P ∏ ∏ ∈ ∈ + - ∝ = λ λ
where l denotes a dependency between two terms ) , , ( R q q D j i denotes the number of time that q i and q j are linked in a sentence of the document ) , ( j i q q D denotes the number of time that q i and q j appear in the same sentence.

) , (

j i q q C , ) , , ( R q q C j i
denotes the equivalent number but evaluated on the whole collection.

P(q i |D)

We estimate P(q i |D) as the probability that a term appear in a document, and we made an interpolation on the collection.

) ( ) ( ) 1 ( ) ( C q P D q P D q P i l i l i λ λ + - =
In the two last estimation, if a lemma or a dependency does not appear in the collection the probability is set to zero, consequently the whole probability will be set to zero. To avoid this, in the query we consider only the dependencies and the lemmas found in the whole collection.

MI(qi, qj|L,D)

For this estimation, we use the same estimation as the one used in the article [START_REF] Gao | Dependence language model for information retrieval[END_REF].

) , (*, ) ,*, (

) (*,*, ) , , ( log ) , , ( R q
C R q C R C R q q C D L q q MI j i j i j i =
where C(x,y,R) denotes the count of link between the words x and y, the * symbolize all possible word.

Training

We apply this model on the Collection of CLEF 2003, the results obtain are presented in the following table where we evaluate the variation of the coefficients l and d . l / d Average precision 0,1/0,5 0.2749 0,2/0,5 0.2724 0,3/0,5 0.2697 0,4/0,5 0.2536 0,5/0,5 0.2495 0,6/0,5 0.2428 0,1/0,9999 0.2778 0,2/0,9999 0.2951 0,3/0,9999 0.2890 Table 6

: varation of l and d

We see that the results are better when the coefficient l is around 3 and when the coefficient d is high. So the results are better when the dependencies in the query are not take in account. This thinks may comes from the fact that we use simple estimations; better estimation on the probability may give better results.

We submit a run for this language model with the coefficient l at 0.3 and the coefficient d at 0.9999, the same experimental condition are used.

Bilingual

For the crosslingual evaluation, we have made two simple run from German and Spanish to French. For those two run, we use the three query fields : XX-title, XX-descr, XX-narr. And we query the French collection with the same fields as in the monolingual experiments. In this evaluation, the queries words are lemmatized and then we translate those lemma with the web dictionary interglot1 .

For the lemmatization, we use TreeTagger2 for the German queries and we use agme lemmatizer [START_REF] Gelbukh | Approach to construction of automatic morphological analysis systems for inflective languages with little effort[END_REF] for the Spanish queries. On these lemmatizers if there is an ambiguity, we conserve all the possible forms. We translate the lemmas with the dictionary and we keep all the translations found. At last, we query the French lemmas index with the derivation from randomness weighting on the documents.

On clef 2003, we obtain an average precision of 0.0902 for the German queries and an average precision of 0.0799 for the Spanish queries.

Results

Monolingual

For this evaluation, we sent three different runs. Two of those runs were based on dependencies with lemmas index with a weighting schema 'nRn nnn' with the coefficient c at 2.25. The first of those two run 'FR0' use only the fields FR-title FR-descr of the queries, the second 'FR1' use the all fields. The last run 'FR2'submitted for the French monolingual task used the language model described in section 3. We can see that as 'FR1' use the field 'FR-narr' for the query his results are lover than the run 'FR0' which doesn't use this field. This artefact may comes from the fact that we have not used a program that process the topic in order to remove unrelevant phrase as 'Les documents pertinents doivent expliquer' (the relevant documents must explains). We observe that the results obtain on CLEF 2005 are lower than those obtain on CLEF 2003 specially when we used the three query fields the result for CLEF 2005 are more than two times lover than the results for CLEF 2003. This result may come from the fact that the narrative part of the queries seems to be shorter in CLEF 2005. Another difference could be seen between 'FR1' and 'FR2' as these two runs show a difference of about

Bilingual

For this experiment, we send two runs for each source language. One of those two run used the topic field XXtitle and XX-descr. The second use in addition of those two fields the field XX-narr. de

Conclusion

For our participation at clef 2005 we began to evaluate the use of syntactic dependency structures extracted by a parser in an information retrieval task. In our first experiment, we tried to exploit the structure by using descriptors that capture a part of the structure. In our second experiment, we exploited directly the structure extracted by the parser in a language model. The two experiments show that the structure is exploitable, but the results are still lower than those obtained using only lemmas with appropriate weightings.

As the syntactic structure has shown to be exploitable in IR some improvements could be applied on this model. We used here the XIP parser, but this parser does not give information on the quality of the structure. Integrating this kind of information on the dependencies extracted could improve the IR results. Using a parser, that extract deeper syntactic dependencies may give better results for the information retrieval task. At last, our language model uses simple estimations, better estimation may improve the results.

Our conviction is that detailed syntactic information, which is already available using existing parsers, is to improve results (especially, precision) in information retrieval tasks. However, such detailed information has to be combined with classical descriptors as, taken alone, it does not show better results. Obviously, we still have to find ways to combine the advantages of classical, raw descriptors with the added value of fine grain syntactic information in a single model. Independently of the task, we see that using the narrative part of the queries lowers our results for the next participation in order to improve our results we have to make an module that selects only the important part of the topic.

Figure 1 :

 1 Figure 1: experimental procedure

Figure 2 :Figure 3 :

 23 Figure 2: XML Information for the sentence : "les manifestations contre le transport de déchets radioactifs par conteneurs." (Demonstrations against the transport of radioactive waste by containers)

Figure 4 :

 4 Figure 4: structure used by the langue model for the sentence: "les manifestations contre le transport de déchets radioactifs par conteneurs en Allemagne."

Table 1

 1

	c	Average precision
	0	4.89	c	Average precision
	2	25.53	0	0.0152
	3	25.50	0,5	0.4362
	4	25.83	1	0.4647
	4.25	25.93	1,75	0.4700
	4.5	26.01	1,5	0.4703
	4.75	26.00	2	0.4687
	5	25.88	2,25	0.4728
	5.5	25.84	2,5	0.4709
	6	25.84	3	0.4577
	10	25.64

and on Table

2

. After that we have evaluated the others weighting methods, results are presented on Table

3

.

Table 1 : Variation of c for nRn nnn (dependencies alone) Table 2 variation of c for nRn nnn (lemmas alone)Table 3 : lemmas or dependencies average precision

 13

Table 5

 5

	).
	c	Average precision
	0	0,0207
	1	0,3798
	1,5	0,3941
	2	0,3947
	2,25	0,3947
	2,5	0,3934
	3	0,3922

Table 4 : Variation of c for nRn nnn (lemmas and dependencies)

 4

						Query weighting
	Document			lemmas				dependencies
	Weighting	nnn	bnn	lnn	npn	ntn	nnn	bnn	lnn	npn	ntn
	nnn	1,82	0,81	1,57	21,43	16,43	9,01	5,56	8,16	18,21	17,96
	lnc	35,02	31,27	36,22	34,30	37,46	18,92	17,46	19,17	21,93	21,94
	ltc	33,13	33,93	35,94	32,86	33,79	21,14	18,94	20,86	21,66	21,66
	nRn	47,28	38,34	45,55	45,23	48,35	26,01	22,56	25,90	24,95	24,94

Table 5 : lemmas and dependencies average precision

 5

Table 8 : bilingual results

 8

		->fr			es->fr
				Title+
		title+ desc title+desc+narr	desc	title+desc+narr
	average precision	6.88	4.98	4.23	3.69
	precision at 5 documents	17.20	12.80	10.80	11.60

http://interglot.com/

http://www.ims.uni-stuttgart.de/projekte/corplex/TreeTagger/DecisionTreeTagger.html


Jean Martinet
email: jean.martinet@imag.fr

Yves Chiaramella
email: yves.chiaramella@imag.fr

Philippe Mulhem
email: philippe.mulhem@imag.fr

A Model for Weighting Image Objects in Home Photographs

Keywords: H.3.1. [Information Storage and Retrieval]: Content Analysis and Indexing -Indexing methods Experimentation, Human Factors, Standardization, Theory Image indexing, Weighting scheme, Photograph indexing

The paper presents a contribution to image indexing consisting in a weighting model for visible objects -or image objects -in home photographs. To improve its effectiveness this weighting model has been designed according to human perception criteria about what is estimated as important in photographs. Four basic hypotheses related to human perception are presented, and their validity is estimated as compared to actual observations from a user study. Finally a formal definition of this weighting model is presented and its consistence with the user study is evaluated.

INTRODUCTION

Content based image retrieval systems exist now for more than 15 years [START_REF] Smeulders | Content-based image retrieval at the end of the early years[END_REF]. Such a research domain faces many cross-community problems, ranging from vision problems, interaction problems and information retrieval problems. Most of the well known approaches (QBIC [START_REF] Faloutsos | Efficient and Effective Querying by Image Content[END_REF], NETRA [START_REF] Ma | NETRA: A Toolbox for navigating large image databases[END_REF], BLOBWORLD [START_REF] Carson | Blobworld: Color-and Texture-Based Image Segmentation Using EM and Its Application to Image Querying and Classification[END_REF]) for CBIR come from computer vision, because this domain tackles many of the fundamentals on which the IR community can build effective models for Content Based Image Retrieval (CBIR). Despite the number of interesting contributions to CBIR systems, little attention has been dedicated to the evaluation of the importance of visual elements appearing in images for improving retrieval effectiveness. This is precisely the core problem addressed in this paper which presents a weighting scheme for image objects. The starting point of this research might be related to the well-known problem of weighting terms in textual indexing. The classical tf.idf weighting scheme [START_REF] Salton | The SMART Retrieval System -experiments in automatic document processing[END_REF] for indexing terms has long ago proved its effectiveness when applied to textual documents, where tf (term frequency) is an estimate of the relative importance of a given term as a content descriptor of a particular document, and idf (inverse document frequency) measures the discriminating power of the term, or its ability to differentiate a document from the others.

If visible objects may be considered as the main content components of images or photographs (like words or phrases in textual information), then what would be the image equivalent of tf? Furthermore, would the classical tf.idf weighting model be also effective in the context of image retrieval? This is the core problem addressed in the context of this study. Due to space constraints we will concentrate here on the problem of providing and experimenting a definition of tf suitable for images.

Our goal then is not about how visible elements can be extracted from images, but to evaluate how important such image elements are when considering image relevance.

According to our definition an Image object (IO) is a 2D representation of a visible, real object which is part of the scene displayed in a photograph. Each image object IO is then named according to a term t of an indexing vocabulary suited for the application. The question is then to assign a weight to this t-IO relation to evaluate to what extent the image containing this IO is relevant for a query about t. In our approach to image indexing such image components are intended to provide a basis for image symbolic indexing, but also potentially to feature-based indexing (or signal indexing) of images.

The notion of occurrence -now very classical when considering weighting of indexing terms for textual documents -is much less intuitive in the case of images. Images are 2D data and in this context one may understand that the relevance of an image showing boats is not only related to the fact that it represents one or more distinct boats. Other 2D perceptive factors are possibly more important in that matter, such as for example the spatial position or the size of these visible boats.

In this study we have investigated four such criteria which are all related to 2D geometry: size, position, fragmentation and heterogeneity. These hypotheses are presented and discussed in section 3.1. Before being included in a relevance model for images these hypotheses need to be confronted to actual human perception and relevance assessments. Section 3.2 describes an experimental validation of this model based on a user study which Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CIKM'05, October 31-November 5, 2005, Bremen, Germany. Copyright 2005 ACM 1-59593-140-6/05/0010…$5.00.

clearly demonstrates that three of these four criteria have a definite impact on the notion of image relevance: size, position and heterogeneity. We have then got some effective clues on which facts an image equivalent of the textual tf could be based. A first attempt towards a formal definition of the three criteria, which is intended to reflect the observed human behaviour about image relevance, is then given in section 4.1. The model is then completed by proposing a combination of these three relevant criteria as a "tf" equivalent for weighting image objects. Section 4.2 presents an experiment demonstrating that this weighting schema is consistent with observed users' relevance assessments. Finally a conclusion on this study and indications about future work is given in section 5.

RELATED WORKS

Well known CBIR systems such as QBIC [START_REF] Faloutsos | Efficient and Effective Querying by Image Content[END_REF], Blobworld [START_REF] Carson | Blobworld: Color-and Texture-Based Image Segmentation Using EM and Its Application to Image Querying and Classification[END_REF] and Netra [START_REF] Ma | NETRA: A Toolbox for navigating large image databases[END_REF] for instance have already included weighting models aimed to enable ranking system's responses. In these systems, whenever the query is based on image example or on explicit description of signal features (e.g. color), the image index contains "pure" signal values such as color histograms for instance. These were never clearly confronted to what users consider important or not important in images. These weighing schemas are more related to a global evaluation of image relevance.

Considering for example the color histogram of an image query, the values defining the histogram do indicate if a particular bin is larger than another. Common histogram distances (Euclidian or other L-norms, quadratic histogram distances used in QBIC) are based on such comparisons of corresponding bins. But then a question might be for example: is the difference involving the larger bin of the image as important (regarding image relevance) as the difference involving its smaller bin?

In fact, all these state-of-the-art approaches bypass one of the central ideas of Information Retrieval since 30 years stating first that effective indexing terms are related to explicit concepts (i.e. they may be denoted based on human language) and second that each term has an associated value -commonly called term weight -expressing its capacity to effectively retrieve documents among a document collection. A weighting schema such as tf.idf clearly tries to capture and combine in a single measure two basic properties of every indexing term: to which extent it accounts for the actual content of a given document, and its ability to discriminate this document among the corpus. This classical approach for term weighting is quite drastically different from pure signal-based similarity (or related distances) measures underlying CBIR. At the moment there are quite few definite and experimentally validated clues about how evolving from color histograms or texture similarity measures to more effective retrieval models integrating the user-based notion of relevance. This in turn refrains CBIR researchers to tackle this basic problem.

Instead of trying to design effective weighting models based on signal analysis and similarity, we choose to focus on a symbolic approach where images are not indexed based on their visual features but based on terms (ie. denoting concepts) providing interpretations of visual elements appearing in the image. Such kinds of approaches exist, at least partially. Some of them define an explicit labeling (via indexing terms) of image regions [START_REF] Fan | Automatic image annotation by using concept-sensitive salient objects for image content representation[END_REF][START_REF] Lim | Building visual vocabulary for image indexation and query formulation[END_REF][START_REF] Town | Content-based image retrieval using semantic visual categories[END_REF][START_REF] Wang | SIMPLIcity: Semantic-Sensitive Integrated Matching for Picture Library[END_REF] based on a learning process. Others also consider regions in the analysis phase but the final labeling process associates terms to the images instead of the corresponding regions [START_REF] Jeon | Automatic image annotation and retrieval using cross-media relevance models[END_REF]. Finally, others globally analyze images and assign them indexing terms as a whole, completely ignoring the notion of image regions [START_REF] Bradshaw | Semantic-based image retrieval: a probabilistic approach[END_REF].

Approaches indexing images as a whole are usually based on binary classification schemes like for example "indoor" and "outdoor" [START_REF] Bradshaw | Semantic-based image retrieval: a probabilistic approach[END_REF][START_REF] Minka | Interactive learning using a "society of models[END_REF][START_REF] Mojsilovic | Capturing Image Semantics With Low-level Descriptors[END_REF] which is much closer to the extraction of meta-information than to semantic-content indexing. When considering image indexing based on image elements, approaches vary again from binary labels [START_REF] Town | Content-based image retrieval using semantic visual categories[END_REF][START_REF] Wang | SIMPLIcity: Semantic-Sensitive Integrated Matching for Picture Library[END_REF] to n-ary ones [START_REF] Lim | Building visual vocabulary for image indexation and query formulation[END_REF][START_REF] Paterno | Fuzzy Semantic Labeling for Image Retrieval[END_REF], this later one being the closest to our approach. As far as we are concerned with information retrieval we consider that including a weighting model in this approach is preferable.

The question then is the design of such a model: what weights to assign to image components, and according to what criteria? Existing models are based on location [START_REF] Lim | Building visual vocabulary for image indexation and query formulation[END_REF][START_REF] Wang | SIMPLIcity: Semantic-Sensitive Integrated Matching for Picture Library[END_REF], on relative surfaces [START_REF] Lim | Home Photo Content Modeling for Personalized Event-Based Retrieval[END_REF], on size, like the area percentage scheme of Simplicity [START_REF] Wang | SIMPLIcity: Semantic-Sensitive Integrated Matching for Picture Library[END_REF] and [START_REF] Ounis | Finding the Best Parameters for Image Ranking: a User-Oriented Approach[END_REF], or on keyblock term frequencies [START_REF] Zhu | Theory of keyblock-based image retrieval[END_REF].

Although interesting and sometimes close to our approach, none of these models has been validated through users' experiments or is complete regarding weighting individual objects. Consequently most of them remain somewhat intuitive or incomplete. As an example of this situation, Jing and al propose in [START_REF] Jing | An Efficient end Effective Recgion-Based Image Retrieval Framework[END_REF] a weighting model based on relevance feedback techniques and write in section 5 "… such as the area percentage, is probably not completely relevant to the semantic importance of regions". Here the word "probably" is important because they do not provide experimental evidence that confirm or infirm their hypothesis (which by the way is infirmed by our experiments!). More generally, none of these models refers to a relevance-based weighting model such as tf.idf and try to experiment whether it might be adapted to image retrieval or not. When looking at the basic principles underlying this model a first problem is then: what would be a tf equivalent for images, or how to determine to what extent an image is "about" a given term? This is the basic problem we want to address here.

Considering this goal and the state of the art related to symbolic indexing of images, we are convinced that what is needed first is to define and to experimentally validate a set of hypothesis about human perception of image relevance. This constitutes the main topic of section 3 below.

IMAGE OBJECTS AND IMAGE ABOUTNESS 3.1 Hypotheses

What do users actually consider as significant in images, and how providing a model reflecting their behavior about image relevance? According to previous experiments in the context of retrieving non-specific images1 [START_REF] Hollink | Classification of user image descriptions[END_REF] and related to home photographs [START_REF] Lim | Building visual vocabulary for image indexation and query formulation[END_REF][START_REF] Mulhem | Advances in Digital Home Image Albums[END_REF], users tend describing images based on visible objects appearing in them. This observation explains why our approach focuses on "visible objects" and not on color regions for instance. It explains also why such visible objects are considered as basic entities in our weighting model.

To be more precise about this important aspect we introduce the following notion of Image Objects (IO). This definition is inspired from the one given in [START_REF] Mechkour | EMIR2. An Extended Model for Image Representation and Retrieval[END_REF]:

- (a)
An IO is related to one and only one semantic interpretation which is defined by a label. A label stands here for an indexing term defining the semantic interpretation of the related IO. In its general definition a label may belong to any kind of indexing language (as part of a particular indexing model). Considering images, such labels usually are natural language words ("tree", "house", "sky", etc.). From now on we will use this term label instead of indexing term.

-(b) An IO is defined as a bi-dimensional projection (or visualization) of one real-world, physical, object (or possibly of several such objects sharing the same label) in an image. An IO then often corresponds to a single image region2 within an image. But according to this definition, an IO may also correspond to several nonconnected image regions if it corresponds to several occurrences of the same type of physical objects (e.g. several "boats"). This possibility has also been extended to the frequent case where the real-world physical object is partially occluded (like a building behind a tree for instance). We will refer further to these two situations as fragmentation cases of an IO.

We define the notion of importance of a given IO relatively to the notion of image content in a much similar way to the standard notion of term importance in the context of text retrieval.

Definition: Considering an image I, and io an image object of I, io is uniquely labeled by a term t (definition (a) above). The importance of io regarding I is directly related to the aboutness3 of I considering t.

For example, one will consider that an IO representing a boat (hence t = "boat") in a given image is important if most users would consider that this image is about a "boat".

Let us consider now our proposed hypotheses about image objects importance. Being associated to an image region, or a set of image regions, an IO is then basically a 2D set of pixels whose basic geometrical characteristics (area, position, etc.) can be easily computed. According to the weighting models reported above in the related works, we propose to define and investigate four hypotheses:

1 Size -Some previous approaches mentioned in the state of the art have quoted the importance of the size criterion [START_REF] Lim | Home Photo Content Modeling for Personalized Event-Based Retrieval[END_REF][START_REF] Wang | SIMPLIcity: Semantic-Sensitive Integrated Matching for Picture Library[END_REF], in which the authors assume a direct relationship between an image object importance and its relative size (visible surface). By definition, the size S of an IO is defined here as the size of its related region(s).

Size Hypothesis: The importance of an image object varies in the same way as its size S.

2 Position -Again some previous approaches mentioned in the previous section ( [START_REF] Lim | Building visual vocabulary for image indexation and query formulation[END_REF]) have quoted the potential importance of this criterion. We share this opinion and want to investigate possible formulations and their experimental validation. Here the position of an IO is determined by the position of its barycenter.

Position hypothesis: the importance of an image object is maximal when its position P is at the center of the image, and decreases when its distance from the image center increases.

3

Fragmentation -To our knowledge, the possible impact of fragmentation on the importance of an IO has never been investigated. We thought that it could indeed have such an impact, when compared to a completely connected, appearance of the same real object or that multiple occurrences of a given type of object could also have the same kind of impact. Fragmentation F of an IO is a more complicated feature taking into account both the number and the size of the image regions composing the IO. Since this criterion has been proved ineffective in the experiment (see section 3.2), no formal modeling for F is given here.

Fragmentation Hypothesis: the importance of an image object is maximal when it is not fragmented, and decreases when its fragmentation F increases.

4

Homogeneity -Is the importance of any visible object in an image depending on the fact that it appears alone, or almost alone in the image? Is this importance decreasing when the image object appears among several others (of different nature) in the same image? What we call the homogeneity criterion H of an image expresses the extent to which an image presents several types of IO. Like the fragmentation criterion, this particular criterion has never been considered before.

We suppose here that the more different IO occur in an image, the more the cognitive overload 4 of the user increases, and thus the more the detection of a particular IO becomes difficult. The homogeneity criterion H is then maximal when the image contains only one type of IO and decreases when the number of occurrences of other types of IO increases. The definition of H includes the number and the relative surface of the various IO in a given image (see section 4.1).

Homogeneity Hypothesis: the importance of an image object varies in the same way as the homogeneity H of its embedding image.

Of course we do not claim that these four criteria are the only ones related to an effective definition of the importance of an IO.

We propose here a first step for future evaluations of geometrical features of image objects on their importance. To our knowledge no similar study has been published about such a basic problem.

From the information retrieval point of view, testing the validity of such hypotheses is a first important step towards a wellfounded weighting model for image objects. This experimentation process is presented in the following section.

Validation

The goal of the user's study here is to check and evaluate the four hypotheses described above. The goal of this experimental validation is to collect user's feedback preferences over images from a test set and then analyse these preferences.

Our validation is based on presenting couples of images to each participant. For a given pair (A, B) of images, we focus on the image objects related to the same term t. For each image of the couple, we define for the t-IO its size value, big or small, its position value, centred or lateral, its fragmentation value, fragmented or aggregated, and the homogeneity value for the image, homogeneous or heterogeneous. Figure 1 presents the 16 (=2 4 ) typical examples of images considering the possible values of the four criteria, considering the disc (or group of discs) as the term t. When considering a couple (A,B), we denote by V A (resp. V B ) the criterion variation from B to A (resp. A to B), according to the hypotheses. Assume that the situation for the t-IO in A is (big, centred, aggregated, heterogeneous), meaning that the t-IO is in the center of the image, big, aggregated, and that the image A is heterogeneous. Assume also that the situation for an image B is (small, centred, aggregated, heterogeneous). The image 5 (resp. 6) of Figure 1 is an example of A (resp. B). The criterion variation V A is only based on the size S (the other criterion values being equal), big in image A and small in image B, denoted by (1, 0, 0, 0) according to the size hypothesis. Dually, the criterion variation V B is denoted by (-1, 0, 0, 0).

Before actually running the experiments with participants, we determine the physical objects on which we focus on. The set of physical objects selected has to cover a wide range of possible objects appearing in home photographs, because we do not know if the hypotheses hold across different kinds of physical objects.

We have chosen the following categories of objects to be studied:

-Boat: this category is used to reflect the fact that many non living objects occur in home photographs, -Bird: we assume that animals often occur in home photographs, thus leading us to evaluate the criteria on such images, -Children face: people faces are a very important part of home photographs, as underlined in [START_REF] Preece | Human-Computer Interaction[END_REF]. In this test we used photographs showing on average as many girls as boys. For this object category and the following, we assume that many faces are present in home photographs.

-One specific face: in home photographs, many family of relative faces are present. During this experiment, the choice was made to use one of the authors face.

Figure 1: 16 logical image object configurations

As described above, for each of the considered objects we present a list of image pairs on which we evaluate the criteria. The criteria values were defined as a common agreement of three people (the authors of this paper) so that the visual differences between configurations are clear, easily visible and no confusion is possible. For each of the first 3 categories of physical objects, 120 pairs (i.e. all pairs of image in a set of 16 situations) of images are presented. For the last object category (one specific person), there is no fragmentation variation because the person occurs without occlusion; for this category only 28 photographs pairs are presented. So, a total of 388 (=120×3 + 28) pairs are presented to each participant. The duration of the test is about 30 minutes per participant. The images used in the experiments are normalized in term of size, quality, and typicality 5 . We conducted the experiment with 30 participants (16 women, 14 men) aged from 24 to 50. The participants belong to the research laboratory of the authors, but were not at all aware of the hypotheses tested in the study. Each participant chooses one image from each pair of images presented, according to its aboutness with regard to a given category term t. When a pair of images is presented to a participant, the user interface randomly picks which image is displayed at the top and which image is at the bottom of the window. The participant then selects the image which is the more about t. Consider again the image couple (A, B) presented above, with situation (big, centred, aggregated, heterogeneous) for A, and (small, centred, aggregated, heterogeneous) for B. If the participant selects the image A, then our hypothesis on the size is validated. This is denoted by the fact that the variation V A has a 1 in its first position. If the participant selects B, the size hypothesis is invalidated. By counting how many times the hypotheses are validated or invalidated using the criterion variations, we are able to draw conclusion on their usefulness. Before discussing the findings of the user's study, we verify that the size of the users set is large enough using the "split-half analysis consistency" [START_REF] Martin | Measuring Behaviour: An Introductory Guide[END_REF]. This method randomly splits in two sets X and Y the collected data, and computes the correlation 5 Typicality refers to the most commonly agreed representation of an object. 763 value between the two sets X and Y. If the same conclusions can be drawn from the two sets X and Y, evaluated using the correlation between the sets X and Y, then the sample size is sufficient. For 30 participants, the correlation value is equal to 0.962, meaning that the data obtained are highly consistent.

Table 1 shows the average and standard deviation (between parentheses) of each criterion variation considered independently: -Big Size selected (BS), -Center Position selected (CP), -Aggregated Object selected (AO), -Homogeneous Image selected (HI). Each of the values displayed corresponds to the count of participants choosing the criterion variation divided by the overall count of variation of the criterion presented to the participants. The value 0.76 for the criterion Big Size for boat indicates then that 76% of the time the participants chose the variation denoting that the image with a big boat was selected instead of the image with the small boat. It seems that the Big Size criterion variation, as well as the Center Position and Homogeneous Image criterion variations, is consistent with its related hypotheses. Conversely, it seems that the Aggregated Object criterion variation (average values between 0.38 and 0.60) is less important than the three others. Table 1 shows that for the named face object, the Big Size criterion is very large, showing that a big named face is almost always selected. On the contrary, the Center Position criterion value for named face is only 0.70 with a standard deviation of 0.27, showing that for named face the position seems to be less important. Considering criterion variations separately is not enough: we need to study the behaviour when combining them. Table 3 presents the Bernoulli probabilities for the random variables considering CP, BS, AO and HI together, and the random variables considering the dual choices for one criterion variation (the dual choice is written in parentheses, e.g. (BS) means that the image with the big size object was not selected). The last column of Table 3 Table 3 shows that the dual value of the AO has a very small effect on the probability (0.93 versus 0.91) when considering the other criteria, confirming one more time than the Fragmentation variation of image objects is not significant. However, we see also that the combination of the three criteria BS, CP and HI gives a higher probability that when we consider the dual value of any of these three criteria. All of these probabilities are highly significant (significance level smaller than 0.001), as seen on the last column of Table 3. This leads us to consider that the combination of these criteria behaves more in an additive way than a multiplicative one because one dual choice does not impact a lot the overall probability value. In summary, the results for Big Size, the Center Position and the Homogeneity of Image criterion variations confirm the hypotheses 1, 2 and 4 described in section 3.1, and the Fragmentation hypothesis is not confirmed.

WEIGHTING IMAGE OBJECTS 4.1 The Weighting Model

We have shown in the previous section that three out of four hypotheses are valid for the definition of IO importance: the ones concerning the size and the position of the IOs and the homogeneity of the images. An important problem we are now faced with is the integration of these criteria within a model capable of reflecting the overall importance of an IO. We use the probability theory and Shannon information theory [START_REF] Shannon | A mathematical theory for communication[END_REF], which represents a formal framework suited for IO importance modeling. The aim is to associate an importance value for IOs based on each criterion value, where the criterion values are themselves related to IOs own geometric features. We first give a modeling of the three criteria, and then we show how they are combined into a single model of importance for IOs.

Size of IOs is directly related to their surface, and consequently our size criterion is based on IOs relative surface. Human perception of surfaces being rather logarithmic than linear [START_REF] Rojet | Design considerations for a space-variant visual sensor with complex-logarithmic geometry[END_REF], we define a relative surface according to the following formula:

) ( log ) log( ) log( ) , S( o n I o n n n I o I = =
where I is a given image of size n I (in pixels), and o is an image object in I of size n o . If we consider two IOs io 1 and io 2 appearing in an image im, io 1 being twice as big as io 2 , S(io 1 , im) is smaller than 2×S(io 2 , im). S(o,I) is an increasing function of n o (according to our first hypothesis) with values in [0,1].

The position criterion P is integrated in our model by defining a non-uniform probability density function:

) ( ) , P( o p I o I =
where p I is a distribution of probability that gives higher probability values to IOs in the center of an image I. An example of such a distribution is given in Figure 2, where I is a simplified image seen as a 1D segment, and a 2D distribution is shown above I. Two image objects io 1 and io 2 are represented as parts of the segment, and their associated probabilities correspond to the black areas below the probability density curve. Hence the probability associated with io 1 is higher than the one associated with io 2 . The position values belong to [0,1], and they are greater for IOs in the center of the image, according to our second hypothesis.

We define a spatial entropy SH computed from the spatial distribution of IOs in an image, in order to represent the "disorder" in the image:

() I o I o n n I o n n I log ) ( SH - × = ∑ ∈
The above definition of spatial entropy makes it possible to combine in a well known framework two important aspects of the IO surface distribution: the number of IOs and their relative size. Spatial entropy values are larger when there are many objects with the same size in the image (when one object in particular is less easily visible), and it is smaller when there are few objects with different sizes (i.e. when one object is bigger than the others and therefore more easily visible) in the image. To be consistent with our last hypothesis, the homogeneity criterion is defined as the complementary value to 1 of SH to which we apply a normalization factor. The homogeneity H is defined according to the following formula:

() I n I I I I log ) ( SH 1 ) ( SH ) ( SH 1 ) ( H max - = - =
where SH max (I) is the maximum value of the spatial entropy, corresponding to the virtual case where the image is composed of n I 1-pixel IOs. The homogeneity values are large for homogeneous images according to our last hypothesis, and they belong to [0,1]. This value is the same for all IOs in one image.

We now describe how the three criteria are combined into a single importance model for IOs. The importance value of an IO should be large when all criterion values are large, and inversely.

the importance value should not be close to zero when one of the criterion values is close to zero while the two others are large (see Table 3). For these reasons, a good candidate for the combination is the addition. Since the three criteria might have different variation ranges on different collections, a normalization scheme is applied in order to align both lower and upper bounds of values [START_REF] Lee | Analyses of Multiple Evidence Combination[END_REF]:

min_value max_value min_value ed_value unnormaliz _value normalized - - =
where min_value (resp. max_value) is the minimum value (resp. the maximum value) of the criterion value found the whole collection. We define now the importance of an image object o in an image I as:

) ( H_norm ) , P_norm( ) , S_norm( ) , ( Importance I I o I o I o + + =
where S_norm, P_norm and H_norm are the normalized values of criteria S, P and H respectively. Hence, our definition of IO importance combines the three criteria that have been experimentally validated to reflect the aboutness of images with regard to the semantic interpretation of IOs. Next section presents a final experiment dedicated to estimate how well our model fits the users' perception of importance.

Experiment

The objective of this experiment is to evaluate the quality of our weighting model with regard to users' aboutness assessments. For this purpose, we have designed an image test collection of 800 manually indexed and segmented home photographs, disjoint from the first experiment image collection. This collection contains a wide range of authors holidays photographs. Importance values of objects are computed according to the formula given in Section 4.1. The weighting model is evaluated by comparing our system ranking of the images -in decreasing order of importance -to an assessors' ranking -in decreasing order of aboutness -on a set of 20 queries consisting of one single term (e.g. "Horse", "Pond", "Foggy_sky", "Bridge", "American_Flag", etc.). These terms correspond to different kinds of objects (natural like "River" and non-natural like "Bridge"), and also to different levels of genericity of the objects (specific like "American flag" and generic like "People"). The system ranking for each query is defined as the images sorted in decreasing order of importance values for the term. A group of 4 assessors7 -2 women and 2 men aged from 23 to 25, who have a good knowledge of the collection -have ranked relevant images for each query (on average, 6 relevant images have been ranked for each query). An average ranking is generated [START_REF] Ounis | Finding the Best Parameters for Image Ranking: a User-Oriented Approach[END_REF] and used as a reference to which the system ranking is compared. This comparison indicates how close to the users' perception of aboutness our weighting model is. The comparison is based on the following divergence function (inspired from [START_REF] Ounis | Finding the Best Parameters for Image Ranking: a User-Oriented Approach[END_REF]) that gives low divergence values to similar rankings, while penalizing more system ranking errors at the top ranked images:

( ) ( ) ( ) ( ) ∑ ∈         - × = ) , 1 ( 2 n , , , mdv 1 )
, ( divergence

n i i S rank i S rank i U rank S U
where U and S are the user ranking and the system ranking of n images respectively, rank(U,i) and rank(S,i) are the ranks of image i in the user ranking and in the system ranking respectively ; mdv n is the maximum divergence value for n items 8 . This divergence function gives the value 0 when U=S, and it gives the value 1 when U and S are in reverse order. For instance, the ranks for "Tree" are u= [START_REF] Fan | Automatic image annotation by using concept-sensitive salient objects for image content representation[END_REF][START_REF] Jing | An Efficient end Effective Recgion-Based Image Retrieval Framework[END_REF][START_REF] Baillargeon | Probabilités, Statistiques et Régression[END_REF][START_REF] Faloutsos | Efficient and Effective Querying by Image Content[END_REF][START_REF] Hollink | Classification of user image descriptions[END_REF][START_REF] Carson | Blobworld: Color-and Texture-Based Image Segmentation Using EM and Its Application to Image Querying and Classification[END_REF][START_REF] Bradshaw | Semantic-based image retrieval: a probabilistic approach[END_REF][START_REF] Jeon | Automatic image annotation and retrieval using cross-media relevance models[END_REF] and s= [START_REF] Fan | Automatic image annotation by using concept-sensitive salient objects for image content representation[END_REF][START_REF] Jing | An Efficient end Effective Recgion-Based Image Retrieval Framework[END_REF][START_REF] Carson | Blobworld: Color-and Texture-Based Image Segmentation Using EM and Its Application to Image Querying and Classification[END_REF][START_REF] Bradshaw | Semantic-based image retrieval: a probabilistic approach[END_REF][START_REF] Faloutsos | Efficient and Effective Querying by Image Content[END_REF][START_REF] Hollink | Classification of user image descriptions[END_REF][START_REF] Baillargeon | Probabilités, Statistiques et Régression[END_REF][START_REF] Jeon | Automatic image annotation and retrieval using cross-media relevance models[END_REF], meaning that the image 1 has been ranked at the 3 rd position by the assessors, while it is ranked at the 7 th position by the system. The divergence value for u and s is:

( ) 10
. 0 We use this divergence function instead of classical recallprecision measures because we are interested in the order of relevant images, rather than a simple binary relevance of images.

The system firstly implements each criterion individually, then their combination. The divergence value (DV) averaged over the 20 queries is 0.24 when considering the surface only, 0.31 for the position, and 0.63 for the homogeneity. The latter DV is high because relevant images are sorted according to their homogeneity values, which does not take into account the query term. When combined together, the 3 criteria perform better that separately, as shown in Figure 3: DVs range from 0 (for "Pool", "River" and "Temple") to 0.26 (for "Fortress"). The results obtained depend neither on the number of relevant images (shown in parentheses in Figure 3), nor on the type of objects considered (natural vs. non-natural, generic vs. specific or living vs. non-living). The high DVs (greater than 0.20) correspond to the queries "Fortress", "Geyser", "Motorbike" and "Sculpture". 8 The maximum divergence value is reached when the two rankings are in a reverse order.

For these query results, some of the relevant images are very similar with respect to their visual configuration, despite variations at an aesthetic level. Hence, the system is unable to discriminate between these images as the importance values, based on the visual configuration of images, are close to one another. However the average DV is 0.13, which means that our system ranking is very close to the users' perception of aboutness. These two results lead to conclude that our weighting model is adequate according to our second study. This experiment provides a validation to both our criteria modeling and their combination.

CONCLUSION

In this paper we have proposed a contribution to image indexing based on a weighting model for image objects. This model was viewed as a fundamental step for testing an image counterpart of the well-known tf paradigm in textual indexing. The model is fundamentally based on four perception criteria related to 2D geometry of images, namely surface, position, fragmentation and homogeneity. These four criteria have been thoroughly confronted to actual human perception of images, and to aboutness assessments through an experiment involving real users. Three of these criteria have been experimentally validated in this process, formally modeled and further combined within a single weighting model for image objects. Finally, the effectiveness of this weighting schema has been successfully compared to human-based aboutness assessments for images given symbolic definitions of topics.

Though we understand that several other perception criteria could be also considered as a basis for such a model, we consider that we have already got a good basis for designing and testing a tf.idf weighting schema adapted for images.

Future works are then mainly aimed at designing and testing a tf.idf approach for images, based on a test collection that we are designing by now. Extensions of the weighting model to other perception criteria -like visibility of image objects related to contrast or luminosity -are also foreseen, in just the same experimental and formal way used for the four criteria described in this paper. Through this first extensive attempt we have certainly comforted our feeling that in this area, modeling has to be grounded and validated by solid experiment.

Of course nobody knows by now whether or not such a weighting model -which has been successful for text -will be also effective for images. We view this as a much interesting and valuable challenge to face in the perspective of improving symbolic indexing and retrieval of images.

Figure 2 :

 2 Figure 2: Example of non-uniform distribution of probability.

Figure 3 :

 3 Figure 3: Divergence values for all query terms

Table 1 : Average (and standard deviation) for the four criterion variations and the four object categories.

 1 Table1also confirms the fact that the obtained results depend for some part on the category of the considered physical object.To go further in the evaluation of the results, we apply a statistic test to our data. We consider the discrete Bernoulli random variables6 Y {BS} , Y {CP} , Y {AO} , and Y {HI} related to the criterion variations BS, CP, AO and HI of Table1. For each of these discrete random variables Y c , we validate statistically our results by applying a unilateral T-test[START_REF] Baillargeon | Probabilités, Statistiques et Régression[END_REF], with the null hypothesis H 0 : Y c > 0.5, and the alternative hypothesis H 1 : Y c ≤ 0.5. The probabilities associated to the T-test are presented in the last column of Table2, where the second column presents the probability of the criterion variations averaged over the four categories. The last column of this table presents the probability to reject wrongly the H 0 hypothesis when H 0 is true. The results for BS, CP and HI are highly significant with significance value smaller that 0.001. We then conclude that hypotheses 1, 2 and 4 are validated. The T-test for the Aggregated image object gives a

	Criterion	Boat	Bird	Children	Named
	Variation			Face	Face
	Big Size	0.76	0.74	0.77	0.97
		(0.15)	(0.18)	(0.16)	(0.09)
	Center Position	0.84	0.82	0.77	0.70
		(0.13)	(0.12)	(0.15)	(0.27)
	Aggregated	0.38	0.53	0.60	N/A
	Object	(0.20)	(0.25)	(0.25)	(N/A)
	Homogeneous	0.88	0.65	0.81	0.86
	Image	(0.12)	(0.19)	(0.12)	(0.22)

6

Bernoulli random variables are used here because we evaluate probabilities of binary events.

Table 2 : Probabilities and T-test values over the four criterion variations.

 2

Table 3 : Probabilities and T-test values combining the four criterion variations.

 3 presents T-test results with the same H 0 and H 1 as in Table2.

	Random variable	Probability	T-test (H 0 )
	Y {BS, CP, AO, HI}	0.93	2.80 E-15
	Y {(BS), CP, AO, HI}	0.83	1.55 E-10
	Y {BS, (CP), AO, HI}	0.86	1.47 E-08
	Y {BS, CP, (AO), HI}	0.91	1.63 E-15
	Y {BS, CP, AO, (HI)}	0.80	6.20 E-10

Non specific images here means non application-specific images, like medical images or satellite images usually are, for example.

As usual an image region here is a set of connected image pixels.

In our opinion aboutness is a notion mainly local to a given document. It might not be confused with relevance, a notion that usually compares a particular document to the whole corpus. All the hypotheses presented here are purely local to images and thus cannot be directly assimilated to relevance evaluation. A relation between these two notions is somewhat illustrated by the tf.idf weighting model: given a term t of document D, tf of t stands for the aboutness of D considering t, while tf.idf is an estimate of the relevance of D considering t.

Defined as "excessive demand made on the cognitive processes, in particular memory" in[START_REF] Preece | Human-Computer Interaction[END_REF] page 717.

Note that none of the assessors have participated in the first experiment.


Mbarek Charhad

Daniel Moraru

Stéphane Ayache

Georges Quénot
email: georges.quenot@imag.fr

SPEAKER IDENTITY INDEXING IN AUDIO-VISUAL DOCUMENTS

The identity of persons in audiovisual documents represents very important semantic information for content-based indexing and retrieval. The task of speaker's identity detection can be carried out by exploiting data elements resulting from different modalities (text, image and audio). In this article, we propose an approach for speaker identity indexing in broadcast news using audio content. After a speaker segmentation phase, an identity is given to speech segments by applying linguistic patterns to their transcription from speech recognition. Three types of patterns are used to predict the speaker in the previous, current and next speech segments. Predictions are then propagated to other segments by similarity at the acoustic level. Evaluations have been conducted on part of the TREC 2003 corpus: a speaker identity could be assigned to 53% of the annotated corpus with an 82% precision.

INTRODUCTION

Thanks to technological progress, digital video databases continuously become more and more common as well as larger and larger. Efficient content-based access to them becomes more and more a critical issue and new and efficient tools are always needed for this.

The great variety of video document contents yields a large number of possible keys for indexing and retrieval. An indexing key can be for instance a simple word appearing in the discourse as well as a topic or category (sports, economy, politics, …) [START_REF] Fiscus | Topic Detection and Tracking, Evaluation (TDT)[END_REF]. In the second case, all words possibly related to a topic will be searched for. Another indexing key that we find very useful is the identity of the speaker. We considered speaker identity indexing in the context of TV news. This type of video document usually exhibits a well defined structure that eases the process of speaker identity detection and tracking.

Approaches based on the use of speech transcription obtained from automatic speech recognition and acoustic-based speaker segmentation are especially interesting since they allows for speaker identity indexing without needing the building of acoustic models of speakers (which is costly and restrict the indexing to learned speakers). These approaches can also be applied to purely audio documents like radio news unlike visual ones based on face recognition and lips motion detection. The visual methods are also limited to learned people and can even miss them if they do not appear from the right angle or at the right scale.

Alternatively again, OCR-based text recognition in the images could provide useful information about speaker identity but the current performance of OCR tools in video images (often degraded due to compression) does not appear sufficient for reliable extraction. The problem of attaching a face identity and/or a speaker identity to a name extracted by OCR also remains difficult.

The speaker identity indexing approach proposed here is base on the use of linguistic patterns applied to a transcription of the audio tracked obtained by automatic speech recognition (ASR). It is similar to the one proposed in [START_REF] Canseco-Rodriguez | Speaker Diarization from Speech Transcripts[END_REF]. The search for parameterized regular expression in the transcription permits to propose speaker identity in speech segments in which it appears, in speech segments following the one in which it appears or in speech segments preceding the one in which it appears. This approach was completed by the propagation of the identity attributed to some segments to other segments attributed to the same speaker by acoustic similarity. Quantitative evaluations were performed using a part of the TRECVID 2003 corpus. This rest of this article is organized as follows: in section 2, we present some previous work on the use of audio information for content based indexing and retrieval. In section 3, we detail our approach for speaker identification using linguistic patterns. We present an evaluation of it in section 4 and we discuss the results and the approach in section 5. Finally, we draw some conclusions and consider some perspectives in section 6.

ISBN: 952-15-1364-0

CONTEXT

The audio track in audio-visual document is a very rich source of information and contains a multitude of indexing keys at both the signal and semantic levels. Audio information constitutes the basis of a large number of approaches in the literature. These approaches distinguish themselves by the specificity of their applicative framework, usually restricted to specific domains.

Audio information is often used alone with great success for audio-visual indexing and retrieval [3][4]. From a semantic point of view, speech is more used than other audio information like music or noise [START_REF] Pinquier | Speech and music classification in audio documents[END_REF]. In [START_REF] Sönmez | Speaker Tracking and Detection with Multiple Speakers[END_REF] the authors propose a method for detecting and tracking different speakers in conversational speech using hidden Markov models. They exploit detection results for document segmentation.

An important use of audio information is the automatic transcription of speech [7][8]. This information brought in a text form gives a lot of information about the content at a level very close to semantics. It permits the use of text based retrieval techniques within audio-visual document bases. This technique has been used in many research works like for instance the ANTS 1 system [START_REF] Brun | ANTS : le système de transcription automatiquedu LORIA[END_REF]. The exploitation of the speech transcription and speaker segmentation in order to identify the speaker remains to be done.

Document Structure

TV news usually has a well defined structure. This structure can help to generate indexing keys. The speaker is often one of the main actors in these documents. We aim at exploiting this information for indexing segments of TV journals. From a general point of view, there exist two ways to perform a segmentation of TV news. The first one in by considering the thematic content: the journal will be seen as a sequence of topics (politics, sports, weather) often separated by jingles, commercials, etc. The second one exploits the studio / reporting structure to produce the segmentation. This is often during this type of segmentation that there are transitions and speaker turns.

Starting from this structure, we aim at identifying the name of the speaker for each segment. We propose to use for this purpose the results of speech transcription and speaker segmentation 1. ANTS 1 (Automatic News Transcription System) developped in the context of the ESTER project

Audio Segmentation

The objective of the audio segmentation is to obtain coherent audio segments. Each segment has a specific content: silence, noise, music, speech, etc. Many approaches for automatic segmentation have been proposed in the literature. For instance, Kwon and Narayanan have proposed in [START_REF] Kwon | Speaker Change Detection Using a New Weighted Distance Measure[END_REF] a technique for document segmentation based on the computation of weighted distances of speaker change points in the audio stream. This approach is useful, especially for the detection of several distinct speakers. Other segmentation techniques, more general, exploit audio characteristics like silences or jingles to infer transitions.

The most useful segmentation for our approach is the segmentation into speakers. Such segmentation is considered as a task belonging to the domain of speech processing. It is generally conducted as a two-step process: the first step is the detection of changes between speakers, often using a Bayesian Information Criteria, and the second step is the merging of similar segments, adjacent or not. Not only the transitions between speakers are detected but also all the segments from a same speaker are identified as such (generally with some errors).

Linguistic Patterns

In addition to its physical structure (studio / reporting) a TV journal has a linguistic structure. This linguistic structure appears in many occasions when there is a speaker change. Journalists often used marked expressions in their discourse to pass the speech turn to each other or to intervening people. We propose to exploit this structure to set up a technique for speaker identity recognition. The expressions often used for speaker transitions will be treated as linguistic patterns for speaker identification.

The advantage of this type of patterns is that they readily permit to distinguish between the identities of speakers and the identities of all other persons appearing in the speech transcriptions.

SPEAKER IDENTITY DETECTION USING LINGUISTIC PATTERNS

Speaker identity detection is obtained from an automatic transcription of what is said [START_REF] Gauvain | The LIMSI Broadcast News transcription system[END_REF] and from an automatic speaker segmentation [10][12]. The speaker segmentation detects speech turns between speakers but it also determines whether different segments are from a same speaker. It must be noticed that the speaker segmentation output does not contain any information about speaker identity. No acoustic model of speaker is used here.

Besides the transcription and speaker segmentation, it is also necessary to have access to general knowledge in order to be able to identify named entities. Lists of names, locations and organizations can be used. We got a lot of them from public sources (like Wikipedia).

Our approach is based on a linguistic analysis of the speech transcription in order to identify passages (patterns) from which we can infer a speaker identity.

Direct Detection

Linguistic patterns are applied to pieces of text that correspond to the transcription of the audio-visual document restricted to a speech segment obtained from speaker segmentation. They correspond to regular expression (usually parameterized) expected to indicate the identity of the person who speaks, who just spoke or who is about to speak. They are applied to each speech segment and, when they are triggered, they permit to predict the speaker identity in the current, previous or next segment. A pattern category is built for each case:

• The first category is for the detection of the identity of the person who is speaking. For instance, a speaker introduces himself: "… This is CNN news I'm [name] …" or during reporting, generally at the end, the person who speaks mentions its identity (often associated to the location or the organization from where he speaks).

• The second category is for the detection of the identity of the person that just spoke, for instance: "Thank you (very much) [name] …".

• The third category is for the detection of the identity of the person which is about to speak. This transition often corresponds to a transition from studio environment to a reporting. For instance, at the end of the discourse of the anchor person: "… [name] has the latest" or "… [name] reports".

A list of patterns has been built for each category. The use of appropriate linguistic patterns permits to discriminate between the identity of a person simply mentioned in the discourse and the identity of an actual speaker. Table 1 shows some example of linguistic patterns from the three classes. Some of these patterns are specific to the corpus that we use (TRECVID 2003 and 2004: TV news from ABC and CNN).

The "Good morning" linguistic pattern is often used to indicate a speaker either in the previous or in the next speech segment. In this case, this is the location where the pattern has been found (close to the beginning or close to the end) that makes the decision. Our linguistic patterns are all parameterized. They have at least a parameter for a name, a surname or a couple (name, surname). They can also include parameters for geographic locations, for organizations (CNN or ABC for instance) and/or for expressions like "good morning", "good evening" or "thanks".

Patterns for current segment

For each segment, we analyze the content in order to detect patterns providing the identity of the speaker. Patterns are generally found at the end or at the beginning of speech segments. For instance, the following pattern infers a speech turn between the anchor person and a reporter "… ABC's Sheila Macvicar has the latest."

For the recognition of people's first name, we use a list of 12,400 first names (each tagged as male or female). Family names are negatively filtered from a list of (nonname) English words. Family names almost always appear after a first name. We used a total of 20 linguistic patterns.

Propagation

In the case of TV news, the speaker usually introduce himself only once during its first speech turn. Therefore, the linguistic pattern approach alone permits to index only small parts of video documents. We have therefore completed it by propagating detected identities to additional speech segments using similarity at the acoustic level. In this part, we propose to propagate a speaker identity to all the segments attributed to a same speaker at the acoustic level (during the speaker segmentation phase) each time an identity has been assigned to any of them by the application of linguistic patterns.

EVALUATION

Evaluations have been conducted on a part of the TREC video 2003 corpus [START_REF] Smeaton | TRECVID 2003 -An Introduction[END_REF]. We used the speech transcription and the speaker segmentation distributed with the TREC video corpus and produced by LIMSI [START_REF] Gauvain | The LIMSI Broadcast News transcription system[END_REF]. The word error rate has not been evaluated on this corpus but from similar sources, it is estimated to be between 10 and 15%. The speaker segmentation quality has not been evaluated. It includes errors both at the speaker transition level (some transitions between speakers are missed and some are wrongly inserted) and at the speaker clustering level (some segments from different speaker are attributed to a single speaker and some segments from a same speaker are labeled as coming from different speakers). The global quality of the speaker segmentation and clustering was found quite good however and it bas been used as it was provided. Speaker identity recognition is not an official task of the TRECVID 2003 evaluation campaign. However, this corpus along with its various associated data sets contains everything that is needed for the evaluation of this task, except ground truth about speaker identity. We manually annotated it each time it was available in four TV journals (two from ABC and two from CNN) of about half an hour each. Table 2 It can be noticed that the propagation of predictions using acoustic similarities introduce very little errors while it significantly increases the proportion of indexed signal. Errors are due to bad location of speech segment boundaries, to bad associations of segments to a same speaker or to errors in people's name in the speech transcription.

DISCUSSION AND FUTURE WORK

The transcription of speech is a very rich source of semantic information. In the case of audio-visual documents, there is generally a relation between audio information (what is said) and visual information (what appears on the screen). In the approach presented in this paper we have exploited the results of speech transcription. The detection of the speaker identity is part of the larger problematic of the modeling of the semantic audio-visual contents for indexing and retrieval.

We plan to extend the linguistic pattern based approach in order to extract additional information about the speaker including the location (city name, country name) from where he speaks, the topics he is talking about, his function, etc. We also consider using this approach for detecting topic changes in the document.

In order to exploit this approach in the context of a system for content-based search in audio-visual documents, it is interesting to integrate the audio and visual information in a way similar to what has been proposed in [START_REF] Yang | Finding Person X: Correlating Names with Visual Appearances[END_REF]. However, there is not always a correspondence between the name of a speaker and the image content. Thanks to the structure of TV news, it is possible (for this type of documents) to estimate the conditions under which one can see and listen to a person simultaneously. This information is useful in order to estimate the relevance of systems' answers for users as well as the results' accuracy when speakers are not well known.

The temporal alignment of the audio transcription eases the detection process and permits to determine when speaker turns occur. This is similar to what is done on the visual side during shot segmentation. The duration of a video shot is generally short compared to speech segments. In our approach, we propose to exploit the temporal information in order to associate to each audio segment the corresponding key frame(s) in the visual stream. This eases the evaluation of the results by the user. The user can for instance make a query for "passages in which James Walker speaks" and can quickly extract the relevant answers by looking at the displayed key frames.

CONCLUSION

We have proposed, implemented and evaluated an approach for indexing speaker identity into audio-visual documents for content-based retrieval. We defined a set of linguistic patterns for speaker identity detection. These patterns are of three types in order to detect the identity of the speaker in the previous, current or next speech segment (relative to the segment in which they appear).

The main part of the work has been to analyze the text of the transcription coming from ASR. Several semantic information are exploitable in the transcription such as locations, organizations and people.

Evaluations conducted on a part of the TRECVID 2003 corpus have shown that the proposed method permit a speaker identity indexing for about 53 % of speech time in TV news with an accuracy of about 82 % (47 % of speech signal is not indexed, 43 % is correctly indexed and 10 % is wrongly indexed).

Speaker indexing permits both a conceptual indexing and a relational indexing: the knowledge of speaker identity allows to infer the relations "speaks" or "speaks about".

In the case of TV news, it is sometimes very difficult to determine from the audio track only who the speaker is. For instance, the identity of intervening people is often displayed in the image track and rarely cited in the discourse. Therefore, we plan to complete our detection approach using additional information sources like automatic text recognition in images. Also, detecting the identity of intervening people can still be done using linguistic patterns but less frequent ones.

Table 1 :

 1 examples of linguistic patterns

	[name] for ABC news
	I'm [name]
	[name] CNN
	[name] ABC
	Patterns for previous segment
	thank you … [name]
	thanks … [name]
	[name] reporting
	good morning [name]
	Patterns for next segment
	tonight with [name]
	ABC's [name]
	[name] reports
	good morning [name]

Table 2 :

 2 summarize the characteristics of the used (sub)corpus. characteristics of the corpus used for evaluation

	Total video duration	7009.0 s
	Total speech duration	5249.1 s
	Total news duration	4250.3 s
	Speech within news	3767.1 s
	Annotated speech within news	3677.5 s
	Table 3 shows the results obtained for identity detection
	by linguistic patterns. Patterns specialized in the detection
	of the identity of the speaker in the previous, current and
	next speech segments permits a prediction in 1.0 %, 6.8 %
	and 7.0 % of cases (in speech duration) respectively. A
	direct prediction is therefore made in 14.8 % of cases.
	Propagation of these predictions using acoustic
	similarities permits a prediction in 52.7 % of cases. The
	rates of correctness (accuracy) of the indexed parts are of
	100 %, 90.2 %, 74.1 %, 83.3 % and 82.4 % respectively.
	The source of the documents (ABC or CNN) was not used
	during the identification process.

Table 3 :

 3 Results for speaker identity indexing. The percentage of the predicted duration is relative to the total duration manually annotated. The percentage of the correct duration is relative to the predicted duration.

ACKNOWLEDGEMENTS

This work has been sponsored by an INPG BQR project and by the PENG -IST-004597 European project.


Stéphane Ayache
email: stephane.ayache@imag.fr

Georges M Quénot
email: georges.quenot@imag.fr

Jérôme Gensel

Ichi Satoh

CLIPS-LSR-NII Experiments at TRECVID 2005

This paper presents the systems used by CLIPS-IMAG laboratory. We participated to shot segmentation and high-level extraction tasks. We focus this year on High-Level Features Extraction task, based on key frames classification. We propose an original and promising framework for incorporating contextual information (from image content) into the concept detection process. The proposed method combines local and global classifiers with stacking, using SVM. We handle topologic and semantic contexts in concept detection performance and proposed solutions to handle the large amount of dimensions involved in classified data.

Introduction

The CLIPS-IMAG team have participated to the Shot Segmentation task with few modifications from previous participations. The emphasis have been laid on the High-Level Feature Extraction Task, with the use on image cue only.

We previously proposed and evaluated a framework for incorporating contextual information into image indexing process [START_REF] Ayache | Context-based Cconceptual Image Indexing[END_REF]. We assume that both semantic and local contexts can increase the accuracy of a classifier. We present a general framework for contextual image indexing using vectorbased classifier. The main difficulty is to fuse a lot of information while managing the curse of dimensionality problem. We want to incorporate both local and inter-concept information in the decision process, which can need a lot of dimensions. In addition, supervised classifiers need a number of samples, which is correlated to the dimensionality problem.

Context

New issues in Content-Based Image Indexing (CBII) field are arising for the reduction of the well-known semantic gap. In order to improve concept extraction, many approaches take into account the context. To do so, some approaches fuse local and global descriptors [START_REF] Murphy | Freeman Object detection and localization using local and global features[END_REF][START_REF] Garg | Fusion of Global and Local Information for Object Detection[END_REF] combined using boosted classifiers [START_REF] Amores | Boosting Contextual Information in Content-Based Image Retrieval[END_REF].

Other define context as spatial relationships between objects within an image [START_REF] Singhal | Probabilistic spatial context models for scene content understanding[END_REF][START_REF] Torralba | Freeman Contextual models for object detection using boosted random felds[END_REF][START_REF] Carbonetto | A statistical model for general contextual object recognition[END_REF] using probabilistic frameworks. However, in order to deal with a large amount of local descriptors and simplify computation, such approaches first detect points of interest and then assume them independent.

In their work, [START_REF] Iyengar | Discriminative model fusion for semantic concept detection and annotation in video[END_REF][START_REF] Naphade | On supervision and statistical learning for semantic multimedia analysis[END_REF][START_REF] Snoek | The MediaMill TRECVID 2004 Semantic Video Search Engine[END_REF] handle semantic relationships [START_REF] Fink | Mutual boosting for contextual influence[END_REF] between concepts using Stacked Classifiers [START_REF] Wolpert | Stacked Generalization[END_REF][START_REF] Lisin | Learned-Miller Combining Local and Global Image Features for Object Class Recognition In CVPR[END_REF]. They first classify intermediate concepts, and learn their relationships in the context of a higher level concept by a second-level classifier. In TRECVID'04 experiments, [START_REF] Snoek | The MediaMill TRECVID 2004 Semantic Video Search Engine[END_REF]

A Three-Level framework

We use several classifiers and arrange them into networks, using stacking. The idea is that the correlation between the input (low-level features) and the output (concepts) is too weak to be efficiently recovered by a single "flat" classifier even if the low-level features have been carefully chosen. Combining classifiers in a multi-level framework allows extraction of intermediate-level data from low-level features and other classifiers. Detection of concepts from these intermediate-level data is expected to improve the overall performance (both correctness and computation time) of the system. Classifiers from each level bridge a small part of the semantic gap and are expected to do it well because the correlation between their inputs and their outputs is expected to be better than the correlation between the inputs and the outputs of a single classifier that would bridge alone the whole semantic gap. Also, not only the levels are cascaded but many intermediate data are computed in parallel and the outputs of all of them are combined as inputs to the next level. Also, such architecture allows either inference if "Concept j" is not classified as patches in the second layer, either whole image labeling if "Concept j" is already classify at patch-level.

The objective of the present work is to validate these assumptions and to quantify the benefits that can be obtained from this approach. We propose a multi-layer framework in order to integrate the different contextual information. The first layer extracts Low-Level Features for each patch, the second layer provides a score for patches according to a concept, and the higher level layer assigns a value for a whole image. The first layer is basically the only one that handles low-level features.

We focus on three kinds of concepts:

• Patches-Concepts can be quite easily extracted from a piece of an image (a patch).

We consider them as middle-level concepts as they are useful to help extraction of higherlevel concepts. Examples of such concepts are Sky, Greenery, Water, Wall, Sand, Building,

• Scenes-Concepts are higher-level concepts as they refer to the whole image. Thus, their extraction necessitates a deeper analysis of image's context. Examples are Beach, City, Studio settings, Mountain . . . Such concepts cannot be extracted as patches.

• Objects-Concepts are specific concepts. They are usually detected using specific objectbased detectors, which are much highly timeconsuming. In spite of it, we expect to extract them using our contextual framework.

Low-level features extraction

As we want to handle topologic context, we need to compute low-level features for parts of the image. In order to compute local features, many approaches have been proposed. While automatic and a priori segmented regions are too far from semantic meaning of image, we decided to split images into patches. By doing so, we should be very far from semantic, but with such granularity one patch is more likely to contain a single concept.

The low-level features extractor's process first splits image into overlapped patches. Basically, we used 260 overlapped patches of 32 × 32 pixels (in 352 × 240 images). And then, extracts 9 moments color (3 means + 6 co-variances), Gabor wavelets for texture (3 scales × 8 orientations), and coordinates of patches.

These choices have been made for a baseline system. The main goal here is to explore the use of context for concept indexing. We want to study and evaluate various ways of doing it by combining classifiers into networks. In further work, we plan to enrich and optimize the set and characteristics of low level features, especially for video content indexing. Currently, we expect to obtain representative results from the current set of low-level features.

Use of the topologic and semantic contexts

The idea behind the use of topologic context is that the confidence (or score) for the whole image could be computed more accurately by taking into account the confidences obtained for each patch in the image for the same concept. By doing so, the classifier learn topological distribution of a concept into an image.

The idea behind the use of semantic context is that the confidence (or score) for a single concept could be computed more accurately by taking into account the confidences obtained for other concepts in the same image. We are considering concepts as related one to each other.

Using both contexts, we expect to improve the performance of concept detection at the image level by combining the output of patches-level detectors (classifiers).

We have N level-1 classifiers, each with F inputs and 1 output and N level-2 classifiers, each with N × P inputs and 1 output. All level-1 classifiers are callled P times on a given image and their N × P output values are passed to the corresponding level-2 classifier which is called only once.

First we classify patches for each concept, and then classify the whole image by merging every output. With many concepts, the number of inputs for the second stage can be too large. We can apply different approach to reduce this number, using well known feature selection techniques such as PCA.

In TRECVID05 experiments, we manually choose a set of 5 semantic context concepts per concept.

Baseline, no context, one level

In order to evaluate the patch level alone, and to compare the use of contexts, we define an image score based on the patch confidence values. To do so, we simply compute the average of all of the patch confidence scores. This baseline is very basic it does not take into account any spatial or semantic context. We have here N classifiers, each with F inputs and 1 output. Each of them is callled P times on a given image and the P output values are averaged.

Experiments

We had many hardware problems for the official submission, so we recompute them in the same conditions. We now present these results.

According to our typology of concepts, 8 out of 10 TRECVID'05 concepts are patchesconcepts: People-Marching, Explosion Fire, Maps, Flag-US, Building, Prisoner, Sports and Car. We manually modify official annotations in order to have annotated regions. For this task, we had considered Prisoner as Prisoner clothes and Sports as Sports grounds (tennis, basketball, soccer, ...). We also enriched these intermediate concepts with Sky, Greenery, Skin Face, Water and Road. The two others concepts, which are not patches-concepts (Waterscape and Mountain) were "inferred" using others concepts, such as Water and Greenery.

In order to handle the curse of dimensionality problem, we need to limit the amount of intermediate concepts to merge. For these experiments, we manually did that. Each concept's context is defined by at most 5 intermediate concepts (patchesconcept).

We used SVM classifier [START_REF] Chang | LIBSVM: a library for support vector machines[END_REF] with RBF Kernel, because it has shown good classification results in many fields, especially in CBIR [START_REF] Gosselin | A Comparison of Active Classification Methods for Content-Based Image Retrieval[END_REF]. It is important to use cross validation for parameter selec- In order to obtain the training set, we extract patches from annotated regions, it is easy to get many patches by performing overlapped patches. Annotating whole images is harder as annotators must observe each one.

We collected many positive samples for patches classification, and defined experimentally a threshold for maximum numbers of positive samples. We found that 2048 positive samples is a good compromise to obtain good accuracy with smaller training time. Also, we found that twice negative samples is a good compromise. Finally, we randomly choose negative samples. The Table 2 shows the number of positive image examples, for each concept.

Figure 2 shows Mean Average Precision results for our baseline and the proposed approach. We compare them with Median and Best scores of TrecVid'05 evaluation. It is interesting to notice that Median and Best results include approaches, which also used ASR data.

The use of contexts (MAP5) improves the performance over the baseline (MAP1).

Conclusion

We present a framework for incorporate semantic and topologic contexts into CBII. We compare our approach with a baseline system which don't handle context information. We show that both contextual information improves concepts detection. We now plan to combine such information with other video cues such as ASR and motion.

  used a basis of 22 intermediate concepts. By adding the 10 TRECVID's concepts, they learned semantic context of 32 concepts with Stacking. 1

Figure 1 :

 1 Figure 1: Overall classifier architecture

Figure 2 :

 2 Figure 2: TrecVideo 2005 Results

Acknowledgments

This work has been spported by the ISERE CNRS ASIA-STIC project and the Video Indexing INPG BQR project.