Frederic Abergel

Nikos Paragios

Pascale Legall

Pascal Laurent

Gilles Faye

Iasonas Kokkinos

Marie-Aude Aufaure

Nesrine Marc-Antoine Arnaud

Ben Mustapha

Hana Ben Zineb

Dung Bui

Clément Courbet

Emmanuelle Gallet

Nicolas James

Bilal Kanso

Sofiane Karchoudi

Adrian Maglo

Casio Melo

Adith Perez

Florent Pruvost

Rania Soussi

Olivier Teboul

Konstantin Todorov

Amel Znaidia

Arunvady Xayasenh

Ka Ho Yim

Cédric Zaccardi

Géraldine Carbonel

Catherine Lhopital

Keywords: Automatic Image Annotation, Hierarchical Image Classification, Multimedia Ontologies, Semantic Hierarchies, Knowledge-Driven Models, Ontological Reasoning, Fuzzy-Description Logics. vi Annotation automatique d'images, classification hiérarchique d'images, ontologies multimédia, hiérarchies sémantiques, modèles à base de connaissances, raisonnement ontologique, logiques de Computational Model

It was my pleasure to share with you lunches, table football and skiing. I am also

Résumé

Cette thèse vise à construire et à utiliser des modèles à base de connaissances pour l'annotation automatique d'images. En effet, la plupart des approches d'annotation d'images sont basées sur la formulation d'une fonction de correspondance entre les caractéristiques de bas niveau, ou de niveau intermédiaire, et les concepts sémantiques en utilisant des techniques d'apprentissage automatique. Cependant, la seule utilisation des algorithmes d'apprentissage semble être insuffisante pour combler le problème bien connu du fossé sémantique, et donc pour produire des systèmes efficaces pour l'annotation automatique d'images. L'utilisation des connaissances structurées, comme les hiérarchies sémantiques et les ontologies, semble être un bon moyen pour améliorer ces approches. Ces structures de connaissances permettent de modéliser de nombreuses relations sémantiques entre les concepts, comme par exemple, les relations de subsomption, les relations contextuelles et les relations spatiales. Ces relations se sont avérées être d'une importance primordiale pour la compréhension de la sémantique d'images. En outre, l'utilisation de ces modèles à base de connaissances structurées permet de réduire la complexité du problème de l'annotation d'images à grande échelle, i.e. sur des bases avec un grand nombre d'images et un nombre de concepts assez conséquent.

Dans cette thèse, nous proposons une nouvelle méthodologie pour la construction et l'utilisation de ces modèles à base de connaissances dédiés à l'annotation d'images. Plus précisément, nous proposons dans un premier lieu des approches pour la construction automatique de modèles de connaissances explicites et structurés, à savoir des hiérarchies sémantiques et des ontologies multimédia adaptées pour l'annotation d'images. Ainsi, nous proposons une approche pour la construction automatique de hiérarchies sémantiques. Notre approche est basée sur une nouvelle mesure "sémantico-visuelle" entre les concepts et un ensemble de règles qui permettent de relier les concepts les plus apparentés jusqu'à la création de la hiérarchie finale. Ensuite, nous proposons de construire des modèles de connaissances plus riches en terme de sémantique et qui modélisent donc d'autres types de relations entre les concepts de l'image. Par conséquent, nous proposons une nouvelle approche pour la construction automatique d'une ontologie multimédia qui modélise non seulement les relations de subsomption, mais aussi les relations spatiales et contextuelles entre les concepts de l'image. L'ontologie proposée est adaptée pour raisonner sur la cohérence de l'annotation d'images.

Afin d'évaluer l'efficacité des modèles de connaissances construits, nous proposons de les utiliser par la suite dans un cadre d'annotation automatique d'images. Nous proposons donc une approche, basée sur la structure des hiérarchies sémantiques, pour la classification hiérarchique d'images. Puis, nous proposons une approche générique, combinant des techniques d'apprentissage automatique et le raisonnement ontologique flou, permettant de produire des annotations d'images sémantiquement pertinentes. [START_REF] Sivic | Unsupervised discovery of visual object class hierarchies[END_REF] Baseline method, H-SVM, and the flat classification method. . . . . . 5.9 Recall/Precision curves for the concepts of each level of the hierarchy.

Introduction

Nowadays, image databases are becoming very large and of potential use in many areas, including public domain applications and also domain-specific applications. For instance, the public domain applications regroup internet services such as social networks1 , photo sharing 2,3 , image and video monitoring, and more generally, information retrieval related services and image search engines. Specific-domain applications include medicine, aerial surveillance, audiovisual archiving and many others.

As the amount of these digital images increases, the problem of finding a desired image (or a subset of them) becomes critical. For example, in a distributed medical database (probably worldwide), given the image of a patient, it can be beneficial to find other images of the same modality, of the same anatomic region, and/or of the same disease that was already diagnosed [START_REF] Müller | A review of content-based image retrieval systems in medical applicationsclinical benefits and future directions[END_REF]. Doubtless, this will have a great impact on the clinical decision-making process. Consequently, there is an important need for efficient techniques for storage, indexing and retrieval of multimedia information. In particular, image retrieval is still a big challenge despite 20 years of research.

The image retrieval field is carrying on the set of techniques/systems for browsing, searching and retrieving images from a large collection of digital images. Usually, such systems operate in two steps: i) image indexing: which could be defined as the process of extracting, modeling and storing the content of the image, the image data relationships, or other patterns not explicitly stored, and ii) image search: which consists in executing a matching model to evaluate the relevance of previously indexed images with the user query. Figure 1.1 illustrates the workflow of image retrieval systems. It is common sense that image search in these system is based on the same features (or modalities) than the ones used for image indexing.

This thesis deals with the problem of semantic image annotation. Specifically, we focus in this dissertation on how to model in an effective way the image content. Our approach is based on the building and the use of explicit and structured knowledge models in order to improve image annotation.

Context and Problem Statement

Initially, images were manually annotated by text descriptors (or tags) which are then used by an image retrieval system. This process is called 'iconography'. 'Iconography' means the description and interpretation of works in visual arts 4 . With respect to image retrieval field, 'iconography' means the process of human annotation of images. The iconography may be of prime importance for image retrieval systems since it provides valuable information about image content. It allows to dispose of the visual content description of an image, and very often of its subjective, spatial, temporal and social dimensions which are priceless for understanding image semantics. Online photo sharing systems, such as Flickr, Picasa and Getty Images5 , provide a valuable source of human-annotated photos. However, the iconography requires a considerable level of human labour, and it cannot be considered for large image databases [START_REF] Liu | A survey of content-based image retrieval with high-level semantics[END_REF].

Current approaches for automatic image indexing and retrieval can be classified into text-based approaches or content-based approaches, according to the used content (or modality) to index images -cf. Figure 1.2. In the text-based approaches, images are indexed by a set of text descriptors which are extracted from the surrounding context 6 . Nevertheless, although this paradigm is adopted by many current image

Ontology driven approaches

Information fusion

Semi-Supervised Learning

Figure 1.2: Proposed taxonomy of image retrieval approaches.

search engines (e.g. previous version of Google Image Search7 , Bing Images8 , etc.), it suffers from the following problems:

• As illustrated in Figure 1.3, the surrounding context is not always relevant with the image content, or sometimes only a small part of it describes its content. Consequently, the surrounding context is not always relevant for indexing images. Moreover, these methods do not consider the image content during the indexing process, and therefore there is no guarantee that the provided annotation is relevant with respect to image content.

• Text-based approaches are subject to the subjectivity of image semantics, also known as the problem of subjectivity of human perception9 [Rui et al., 1998, Sethi andComan, 2001]. This problem occurs when the one who provided the image description has a different background, and/or he want to express a different semantics in the image content, compared to the one who is searching the image.

• Currently, many voluminous image databases are generated daily without any surrounding context. For instance, the uploaded pictures of many Facebook users. Text-based approaches are unable to process these images. To overcome these problems, Content-Based Image Retrieval (CBIR) was introduced in the early 1980s. In this type of approaches, images are indexed and retrieved using their content. In particular, many approaches have been proposed to index images by their visual content, modeled by a set of low-level or mid-level visual features [START_REF] Smeulders | Content-based image retrieval at the end of the early years[END_REF]. Nevertheless, although several sophisticated algorithms have been proposed to extract and to store efficiently these visual features, it was shown that these approaches are unable to model the semantic content of images. This is known as the semantic gap problem, which is defined as "the lack of coincidence between the information that one can extract from the visual data and the interpretation that the same data have for a user in a given situation" [START_REF] Smeulders | Content-based image retrieval at the end of the early years[END_REF].

In Figure 1.4, we illustrate an example of the semantic gap problem. Indeed, the images (a) and (b) have similar color histograms (i.e. a similar visual appearance) but depict different concepts, whereas, the images (a) and (d) have different color histograms but depict same concepts, i.e. {"House", "Forest", "Cabin"}.

Hence, the main issue in image retrieval field is how to relate the visual content of images (low-level or mid-level visual features) to its semantic content (or high-level concepts). Consequently, the development of new approaches allowing to narrow 10 the semantic gap has been a core research topic since already ten years. In particular, automatic image annotation or content-based image annotation has been a very active research domain since a decade. Indeed, automatic image annotation can be Chapter 1. Introduction defined as the process of associating to a previously unseen image a textual description (often reduced to a -set of-semantic keywords or high-level concepts) which depicts its content. Once again, a wide number of approaches have been proposed for automatic image annotation. These approaches have focused on the problems of recovering effective image descriptors on one hand, and on the other hand on developing efficient machine learning algorithms in order to provide robust methods that allow mapping visual features into semantic concepts [START_REF] Duygulu | Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary[END_REF], Barnard et al., 2003, Lavrenko et al., 2003, Djeraba, 2003, Carneiro et al., 2007]. These approaches are also known as image classification or image categorization approaches. However, these approaches seem to be also insufficient to bridge the semantic gap, specifically when dealing with broad content image databases [START_REF] Liu | A survey of content-based image retrieval with high-level semantics[END_REF], Deng et al., 2010]. Specifically, automatic image annotation approaches are sensitive to the dimension of the physical representation space of image features and to the size of the semantic space used for describing image concepts. Moreover, these approaches face the scalability problem when the concept number is high and depend on the targeted datasets as well [START_REF] Hauptmann | How many high-level concepts will fill the semantic gap in news video retrieval?[END_REF]. Furthermore, automatic image annotation approaches are subject to many types of uncertainty introduced by machine learning algorithms:

• Uncertainties in input data, i.e. images are subject to noise, outliers, and errors, but also the representation (projection in a given space) of these data introduces some form of uncertainty. This refers to the sensory gap [START_REF] Smeulders | Content-based image retrieval at the end of the early years[END_REF]. The noisy data, in theory, can sometimes have a positive effect on the generalization behavior of the machine learning algorithm, since it is forced to develop some form of invariance and to abstract from the Chapter 1. Introduction noise [START_REF] Hammer | How to process uncertainty in machine learning[END_REF]. But in practice, noisy data usually have a decreasing impact on the method efficiency.

• Uncertainties in model parameters [START_REF] Solomatine | A novel method to estimate model uncertainty using machine learning techniques[END_REF]. Indeed, machine learning algorithms are sensitive to parameter setup. The tuning of these parameters is not an easy task, and is not always relevant for all the considered classes.

• Uncertainties due to the lack of a perfect model structure. Even a representation of data in a high-dimensional space does not guarantee that the data is separable (i.e. find a boundary that separates the output classes).

As a consequence, decision functions should provide a confidence value which allows to judge the certainty or the belief of the output. Moreover, these approaches do not adapt to the user background, nor to the specific semantics of the information sought by him in an image retrieval system, i.e. the meaning sought by a given user in the image content with respect to his background and within (or not) a specific domain application. Besides, in these approaches, the image semantics is often limited to its perceptual manifestation. Nevertheless, as shown in Figure 1.5, the image semantics is not (always) explicitly stored in its visual content, but it is rather an emergent property of the interaction of the user and the image content [START_REF] Santini | Emergent semantics through interaction in image databases[END_REF]. Very often, it also depends on prior knowledge on the domain and on the context of use of the image [Hudelot, 2005, Alm, 2006]. Consequently, we consider that semantic image annotation is a knowledge-based process.

In Figure 1.5, we have illustrated these ideas with some examples where the meaning is not explicitly encoded in the visual appearance of images. For instance, only people with deep knowledge in astronomy may know that the image 1.5(g) depicts the Crab Nebula which is a supernova remnant and pulsar wind nebula in the constellation of Taurus. Also, understanding that the image 1.5(a) depicts love requires a cognitive analysis based on knowledge about human behaviors. Also, the understanding of the image 1.5(e) requires a comprehensive image analysis and a deep cognitive process to figure out that it depicts the global warming. Indeed, one can observe in this image that the ice floe is small and it is crowded by penguins. Figure 1.5(f), which illustrates a fractal Broccoli, is usually interpreted by common people as depicting a Broccoli, but many mathematicians and physicists will only see in it a fractal equation of the form D = lim S→0 ln(N ) ln( 1 S ) , where N is the number of closed balls of diameter S needed to cover the object. So, image interpretation is subjective to the user background knowledge and to the user objective, and therefore automatic image annotation approaches should incorporate knowledge models in order to adapt to the requested semantics by the user in an image retrieval system. Finally, Figures 1.5(c) and 1. 5(d) represent two different concepts but sharing the same name "Jaguar". This refers to the polysemy problem which is very common in information retrieval field. This problem requires to analyze the user query in order to understand which of both concepts the user is looking for, and consequently, there is a need for semantic structures that allow to make explicit the semantics of concepts. depict Jaguars, (e) depicts a global warming scene, (f) depicts a fractal broccoli, and (g) depicts the Crab Nebula which is the result of a supernova that occurred in 1054 AD. This image was taken by the Hubble telescope, and in the center is a pulsar, or a neutron star.

To sum up, automatic image annotation is a very promising and important issue to improve image search and retrieval. Nevertheless, most of the current approaches are still insufficient to satiate the user need due to the following problems:

• Uncertainty introduced by machine learning algorithms.

• Scalability problems in terms of the image number, the dimension of the representation space of image visual features, and the concept number (dimension of the semantic space).

• Semantic gap problem.

• Subjectivity of image semantics (these approaches do not adapt to the user background).

• Sensitivity to the accuracy of the ground truth of the learning dataset.

• Polysemy problem.

Therefore, automatic image annotation is still a challenging problem and current approaches need to be improved. In particular, some recent work have Chapter 1. Introduction proposed to model and to use explicit knowledge about image context in order to answer to the above mentioned problems, as for instance [START_REF] Hollink | Adding spatial semantics to image annotations[END_REF], Maillot et al., 2004, Popescu et al., 2007, Fan et al., 2008a, Hudelot et al., 2008, Tousch et al., 2008, Wu et al., 2008, Simou et al., 2008, Dasiopoulou et al., 2009, Fan et al., 2009, Straccia, 2010, Li et al., 2010]. We refer to these approaches as semantic image annotation, which can be defined as the set of methods mainly based on the use of explicit semantic structures, such as semantic hierarchies and ontologies, in order to narrow the semantic gap. However, semantic image annotation approaches are currently facing the following shortcomings:

• The availability of knowledge structures modeling the image semantics, or failing that, their building or their learning. Indeed, currently there are a number of available knowledge resources which have been used for image annotation, like WordNet [Fellbaum, 1998] or Wikipedia11 , and also multimedia knowledge resources such as LSCOM [START_REF] Naphade | Large-scale concept ontology for multimedia[END_REF], ImageNet [START_REF] Deng | ImageNet: a large-scale hierarchical image database[END_REF], and so on. However, these knowledge sources are not completely suitable for image annotation, since they are built using a conceptual specification which not necessarily reflects image semantics, i.e. they do not consider image content.

• The lack of an unified model for knowledge representation and reasoning.

• Only few approaches have considered this knowledge in a formal framework and exploited reasoning on it.

• At last, unlike other domains, multimedia knowledge extraction is still in its infancy, i.e. much efforts are needed to assess the strength and the limits of these approaches on real data applications.

Objectives

In this dissertation, our objective is to make advances in the field of semantic image annotation by the proposition of new knowledge-based frameworks dedicated to image annotation. Indeed, the achieved contributions in this thesis were motivated by the following assumptions:

• Image semantics is not fully included in the image itself, and therefore the use of explicit structured-knowledge models is necessary for performing accurate image annotations.

• Existing knowledge models are not fully adequate for image annotation. These are either: i) very generic knowledge resources (commonsense ontologies such as WordNet and Wikipedia) which do not reflect image semantics, or ii) built automatically using available data, but often only a single piece of information on images is considered (for instance, visual information in [START_REF] Sivic | Unsupervised discovery of visual object class hierarchies[END_REF] or conceptual information in [START_REF] Snoek | Adding semantics to detectors for video retrieval[END_REF]). Therefore, a new paradigm for building knowledge models should be adopted and which should incorporate the different image modalities (e.g. visual, conceptual, contextual, spatial, etc.) in order to be relevant with respect to image semantics.

• The building of these knowledge models should be performed automatically, using data mining techniques, in order to reduce the scalability problem of knowledge building and to achieve a consistent representation of image semantics.

Consequently, the proposed contributions involve capturing and modeling suitable knowledge models about the image semantics, and thereafter building effective tools to exploit this knowledge in order to improve the image annotation process. The overall goal remains to narrow the semantic gap using the available visual features of images and relevant domain knowledge in order to support the varied search categories, ultimately to satiate the user expectations. Specifically, we address the following issues:

1. The building of explicit knowledge models, in an automatic and effective manner, for the purpose of image annotation.

2. The use of these knowledge models with the aim of improving the image annotation accuracy, through:

(a) An hierarchical image classification framework.

(b) A fuzzy ontological reasoning framework.

Therefore, we believe that the outcomes of our work will allow to address the following problems:

• Narrowing the semantic gap, i.e. providing effective tools that allow mapping the visual features of images into semantics concepts.

• Reducing the uncertainty introduced by machine learning algorithms by the use of fuzzy ontological reasoning with the aim of assessing the consistency of image annotations.

• Reducing the following problems: i) polysemy, ii) subjectivity of image semantics, and ii) sensitivity to the accuracy of the ground truth, by providing explicit semantic structures allowing the reasoning on concepts semantics.

• Reducing the scalability problem of automatic image annotation by the proposition of a hierarchical image classification method, which scales well with large databases.

Chapter 1. Introduction

Contributions

As aforementioned, the aims of this thesis are twofold. As a first step, we have focused our contributions on providing effective tools for the automatic building of structured and explicit knowledge models dedicated to image annotation. This is intended to propose structured knowledge models that are relevant with respect to image semantics. Subsequently, we have concentrated our contributions on the effective use of these knowledge models in order to improve image annotation. Our final purpose is to narrow the semantic gap, i.e. decrease the gap between the visual content of images and their meaning (image semantics). The achieved contributions in this dissertation are detailed in the following subsections.

Building Explicit and Structured Knowledge Models for Image Annotation

Our first contribution in this dissertation deals with the automatic building of semantic hierarchies for the purpose of image annotation. Indeed, as it is detailed in Chapter 3 -Section 3.2.4, our standpoint is that a semantic hierarchy dedicated to image annotation should take account of the visual information, the conceptual information and the contextual one. Consequently, we propose a new image-semantic measure which allows estimating the semantic similarity between image concepts, and which incorporates the aforementioned information in order to provide a meaningful measure of image semantics. Subsequently, a new methodology, based on the previously proposed measure and on a set of effective rules, is proposed in order to automatically build semantic hierarchies suitable for image annotation. This contribution is detailed in Chapter 3, and has served for the following publications [Bannour andHudelot, 2012a, Bannour and[START_REF] Bannour | Combinaison d'information visuelle, conceptuelle, et contextuelle pour la construction automatique de hiérarchies sémantiques adaptées à l'annotation d'images[END_REF]. Thereafter, we propose to go further in the modeling of image-concepts relationships, and consequently the modeling of image semantics. We propose therefore a new approach for building an ontology of spatial and contextual information suitable for reasoning about the consistency of image annotation. Our approach uses visual and conceptual information in order to build a semantic hierarchy that will serve as a backbone of our ontology. Contextual and spatial information about image concepts are afterwards incorporated in the ontology in order to model richer semantic relationships between these concepts. Fuzzy description logics are used as a formalism to represent our ontology and the inherent uncertainty and imprecision of this kind of information. This work is the subject of Chapter 4, and is published in [Bannour and Hudelot, 2013a].

Figure 1.6, illustrates the general workflow of the proposed approaches for building structured knowledge models dedicated to image annotation.

Chapter 1. Introduction

Annotations of Images

Using Structured Knowledge Models for Image Annotation

In order to make an efficient use of the built structured knowledge models, we propose in this dissertation to investigate the contribution of: i) semantic hierarchies, and ii) DL ontologies and ontological reasoning, for image annotation. The overall workflow of our methods is illustrated in Figure 1.7. Given a set of previously unseen (unlabeled) images and a structured knowledge model dedicated to image annotation, our approach allows to predict the set of concepts from the annotation vocabulary that are relevant with respect to the image content/semantics.

Image annotation

Image Dataset Data Processing Information Analysis

Knowledge Driven Framework

Image annotation

Figure 1.7: Using structured knowledge models for image annotation.

We propose therefore a new methodology, based on the structure of semantic hierarchies, to efficiently train hierarchical classifiers. Our method allows decomposing the problem into several independent tasks and therefore it scales well with large databases. We also propose two methods for computing a hierarchical decision function that serves to annotate new image samples. The former is performed using a top-down classifiers voting, while the second is based on a bottom-up score fusion. This methodology is detailed in Chapter 5, and is published in [Bannour and[START_REF] Bannour | Hierarchical image annotation using semantic hierarchies[END_REF]Hudelot, 2013b].

Finally, we propose a novel approach for image annotation, based on hierarchical image classification and a multi-stage reasoning framework for reasoning about the consistency of the produced annotation. In this approach, fuzzy ontological reasoning is used in order to achieve a relevant decision on the belonging of a given image to the set of concept classes. The ontological reasoning is performed using the previously built multimedia ontology (in Chapter 4), and involves reasoning about contextual and spatial information of image concepts. This framework is introduced in Chapter 6, and is published in [Bannour and Hudelot, 2013a].

Organization of this Dissertation

This dissertation is organized as follows.

In Chapter 2, we present a thorough survey on relevant research topics to the image retrieval domain. The covered topics include: image retrieval, automatic image annotation, hierarchical image classification and knowledge-driven approaches for image annotation. We also propose a thorough discussion of the related work in our contribution chapters, so as to put our proposals into perspective.

The main contributions of this dissertation start with Chapter 3. Indeed, we propose in Chapter 3 a new approach for building semantic hierarchies suitable for image annotation. Our approach takes advantages of the different image modalities to compute an image-semantic measure between the different concepts of the annotation vocabulary. This measure is thereafter used to connect the concepts with a higher semantic relatedness till the building of the final hierarchy.

Chapter 4 proposes to go further in the use of structured knowledge models for image annotation. Specifically, we address the following issues: how to extract and store knowledge about image context in an effective way, how to query the ontology to retrieve valuable information about image context, and how to perform reasoning using multimedia ontologies in order to reduce the uncertainty about image annotations. We therefore propose to automatically build multimedia ontologies by mining image databases to gather valuable information about image context. Thus, we show that ontologies are not necessarily limited to be defined by humans, as it was the case for most of existing approaches in the multimedia domain. Thereby, we reduce the scalability problem of ontology building.

In Chapter 5, we explore the contribution of semantic hierarchies for image annotation. A natural way to use semantic hierarchies is to operate within a framework for hierarchical image classification. Consequently, we propose a new approach based on the semantic hierarchy structure to efficiently train the hierarchical classifiers, and for computing an accurate hierarchical decision function which is thereafter used to annotate previously unseen images.

In Chapter 6, we explore the use of formal ontologies and fuzzy ontological reasoning for improving image annotation. Our intent is to reduce the uncertainty of image annotation introduced by the machine learning algorithms. Therefore, we pro-Chapter 1. Introduction pose a new approach for image annotation based on hierarchical image classification and a multi-stage reasoning framework for reasoning about the consistency of the produced annotation.

Finally, this dissertation is concluded in Chapter 7 with a feedback on our contributions and a discussion on the directions that can be borrowed by the presented research topics.

Chapter 2. State of the Art on Image Annotation

Introduction

As introduced in the previous chapter, image indexing and retrieval has been a very active research domain since two decades, hence many approaches have been proposed to solve this problem. These different approaches can be classified in several different ways and from different points of views, as for example, the application domain, the indexing technique, the used content (modality) for image description, and so on. In Figure 1.2, we have proposed a taxonomy to classify image retrieval approaches, based on the used modality for describing image content. In particular, as introduced in Section 1.1, we consider two main families of approaches: i) Text-based approaches: which do not consider the image content during the indexing process, but rely on the only surrounding textual information (HTML tags, metadata, contextual text surrounding the image, etc.), and ii) Content-based approaches: which are basically based on the use of image content (visual and semantic content) in order to index and retrieve images.

The scope of this dissertation is content-based image retrieval, and in particular content based-image annotation. Thus, as illustrated in Figure 2.1, we focus in this chapter on this family of approaches, and specifically on automatic image annotation approaches and semantic image annotation approaches. This chapter is structured as follows. Section 2.2 presents a brief introduction to the image annotation problem. In Section 2.3, we introduce what is meant by image semantics, and we put in evidence the different challenges in order to achieve effective image annotation. In Section 2.4, we propose a review of classical image annotation approaches and we discuss their limitations. Section 2.5 proposes, as a first contribution, a state of the art on the use of implicit/explicit knowledge-based models to improve the annotation process. Section 2.6 proposes a general discussion about existing approaches for automatic image annotation. This chapter is concluded in Section 2.7.

Ontology driven approaches

Information fusion

Semi-Supervised Learning Figure 2.1: Proposed taxonomy of image indexing and retrieval approaches. The scope of our work is in blue boxes.

The Image Annotation Problem

Automatic image annotation can be defined as the process of automatically assigning a text description (often reduced to a set of semantic keywords) to a digital image through a computational model (or a computer system). Automatic image annotation is usually used in image retrieval systems in order to index and retrieve images of interest from a large database. Very often, this task is regarded as an image classification problem (usually a multiclass classification problem), and is involved by the following steps:

1. Gather a training image dataset consisting of a set of images with their textual annotations. These textual annotations generally consist of a set of high-level concepts (semantic concepts) depicting image content, and are usually called the ground truth. The set of all concepts composed the annotation vocabulary.

2. Build a computational model enabling to find a correspondence model between the low-level or mid-level representations of images and the concepts of the annotation vocabulary.

3. Test the system and adjusting the parameters of the computational model.

Problem Formalization.

In a formal way, automatic image annotation can be defined as follows:

Chapter 2. State of the Art on Image Annotation Given:

• DB, a training image database consisting of a set of pairs image/textual annotation , i.e.

DB = {[i 1 , A 1 ], [i 2 , A 2 ], • • • , [i L , A L ]}
, where:

-

I = i 1 , i 2 , • • • , i L is the set of all images in DB,
-L is the number of images in the database.

-C = c 1 , c 2 , • • • , c
N is the annotation vocabulary used for annotating images in I,

-N is the size of the annotation vocabulary.

-A i is a textual annotation consisting of at least a set of concepts {c j ∈ C, j = 1..n i i } associated with a given image i i ∈ DB. According to the used dataset, these textual annotations may also contain other pieces of information, such as information about depicted concepts (bounding box, truncated or not, etc.), image source, user tags and so on.

The objective of automatic image annotation is to build a computational model that enables to associate a set of concepts {c j ∈ C, 1 ≤ j ≤ N } to any given image i i ∈ I. The overall goal is to extend this computational model to previously unseen images (i.e. ∀ i x / ∈ DB) in order to provide them a textual description. Figure 2.2 illustrates the aim of automatic image annotation by an example.

Terminology.

For a sake of clarity, we define in this section the used terms and their meaning. Thereby, we refer to:

• Concept: as a label (or a word) with a precise semantics (a given meaning) used for annotating images. For instance, in Figure 2.2 "Killer Whale" and "Penguin" are concepts. Furthermore, according to our definition a label may correspond to several concepts, as for instance in Figure 1.5, where concept "Jaguar" refers to an animal (c), and to a car (d).

Chapter 2. State of the Art on Image Annotation

• Annotation vocabulary: as the set of concepts used for annotating the images of a given database.

• Image features: as a set of low-level or mid-level descriptors used for representing the visual content of an image.

• Visual similarity, conceptual similarity, contextual similarity as follows:

Definition 1 (Visual similarity). The visual similarity is a measure reflecting the similarity between concepts with respect to their visual appearances.

Definition 2 (Conceptual similarity). The conceptual similarity is a measure reflecting the semantic relatedness between two concepts from a linguistic and/or a taxonomic point of view. With respect to image retrieval field, this measure is usually computed using textual information sources (such as the surrounding context, user tags, etc.), or using commonsense ontologies (such as WordNet [Fellbaum, 1998], Wikipedia1 , etc.).

Definition 3 (Contextual similarity). The contextual similarity is a measure that enables to group the concepts that share a same context. With respect to image annotation, we define this context as the context of appearance (co-occurrence) of concepts in the image database.

Image Semantics

Many recent image annotation approaches have been proposed to narrow the semantic gap problem defined by [START_REF] Smeulders | Content-based image retrieval at the end of the early years[END_REF] as the lack of coincidence between the low-level image descriptions using visual features and the richness of human semantics, i.e. "the interpretation that the image data have for a user in a given situation". According to this definition, images get their semantic meaning as an act of image interpretation or understanding, and it is consequently difficult for an image retrieval system to discern the meaning sought by a user when he is searching for a particular image. This is not a new idea and as recalled by [START_REF] Boucher | Comment extraire la semantique d'une image?[END_REF], the fundamental work of [START_REF] Treisman | A featureintegration theory of attention[END_REF], Marr, 1982, Biederman, 1987] have previously carried this insight. Nevertheless, even if the notion of image semantics seems to be important for image retrieval related tasks, it is still vague and can vary according to the different approaches. Indeed, to our knowledge, only few work have tried to provide a precise definition of the notion of image semantics. Therefore, we propose in this section a tentative to define, through some examples, this notion of image semantics according to our objectives. Then, we relate our definition to the existing ones that have been sparsely defended in the literature.

Our Standpoint on Image Semantics

Semantics is the study of meaning. In linguistics, it is also the study of the meaning or the interpretation of a word, sentence, or other language form. With respect to the image retrieval field, we can define it as the meaning sought by the user in the image content. Figure 2.3 illustrates some images and their associated semantic meaning.

Definition 4 (Image Semantics). With respect to the image retrieval field, image semantics can be defined as the meaning sought by the user in the image content.

As illustrated in Figure 2.3, the intended meaning may be obvious, intuitive and explicit which make it easy to access. However, this meaning is also very often implicit, induced and subtle making it difficult to discern, specifically through a multimedia information retrieval (MIR) system. For instance, in the images 2.3(a) and 2.3(b), how one can define a spectacular jump? How an image retrieval system can model or access this information? As well in the images 2.3(f) and 2.3(g), what is the difference between a race track and a road? How machines can infer this information directly from image pixels? And finally, in Figures 2.3(c),2.3(d) and 2.3(e), the global warming depicted in these images is subtle. Indeed, we can see in the image 2.3(d) a polar bear who has trouble standing on a tiny ice, and in the image 2.3(c) a piece of ice overcrowded by penguins. Based on their background knowledge, humans know that the global warming is responsible of the arctic ice melting. Therefore, humans can infer from these images that the survival of the Antarctic animals, such as penguins and polar bears, are threatened by the ice melting caused by the global warming.

Consequently, our first view on image semantics is that it is not fully, nor explicitly stored in the image pixels, and it is usually hard for a machine to access the image semantics using only image features. Indeed, as stated in [START_REF] Santini | Emergent semantics through interaction in image databases[END_REF], image semantics is rather an emergent property of the interaction of the user and the image content. We can therefore conclude that the image interpretation process requires often a reasoning mechanism over the detected objects in the image, which is usually based on cognition and on the past experiences.

Accordingly, in order to match user expectations in an image retrieval system, it is important to provide tools that allow building and using knowledge models representing the image context (and consequently the image semantics). These will allow reproducing the reasoning process in order to deduce a more consistent image annotation/interpretation. Specifically, we believe that these knowledge models should go further than the simple description of specific objects that may appear in images, and rather model the image context through the description of concepts and the semantic relationships between them. For instance, a good description of the image 2.3(d) is: a "polar bear" is standing on a "tiny ice". This description could probably allow inferring that this image depicts a global warming, since it is meaningless that a big bear is standing on a tiny ice unless he was obliged to be. Hence, if it was obliged to do so, it is possibly because large ice floes have melt. The reason of ice melting could therefore be intuitively deduced.

Chapter 2. State of the Art on Image Annotation

Let us look at a definition for the image semantics from the perspective of the theory of meaning.

Definition 5 (Image Semantics 2). Referring to the theory of meaning, image semantics can be defined as the meaning sought by the user in the image content within a particular context.

According to the above definition, we can notice that the context of searching is very important for understanding the image semantics thought and sought by a user. For instance, depending on the observer's background knowledge, the image in Figure 2.4(a) can be interpreted as: i) a statue, ii) a statue of a Carthaginian general, iii) a statue of Hannibal, iv) a statue of Hannibal counting the signet rings of Roman nobles killed during the battle, statue by Sébastien Slodtz, 1704, Louvre. Also, the image in Figure 2.4(b) can be interpreted as: i) airplane, ii) military aircraft, iii) an aircraft of the Second World War, iv) the Swordfish, which is a two-place torpedo bomber used during the Second World War. All these interpretations are relevant with respect to the observer's background, his past experiences, and the semantics level sought by him in these images. These examples enable us to put into perspective our two next views on image semantics: i) image semantics is a multi-level paradigm, and ii) image semantics is context-sensitive, i.e. it depends on the user's objective and on his background knowledge. Consequently, one of the major challenges of image annotation and image retrieval systems is to be able to extract these different levels of image semantics and to adapt themselves to the user's objective in order to be efficient and useful. For example, a user would like to go beyond a query like "I seek an image that visually looks like this one" or "an image that contains a car", and would prefer to be able to ask queries such as "find me an image that contains a spectacular jump of a killer whale" or "a figure which depicts the global warming" or "find me a picture depicting cars on a racetrack" -cf. Definition 6 (Image Semantics 3). Image semantics is a multi-level paradigm, i.e. there are several levels of semantics (or interpretation) for a given image and the major challenge of image retrieval systems is then to be able to extract such semantics from images and to adapt to the user background in order to be efficient and useful.

In Figure 2.5, we illustrate the multi-level paradigm of image semantics. In particular, we distinguish three semantic levels: objects, partial semantics and full semantics. For instance, if we look at image (d) of Figure 2.4, the semantics at the object level could be {"Bear", "Iceberg"}, the semantics at the partial level could be "Polar Bear standing on a small iceberg" and the semantics at the full level could be "global warming threatens the survival of the polar bears". Therefore, we can notice that the difficulty of processing and extracting the semantics from images increases significantly according to the sought level of abstraction. Currently, most approaches for image retrieval deals with the first level of semantic content. These approaches target to provide efficient methods to learn semantics classes from visual image features. They allow therefore providing a description for images in the form of a set of keywords (e.g. Car, Building, Sky, Dog, etc.). Other approaches are trying to tackle the second level of semantics, i.e. the partial semantics level. These approaches aim at recognizing the concepts (objects) depicted in an image and subsequently try to identify the semantic relationships between them (e.g. a person eating an apple, penguins on ice floes, etc.). The last category of semantic description, i.e. the real context one, consists in identifying the full semantic context of a given image (e.g. see the description of images 2.3(c) and 2.4(a)). To our knowledge no attempts have focused on this kind of semantic description because of its difficulty, and the lack of a computational model allowing to model/infer such knowledge.

As a consequence, our last view on image semantics is that contextual knowledge, i.e. the semantic relationships between the different concepts, are very important information to achieve image annotation at the partial or the full semantics level.

In the next section, we show that these views on image semantics have been partially defended in the literature.

Related Work on Image Semantics

Following the idea of [Marr, 1982], some work have proposed multiple-level paradigms for image representation. [Jaimes and fu Chang, 2000] proposed a pyramidal indexing structure composed of 10 levels visual structure. The authors have considered that these levels can be divided into perceptual levels (i.e. the syntactic description of the image) and into semantic levels (i.e. the semantic description of the image). They differentiated 6 different semantic or conceptual levels consisting of: generic objects, generic scene, specific objects, specific scene, abstract objects and abstract scene. These 6 different conceptual levels can be related to the three different conceptual image description levels proposed by [Shatford, 1986, Shatford, 1994], i.e. generic, specific and abstract content which can be themselves considered as a generalization to generic image of the 'pre-iconographic', 'iconographic' and 'iconologic' levels of expression proposed by the art historian [Panofsky, 1972]. The authors have also stated that a priori knowledge is very useful to interpret images at high semantic levels, i.e. the higher the level is, the more knowledge is involved in the interpretation of images.

In [START_REF] Enser | Towards a comprehensive survey of the semantic gap in visual image retrieval[END_REF], the authors proposed to extend a model initially proposed by [Jörgensen, 1996, Jörgensen, 1998] and proposed to separate the image content into perceptual, generic-interpretive, specific-interpretive and abstract attributes. Interpretive attributes correspond to the semantic level and "are those which require both interpretation of perceptual cues and application of a general level of knowledge or inference from that knowledge to name the attribute" [Jörgensen, 1996].

In [START_REF] Santini | Emergent semantics through interaction in image databases[END_REF], the authors proposed a discussion on image semantics and its role in the design of image databases. In particular, the authors defended the interesting idea that the semantics is not an intrinsic property of the images but an emergent property which depends on the interaction between the user and the database. In particular, the authors emphasize the idea that the semantics depends on the contextual knowledge and on the cultural background of the user. Similar ideas are defended in [START_REF] Hudelot | Symbol grounding for semantic image interpretation: From image data to semantics[END_REF] in the context of image understanding. [Enser, 1993, Armitage and Enser, 1997, Eakins, 2002] have studied the nature of the image database query task. For instance, [Eakins, 2002] mentioned three kinds of queries in CBIR: 1) retrieval by primitive or perceptual features (e.g. find pictures sharing visual features with this query image), 2) retrieval by naming objects (e.g. find pictures of "Car"), and 3) retrieval by abstract attributes which involves reasoning about the objects and their relationships by taking into account different kinds of knowledge (contextual knowledge, cultural knowledge, personal background knowledge and domain knowledge). The authors have considered that the two last kinds of query are situated at the semantic level.

In [START_REF] Hare | Bridging the semantic gap in multimedia information retrieval: Top-down and bottom-up approaches[END_REF], the authors tried to characterize the semantic gap. In particular, they made the distinction between the gap between image descriptors and image labels and the gap between object labels and the full semantics. For the authors, full semantics implies to consider not only high-level concepts but also the relationships between these different concepts.

Finally, the key work of [START_REF] Smeulders | Content-based image retrieval at the end of the early years[END_REF] remains the most complete currently existing work setting out the image semantics problem. The authors have succeeded to highlight the main challenges of image retrieval systems, even if many of the mentioned issues have been partially identified before. Specifically, among their statements, the authors mentioned that "on the coverage side, labeling is seldom complete, context sensitive, and, in any case, there is a significant fraction of requests whose semantics can not be captured by labeling alone". However, according to the current advances in image retrieval field, automatic image annotation remains the only alternative to solve the problem of semantic gap and image retrieval related tasks. Indeed, we believe that, in anyway, automatic image annotation is the preliminary step for any process of image description and interpretation. Nevertheless, as aforementioned by the previous work, the image semantics is a subjective property acquired by the end user during the image interpretation process. Therefore, we believe that the use of knowledge models about image semantics and the use of inference mechanisms are priceless in order to automatically perform semantic image annotation and interpretation.

An Overview of Classical Image Annotation Approaches

The rapid growth of multimedia content comes with the need to effectively manage this content by providing mechanisms for image indexing and retrieval that can meet user expectations. Towards this goal, semantic image analysis and interpretation has been one of the most interesting challenges during this last decade, and several attempts have addressed the, previously introduced, semantic gap problem.

In particular, a typical method for narrowing the semantic gap is to perform automatic image annotation. Automatic image annotation was introduced in the early 2000s, and first efforts focused on statistical learning approaches as they provide powerful and effective tools to establish associations between the visual features of images and the semantic concepts [START_REF] Barnard | Matching words and pictures[END_REF], Lavrenko et al., 2003, Monay and Gatica-Perez, 2003, Carneiro et al., 2007]. A recent review on automatic image annotation techniques is proposed in [START_REF] Zhang | A review on automatic image annotation techniques[END_REF].

Early efforts aiming to narrow the semantic gap have focused on providing mechanisms/methods for mapping low-level features (such as color, texture, shape and salient points) directly to some specific semantic concepts such as indoor/outdoor [START_REF] Szummer | Indoor-Outdoor image classification[END_REF]], nature, animal, food [START_REF] Smith | Visually searching the web for content[END_REF], pedestrian [START_REF] Papageorgiou | Trainable pedestrian detection[END_REF] and so on. Therefore, many dedicated detec-Chapter 2. State of the Art on Image Annotation tors based on simple decision rules have been proposed. We refer to these approaches as specific-concepts dedicated approaches, since they are dedicated to a specific set of concepts with a specific low-level representation of images. These approaches have quickly become cumbersome and impractical following the normal request of a larger annotation vocabulary. Indeed, it would be impossible to build a detector for each potential concept, as they are too many. To overcome the limitations of the methods dedicated to specific concepts, other approaches have proposed to build generic detectors/classifiers. More precisely, given a set of positive and negative training samples, generic classifiers are trained separately using a single approach. We refer to these approaches as image classification approaches or concept-detectors based approaches if they also aim at localizing the concept in the image. However, since these approaches do not consider concept-specific knowledge/properties, they may be less accurate than specific-concepts based methods, but allow in the other hand to deal with a large-scale annotation vocabulary [START_REF] Snoek | The challenge problem for automated detection of 101 semantic concepts in multimedia[END_REF]. In the remaining of this section, we review some of the existing approaches for image annotation, emphasizing each time the difference between each category of approaches and their limitations.

Concept-Detectors Based Approaches

As aforementioned, the concept-detectors based approaches (also called object detection approaches) aim at building a (set of) specific concept detector(s), i.e. a dedicated detector to each specific concept/object of interest. The main difference between this category of approaches and image classification based approaches, is that the former one requests to slide a window across the image (possibly at multiple scales), and to classify each such local window as containing the target concept or a background [START_REF] Papageorgiou | A trainable system for object detection[END_REF], Viola and Jones, 2001, Mohan et al., 2001, Dalal and Triggs, 2005, Torralba et al., 2007, Wei and Tao, 2010]. Consequently, concept-detectors based approaches allow identifying and localizing the recognized concepts in processed images, i.e. predicting in a given image the bounding box and the label of each detected concept from the annotation vocabulary -cf. Figure 2.6. While the image-classification-based approaches allow only predicting in a given image the presence/absence of a set of concepts from the annotation vocabulary (c j ∈ C).

More precisely, the concept-detectors based approaches, also called sliding-window object detection methods, consist in defining a fixed-size rectangular window and applying a classifier, such as boosting [START_REF] Viola | Rapid object detection using a boosted cascade of simple features[END_REF] or support vector machine [START_REF] Papageorgiou | A trainable system for object detection[END_REF], to the sub-image defined by the window. The classifier extracts image features from within the window and returns the probability that the window bounds a particular object. The process is repeated on successively scaled copies of the image so that objects can be detected at any size.

This technique has shown impressive results for some specific concepts such as face detection [Osuna et al., 1997, Viola andJones, 2001] or pedestrian detection [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF], but these approaches have failed to generalize to other concepts. Doubtless, the most famous object/concept detector is the one proposed by [START_REF] Viola | Rapid object detection using a boosted cascade of simple features[END_REF] for face detection, as it provides very competitive object detection rates in real-time. They proposed to use a cascade of weak classifiers in order to create a strong classifier with a good detection rate. As illustrated in Figure 2.7, a sub-window (also called a patch) of a given image is classified as containing a face only if it passes tests in all nodes in the cascade. Usually, non face patches are quickly rejected by the early nodes. [START_REF] Viola | Rapid object detection using a boosted cascade of simple features[END_REF] proposed to use: i) the integral image representation to compute rapidly the image features (called rectangle features), and ii) the AdaBoost algorithm [START_REF] Schapire | Boosting the margin: A new explanation for the effectiveness of voting methods[END_REF]] to select rectangle features and combine them into a cascade of classifiers. [START_REF] Viola | Rapid object detection using a boosted cascade of simple features[END_REF].

Chapter 2. State of the Art on Image Annotation [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF] proposed a method for pedestrian detection which achieves high accuracy. They proposed to use a single filter on Histogram of Oriented Gradients (HOG) features to represent an object category. This detector uses a sliding window approach, where a filter is applied at all positions and scales of an image. The classifier allows to detect whether or not there is an instance of the target concept at a given position and scale. Their method has shown very good performance on the INRIA human dataset. [START_REF] Mikolajczyk | Human detection based on a probabilistic assembly of robust part detectors[END_REF] proposed a method for human detection based on a probabilistic assembly of robust part detectors. These parts were represented by co-occurrences of local features which capture the spatial layout of the part appearance. Separate detectors were then trained for each part using AdaBoost algorithm. The target location was determined by maximizing the joint likelihood of body parts occurrences combined according to the geometric relations.

Other approaches have tackled the problem of concept detection in a generic way, i.e. without targeting specific concepts. [START_REF] Torralba | Sharing features: efficient boosting procedures for multiclass object detection[END_REF], Torralba et al., 2007] proposed an approach allowing to share features among different object categories, thus providing a faster solution for multi-classes object detection. [START_REF] Lampert | Beyond sliding windows: Object localization by efficient subwindow search[END_REF] proposed to consider the object detection problem as finding the optimal bounding box that gives the highest detection score in the image. They proposed an efficient variant of the branch-and-bound method for retrieving the optimal bounding box in the image. In [START_REF] Felzenszwalb | Object detection with discriminatively trained part-based models[END_REF], an object detection system based on mixtures of multiscale deformable part models is proposed. This system is able to represent highly variable object classes and achieves state-of-the-art results in the PASCAL object detection challenges.

Object detection is one of the fundamental tasks of visual recognition and computer vision, and consequently the literature is very rich concerning this topic. The aim of this section is not to provide an exhaustive review of this kind of approaches, i.e. concept-detectors based approaches. Therefore, for more information the reader is suggested to refer to [Galleguillos and Belongie, 2010].

Discussion

Concept-detectors based approaches have shown good accuracy for detecting specific objects/concepts, such as faces, pedestrians, cars, etc. These approaches select the parameters of the model so as to minimize the detection error on a set of training images. Such approaches aim to directly optimize the decision boundary between positive and negative samples [START_REF] Felzenszwalb | Object detection with discriminatively trained part-based models[END_REF]. However, these approaches seem unlikely to scale up to the detection of a large number of concepts, or to many different views of objects, since each classifier computes many image features independently [START_REF] Torralba | Sharing visual features for multiclass and multiview object detection[END_REF]. These features typically involve convolutions with part templates [START_REF] Fergus | Object class recognition by unsupervised scale-invariant learning[END_REF] or with a set of basis filters [Viola andJones, 2001, Dalal andTriggs, 2005]. Therefore, computing these features is slow and requires a big set of training samples to find the useful features.

Moreover, sliding window based approaches are computationally-expensive. For instance, given an image of size n * n and a window of size r * r, a histogram-based sliding window approach needs to scan n2 windows, to scan r 2 pixels per window to Chapter 2. State of the Art on Image Annotation construct the histogram, and to scan B bins of the histogram to evaluate the objective function [START_REF] Wei | Efficient histogram-based sliding window[END_REF]. Thus, the overall complexity of this kind of approach is O(n 2 (r 2 + B)), which is unaffordable when either n, r or B is large. Many attempts have been proposed to reduce the complexity of sliding window based approaches, but despite this, the complexity is still very high with respect to current needs for detecting concepts in images and videos.

Image Classification Based Approaches

Image classification based approaches allow predicting the presence/absence of a set of concepts (from the annotation vocabulary c j ∈ C) in a given image. These approaches usually follow the scheme illustrated in Figure 2.8. These, either extract:

• global features (computed on the whole image or by the use of dense sampling) [START_REF] Tong | Support vector machine active learning for image retrieval[END_REF], Monay and Gatica-Perez, 2004, Carneiro et al., 2007, Lazebnik et al., 2006], or

• require a prior segmentation of the image as regions/blobs 2 [START_REF] Barnard | Learning the semantics of words and pictures[END_REF], Carson et al., 2002, Duygulu et al., 2002, Blei and Jordan, 2003, Romdhane et al., 2010], or as blocks [START_REF] Feng | Multiple bernoulli relevance models for image and video annotation[END_REF], Carbonetto et al., 2004]. Thereafter, these approaches carry out the extraction of features. A feature is defined to capture a certain visual property of an image, either on the whole image or on a region of it. Most commonly used features for image classification are the ones reflecting: color, texture, shape, and salient points in images. Feature description is subsequently performed to assign a signature to the extracted features. Finally, a machine learning algorithm is trained to recognize/detect the concepts from the annotation vocabulary using the visual features of images.

Early efforts in image classification based approaches have focused on the extraction of reliable specific semantics, for instance, indoor vs. outdoor [START_REF] Szummer | Indoor-Outdoor image classification[END_REF]], city vs. landscape [START_REF] Vailaya | On image classification: City vs. landscape[END_REF]] and human vs. horses [START_REF] Forsyth | Body plans[END_REF], buildings vs. non-building [START_REF] Li | Consistent line clusters for building recognition in cbir[END_REF], etc.

These approaches have addressed the problem of extracting semantics using supervised learning techniques.

Consequently, Bayesian classifiers [START_REF] Vailaya | Image classification for content-based indexing[END_REF], Support Vector Machines [START_REF] Griffin | Learning and using taxonomies for fast visual categorization[END_REF], multiple instance learning [START_REF] Zhang | Content-based image retrieval using multiple-instance learning[END_REF], statistical models [START_REF] Hastie | The Elements of Statistical Learning[END_REF], k -NN [START_REF] Tang | Image annotation by knn-sparse graph-based label propagation over noisily tagged web images[END_REF], and artificial neural networks [START_REF] Romdhane | IMIOL: a system for indexing images by their semantic content based on possibilistic fuzzy clustering and adaptive resonance theory neural networks learning[END_REF] are often used to learn high-level concepts (i.e. semantic content) from low-level features (i.e. visual content). This is achieved by collecting a set of positive and negative training images for the concept of interest, and therefore a binary classifier is trained to detect this concept. The process is repeated for each concept of the annotation vocabulary. These classifiers are then applied to all database images which are, accordingly, annotated with respect to the presence or absence of the concepts from the annotation vocabulary.

Other approaches have attempted to address the problem more generally by using unsupervised learning [START_REF] Barnard | Learning the semantics of words and pictures[END_REF], Duygulu et al., 2002, Blei and Jordan, 2003, Lavrenko et al., 2003]. Usually, these approaches employ probabilistic models to explain the co-occurrence between image features and semantic labels. The basic idea of these approaches is to introduce a set of latent variables that encode hidden states of data, and where each state induces a joint distribution on the space of semantic labels and image visual descriptors (local features computed over image neighborhood) [START_REF] Torralba | Sharing visual features for multiclass and multiview object detection[END_REF]. During the training process, each image is assigned a set of labels from the annotation vocabulary, these images are thereafter segmented into a set of regions (using a block-based decomposition [START_REF] Feng | Multiple bernoulli relevance models for image and video annotation[END_REF], Carbonetto et al., 2004] or using image segmentation methods [START_REF] Barnard | Learning the semantics of words and pictures[END_REF], Duygulu et al., 2002, Blei and Jordan, 2003, Lavrenko et al., 2003]), and an unsupervised learning algorithm is performed on the whole database to estimate the joint density of semantic labels and visual features. Therefore, the annotation of previously unseen images is performed as follows. Firstly, visual feature vectors are extracted from these images. Then, these feature vectors are used to instantiate the joint probability model. Thereafter, the state variables are marginalized, and finally the set of labels that maximize the joint density of concepts and visual appearance are assigned to the input image [START_REF] Carneiro | Supervised learning of semantic classes for image annotation and retrieval[END_REF].

In the following, we propose a brief overview of the different categories of image classification according to the used machine learning paradigm (i.e. supervised, unsupervised or semi-supervised). We refer to the notations of Section 2.2.

Chapter 2. State of the Art on Image Annotation

Supervised Learning

The formulation of automatic image annotation as a supervised learning problem has been proposed in [Carneiro andVasconcelos, 2005, Carneiro et al., 2007]. It consists firstly in performing a training phase, where N =| C | distinct classifiers are trained, each to detect the presence/absence of a given concept c i ∈ C. Let x v k be any visual representation of an image i k (a visual features vector), Y i a random variable such that Y i = 1 if i k is annotated with the concept c i , otherwise Y i = 0, X a random vector of visual features. The training of a classifier classif ier λ is performed as follows:

i) all images in the training dataset annotated with c λ are collected as positive samples, remaining images from the training dataset are considered as negative samples, and ii) some density estimation algorithm is trained to estimate P X|Y i (x v k |j), i.e. the conditional density that the feature vector x v k of the input image i k could be associated/or-not to the the semantic class c i (where j ∈ {0, 1}).

According to the statistical decision theory [START_REF] Duda | Pattern Classification[END_REF], the decision function to annotate an image i k with a concept c i could be easily computed as:

P X|Y i (x v k |1)P Y i (1) ≥ P X|Y i (x v k |0)P Y i (0) (2.1)
Equation 2.1 produces, for a given input image i k , an output P which consists of a set of candidate concepts c i ∈ C and their confidence values

α i = P Y i |X (1|x v k ), i.e. P : (c 0 , α 0 ), (c 1 , α 1 ), • • • (c m , α m ) , 0 ≤ m ≤ N .
For instance, a simple way to perform image annotation using supervised learning techniques is to train a set of binary classifiers, one for each concept from the annotation vocabulary [Tong and[START_REF] Goh | SVM binary classifier ensembles for image classification[END_REF]. Other approaches have also applied this method in order to annotate images in restricted domains, to distinguish cities from landscapes [START_REF] Vailaya | On image classification: City vs. landscape[END_REF], to detect indoor scenes from the outdoor ones [START_REF] Szummer | Indoor-Outdoor image classification[END_REF], and so on. [START_REF] Smith | Multimedia semantic indexing using model vectors[END_REF] presented a two-step Discriminative Model Fusion (DMF) approach to discover the implicit/indirect relationships between concepts by constructing model vectors based on detection scores of individual classifiers. Support vector machines are then trained to refine the detection results of the individual classifiers.

[ [START_REF] Li | Automatic linguistic indexing of pictures by a statistical modeling approach. Pattern Analysis and Machine Intelligence[END_REF]] used a statistical modeling approach to convert images into keywords. They considered the images of any given concept as instances of a stochastic process that characterizes this concept. Thus, they computed the extent of association between an image and the textual description of a concept as the likelihood of the occurrence of the image based on the characterizing stochastic process. [START_REF] Boutell | Learning multi-label scene classification[END_REF] applied multi-label learning techniques for scene classification. They decomposed the multiclass learning problem into several independent binary classification tasks. They also provided many labeling criteria to predict a set of labels, for a given test image, based on the outputs of the different binary classifiers.

Chapter 2. State of the Art on Image Annotation

Other approaches have proposed to use Multiple-Instance Learning in order to perform image annotation [Chen andWang, 2004, Carneiro et al., 2007]. In Multiple-Instance Learning (MIL) approaches, each image is treated as a bag of instances, and the goal is to maximize the diverse density or the soft margins between the negative and the positive samples of a given concept. For example, [START_REF] Carneiro | Supervised learning of semantic classes for image annotation and retrieval[END_REF] proposed a supervised multiclass labeling (SML) method, in which images are represented as bags of localized feature vectors, and a mixture hierarchies is built to learn the correspondence between images and their labels. The mixture hierarchy was composed of one mixture density estimated for each image and the mixture density estimated on all images annotated with a same given concept. [START_REF] Chen | Image categorization by learning and reasoning with regions[END_REF] proposed a MIL framework named Diverse-Density-SVM (DD-SVM). DD-SVM learns first a collection of instance prototypes according to a Diverse Density (DD) function. The training is performed in a feature space constructed from a mapping defined by the local maximizers and minimizers of the DD function. Through the feature mapping, DD-SVM essentially converts MIL to a standard supervised learning problem.

Unsupervised Learning

Unsupervised learning approaches for image annotation [START_REF] Barnard | Learning the semantics of words and pictures[END_REF], Duygulu et al., 2002, Jeon et al., 2003, Blei and Jordan, 2003, Lavrenko et al., 2003] aimed at finding hidden structure in unlabeled images. These structures are represented by a set of states which define each a joint distribution for semantic labels and image features [START_REF] Carneiro | Supervised learning of semantic classes for image annotation and retrieval[END_REF]. Three categories of approaches exist in the literature, which differ according to the definition of the states of the hidden variable:

• associating a state to each image in the database [START_REF] Lavrenko | A model for learning the semantics of pictures[END_REF],

• associating a state to each image cluster [Barnard andForsyth, 2001, Duygulu et al., 2002],

• or associating a state to a topic3 [Blei andJordan, 2003, Monay and[START_REF] Monay | [END_REF].

The form of the overall model is as follows :

P X|W (x v k |c i ) = S l=1 P X|L (x v k |l)P W |L (c i |l) (2.2)
where x v k is a visual representation of the image i k (a visual features vector), c i ∈ C is a concept from the annotation vocabulary, X a random vector of visual features, W is a random variable that takes value in {1, . . . , N } such that W = i if and only if X is a sample of c i , S is the number of possible states of L.

Equation 2.2 illustrates a mixture model, and therefore learning is usually based on the Expectation-Maximization (EM) algorithm [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF].

Chapter 2. State of the Art on Image Annotation

For more information, the readers are suggested to refer to [START_REF] Duda | Pattern Classification[END_REF], Carneiro et al., 2007].

[ [START_REF] Mori | Image-to-word transformation based on dividing and vector quantizing images with words[END_REF] proposed a co-occurrence model based on the co-occurrence of words and image regions created using a regular grid. [START_REF] Duygulu | Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary[END_REF] proposed a Machine Translation model (MT) to translate image blobs into concepts. First, the image is segmented into regions using a segmentation algorithm. Features are then computed on each image region, and blobs are generated by clustering the image features for these regions across all images. Therefore, each image is generated by using a collection of these blobs. Finally, the proposed translation model applies one of the classical statistical machine translation models to translate from the set of keywords of an image to the set of blobs forming the image. [START_REF] Jeon | Automatic image annotation and retrieval using cross-media relevance models[END_REF] proposed a Cross-Media Relevance Model (CMRM), which considers that concepts and blobs are conditionally independent in a given an image. [START_REF] Blei | Modeling annotated data[END_REF] extended the Latent Dirichlet Allocation (LDA) model to propose a Correlation LDA model which relates words and images. In their approach, they used a Dirichlet distribution to generate a mixture of latent factors, which is then used to generate words and regions. Expectation-Maximization is again used to estimate this model. [START_REF] Lavrenko | A model for learning the semantics of pictures[END_REF] proposed a model called Continuous-space Relevance Model (CRM), which is closely related to the models proposed by [Blei andJordan, 2003, Jeon et al., 2003], but it brings some improvements. CRM makes no assumptions about the topological structure, it directly models continuous features, i.e. does not rely on clustering, and consequently does not suffer from the granularity issues.

Some other approaches proposed to use Latent Dirichlet Allocation (LDA) and graphical models to learn a joint distribution model for a set of keywords and image regions [START_REF] Barnard | Learning the semantics of words and pictures[END_REF], Barnard et al., 2003, Feng and Lapata, 2010]. These approaches assume that a variable (representing the hidden state) exists in the data generative process, and which links the concepts and the visual feature vectors through conditional relationships. Indeed, several variations of Latent Dirichlet Allocation (LDA) based mixture models are based on this assumption. Therefore, in these approaches, images are represented as a collection of region-based image features, and are modeled by Gaussian distributions, while concepts are modeled with multinomial distributions. In [Barnard andForsyth, 2001, Barnard et al., 2003], several methods to model the joint distribution of words and blobs are discussed. Subsequent to the learning of joint distributions, image annotation in these approaches is regarded as a problem of likelihood estimation between the blobs and the words. However, the accuracy of these models is sensitive to the quality of image segmentation. In the same context, [Monay andGatica-Perez, 2003, Monay and[START_REF] Monay | [END_REF] proposed to use Latent Semantic Analysis (LSA) and Probabilistic Latent Semantic Analysis (PLSA) for image annotation.

Semi-Supervised Learning

Some recent work for image annotation have proposed to use semi-supervised learning to draw benefits from both, labeled and unlabeled (or weakly labeled) images. Indeed, due to the lack of available labeled training data, semi-supervised learning is a good Chapter 2. State of the Art on Image Annotation alternative for automatically annotating large-scale image databases with semantic concepts. In semi-supervised learning, a number of labeled examples are usually required for training an initial weakly useful predictor which is in turn used for exploiting the unlabeled examples [START_REF] Guillaumin | Multimodal semi-supervised learning for image classification[END_REF]. Therefore, these approaches try to tackle the image annotation problem by incorporating the information gathered from labeled images in a large amount of unlabeled data. [Zhu, 2006] proposed a good introduction to Semi-Supervised Learning (SSL).

Current semi-supervised learning based approaches can be categorized into three main paradigms.

• In the first, a generative model, such as Naive Bayes classifiers or a mixture of Gaussians, is used to model the joint probability distribution over observations and label sequences. The EM algorithm is then performed in order to model the label estimation or the parameter estimation process [START_REF] Guillaumin | Multimodal semi-supervised learning for image classification[END_REF].

• The second paradigm deals with the regularization of the learning process using the unlabeled data [START_REF] Kück | A constrained semi-supervised learning approach to data association[END_REF], Fergus et al., 2009, Wang et al., 2006].

For instance, some approaches use a graph where the nodes are set as the image visual descriptors and the edges are used to encode the similarity between the image samples. Therefore, the label smoothness can be enforced over the graph as a regularization term. For example, [START_REF] Fergus | Semi-supervised learning in gigantic image collections[END_REF] specified the label prediction function using smooth eigenvectors of the graph Laplacian which are calculated by a numerical method.

• The third paradigm, called co-training or also multiview learning, works under a two-view4 setting. It assumes that features can be split into two sets and that the two sets are conditionally independent given the class. This paradigm requires training two separate classifiers with the labeled data, on the two subfeature sets respectively. Then, the unlabeled data is classified by each classifier, which must agree on the much larger unlabeled data as well as the labeled one.

Each classifier is also able to 'teach' the other one with the unlabeled examples using the most confident prediction. For example, [START_REF] Zhou | Semi-supervised learning with very few labeled training examples[END_REF] proposed to take advantages of the correlations between the views using canonical component analysis. Their proposed method can perform semi-supervised learning with only one labeled training example.

Discussion

In order to compare the previously defined categories of image classification approaches, we propose in Table 2.1 a list of advantages and limitations with respect to each category. A more detailed comparison of these approaches is provided in Section 2.6. As one can deduce from this table, none of these categories is optimal in general terms, but each can be suitable with respect to a given image annotation problem. Given a specific image annotation task, the criteria for choosing between these categories can be summarized as follows:

• Availability of a sufficient amount of images for training the classifiers.

• The size of the annotation vocabulary (scalability).

• The need for a natural ranking of keywords.

• The required level of precision for the image annotation task.

Image Classification Based Approaches

Approaches Pros Cons

Supervised Learning

Are more intuitive to implement.

Allow designing features and tuning the learning algorithm for each classification task.

✗ Do not scale well with a large vocabulary (require training separated classifier). ✗ Do not allow natural ranking of keywords since the probabilities obtained from the different classifiers are not comparable.

Unsupervised Learning

Are more scalable in terms of training (when image database is large and/or concept number is high).

Produce a natural ranking of keywords.

✗ Do not guarantee that the semantic annotations are optimal under a recognition or retrieval sense (since keywords are not explicitly treated as classes).

Semi-Supervised Learning

Require few labeled data to train efficient classifiers.

Draw benefits from weakly labeled images.

✗ Scale polynomially with the number of images. ✗ Are impractical for use on gigantic collections with hundreds of millions of images and thousands of classes [START_REF] Rohrbach | What helps? where? and why? semantic relatedness for knowledge transfer[END_REF].

Hierarchical Image Classification

Many recent approaches have proposed to use hierarchical structures in order to address the scalability problem of automatic image annotation. In this dissertation we have also addressed this problem, i.e. the use of semantic hierarchies for image annotation and hierarchical image classification. Therefore, in order to put our contributions into perspective and to compare to existing approaches, we provide a detailed description of the hierarchical image classification approaches in Section 5.2. Moreover, we believe that these hierarchies are bearers of knowledge, either, visual or conceptual, and consequently we provide a discussion on the used hierarchy structures Chapter 2. State of the Art on Image Annotation for image annotation in Section 3.2. In the following, we only introduce the formal definition of hierarchical image classification.

From a formal point of view, the hierarchical image classification problem can be defined as follows.

Given:

• DB, a training image database consisting of a set of pairs image/textual annotation , i.e.

DB = {[i 1 , A 1 ], [i 2 , A 2 ], • • • , [i L , A L ]}
, where:

-

I = i 1 , i 2 , • • • , i L is the set of all images in DB,
-L is the number of images in the database.

-

C = c 1 , c 2 , • • • , c N is the annotation vocabulary used for annotating im- ages in I,
-N is the size of the annotation vocabulary.

-Each textual annotation A i consists of a set of concepts {c j ∈ C, j = 1..n i i } associated with a given image i i ∈ DB.

• C ′ = c ′ 1 , c ′ 2 , • • • , c ′ N ′ a set of visual classes 5
(or concepts, depending on the nature of the hierarchy) that link all the concepts of C in a hierarchical structure.

• We define:

-C as a set of classes (or concepts) such that C ⊆ C and C ′ ⊂ C , -X a random vector of visual features (i.e. X = {x v k | ∀k ∈ [1, L] , x v k is a visual representation of the image i k } ),
-(C , ⊑) a class hierarchy, where ⊑ is a partial order representing the subsumption relationship, i.e. (∀c i , c j ∈ C :

c i ⊑ c j ⇐⇒ c i is subsumed by c j ), -T a set of examples (x v k , S k ), with x v k ∈ X and S k ⊆ C such that c i ∈ S k ⇒ ∀c i ⊑ c j : c j ∈ S k .
Objective: Find an objective function f : X → C such that:

f (x v k ) = argmax c i ∈C   c j ∈S k P X|Y i (x v k |c j )   (2.3)
Therefore, the aim of hierarchical image classification is to find an objective function (or a decision function) f (x v k ) that maximizes the probability of association between visual representation of images and the concepts from the annotation vocabulary (c j ∈ C), while taking into account the local probabilities throughout a given path in the hierarchy. These local probabilities correspond to the belonging likelihood of the visual representation of a given image to the intermediate classes (or concepts) of the hierarchy. Thus, hierarchical image classification introduces a dependency between the concept nodes of a given path in the hierarchy in order to achieve a relevant decision about the belonging of images to the leaf concepts. For more information, see Section 5.5.

Discussion

Current approaches for automatic image annotation have participated to reduce partially the semantic gap problem by providing a set of methods allowing to link the visual content of images to semantic concepts. Within this context, many approaches based on machine learning techniques have been proposed in order to model the correlation between image features and the concepts. Indeed, automatic image annotation has been considered in the last decade as a multi-class classification problem. However, these approaches, even if they adequately describe the visual content of images, are often limited to detect perceptual manifestations of semantics, and then are unable to model the image semantics as it is perceived by humans (cf. Section 2.3). They also have many limitations when dealing with broad content image databases [START_REF] Liu | A survey of content-based image retrieval with high-level semantics[END_REF], i.e. the obtained performances vary significantly according to the considered concept number and the targeted image datasets as well [START_REF] Hauptmann | How many high-level concepts will fill the semantic gap in news video retrieval?[END_REF], Deng et al., 2010]. This variability may be explained by the huge intra-concept variability and the wide inter-concept similarities on their visual properties that often lead to conflicting annotations [START_REF]Mining multilevel image semantics via hierarchical classification[END_REF]. Thus, it is clear that there is a lack of coincidence between the high-level semantic concepts and the low-level features of images, and that the image semantics is not always correlated with its visual appearance.

Consequently, and always in the quest for models that could help to map successfully low-level features into high-level semantic concepts, some approaches proposed to use knowledge-driven frameworks for image annotation. These approaches rely on the use of (explicit or implicit) contextual knowledge in order to improve the image annotation accuracy. Two categories of approaches can be distinguished:

1. using information fusion stemming from multiple sources, and 2. using explicit semantic structures, such as semantic hierarchies and ontologies, for modeling knowledge about image context.

This knowledge is thereafter injected into the process of image analysis/annotation in order to achieve a relevant decision about the image content. We refer to these approaches as Semantic Image Annotation approaches. In the next section we introduce these approaches and we review some of the relevant work.

Chapter 2. State of the Art on Image Annotation

Semantic Image Annotation: Towards Knowledge -Driven Approaches for Image Annotation

Objects in the real world are always seen embedded in a specific context, and the representation of this context is essential for the analysis and the understanding of images. Contextual knowledge may stem from multiple sources of information, including knowledge about the expected identity, size, position and relative depth of an object within a scene [START_REF] Gronau | Integrated contextual representation for objects' identities and their locations[END_REF]. For instance, topological knowledge can provide information about which objects are most likely to appear within a specific visual setting, e.g. an office typically contains a desk, a phone, and a computer, but it is unlikely that it contains a bed. Spatial information can also provide information about which locations within a visual setting are most likely to contain objects, e.g. in a beach scene, the sky is usually placed at the top, while the sea is below. Given a specific context, this kind of knowledge can help reasoning on data to improve image annotation [Hudelot et al., 2008, Neumann andMöller, 2008].

Definition 7 (Contextual information/knowledge). By contextual information, we mean the collection of relevant conditions and surrounding influences that make a situation unique and comprehensible. While contextual knowledge is the information, and/or skills that have particular meaning because of the conditions that form part of their description.

Consequently, it is of prime interest to make efficient use of contextual knowledge in order to narrow the semantic gap, and to improve the accuracy of image annotation. In the following, we introduce semantic image annotation approaches which make use of contextual knowledge, either implicitly or explicitly. Firstly, we introduce the set of approaches for image annotation using information fusion as a tentative to (implicitly) capture the image semantics. Thereafter, we introduce the second category of approaches which use explicit semantic structures, such as semantic hierarchies and ontologies, in order to narrow the semantic gap.

Information Fusion for Image Annotation

The first notable attempt for using contextual information (perceptual context) for image annotation was proposed by [START_REF] Lavrenko | A model for learning the semantics of pictures[END_REF]. The authors proposed a statistical generative model which looks at the probability of associating words with image regions. They used the surrounding visual context by computing a joint probability of image features over different regions in an image using a training set. This joint probability is thereafter used to annotate and retrieve images. Thus in this model, the association of different regions provides context while the association of words with image regions provides meaning.

Recent work in computer vision have also stressed the importance of contextual information for improving object recognition in real world images [Oliva andTorralba, 2007, Galleguillos and[START_REF] Galleguillos | [END_REF]. These approaches are based on the Biederman's semantic relations [Biederman, 1972] to achieve robust object categorization in real world scenes. These semantic relations, called contextual features, can be classified into three: 1. Semantic context, which can be defined as the likelihood of finding an object in some scenes but not others [START_REF] Rabinovich | Objects in context[END_REF], Galleguillos et al., 2008, Divvala et al., 2009] 2. Spatial context, which can be defined as the likelihood of finding an object in some spatial locations with respect to other objects in the scene [START_REF] Russell | Object recognition by scene alignment[END_REF], Heitz and Koller, 2008, Galleguillos et al., 2010] 3. Scale context, which is a contextual relation based on the scales of an object with respect to others [Torralba, 2003, Torralba et al., 2010, Gould et al., 2009] In all these approaches, the used contextual information is about image features, i.e. contextual features, and is incorporated at the image processing level. Therefore, these approaches do not fall within the scope of our research, and we will not discuss further on them. A comprehensive survey about this category of approaches can be found in [Galleguillos and Belongie, 2010].

Subsequently, several approaches have proposed to combine information collected from the different image modalities in order to improve the image annotation accuracy. These approaches use a set of techniques, called semantic combination, in order to efficiently fuse these information (or modalities). Merged information usually stems from a couple (or more) of multimedia modalities, i.e. fusion of visual and textual information [Tollari, 2006, Clinchant et al., 2011], fusion of textual information and user tags [START_REF] Guillaumin | Multimodal semi-supervised learning for image classification[END_REF], Rohrbach et al., 2010, Znaidia et al., 2012], fusion of multiple images [START_REF] Tommasi | Discriminative cue integration for medical image annotation[END_REF], etc. These approaches, called cross-media fusion approaches, are based on the following paradigms:

• Exploitation of the diversity between the different information sources.

• Exploitation of the dependency between the multimedia modalities.

• Combination of multiple complementary information sources and generalizing over them.

• Cooperative combination of features in order to get a more precise representation of the world.

As introduced by [START_REF] Clinchant | Semantic combination of textual and visual information in multimedia retrieval[END_REF], three types of information fusion techniques have been proposed: 1. Early fusion: which consists in representing the multimedia objects in a multimodal feature space (e.g. concatenation of features stemming from images and textual information) [START_REF] Lavrenko | A model for learning the semantics of pictures[END_REF], Tollari, 2006, Rasiwasia et al., 2010].

Chapter 2. State of the Art on Image Annotation 3. Transmedia fusion: which consists in a diffusion processes that act as a transmedia pseudo-relevance mechanism [START_REF] Jeon | Automatic image annotation and retrieval using cross-media relevance models[END_REF], Bruno et al., 2008, Ah-Pine et al., 2009].

Although, [START_REF] Maillot | Inter-media pseudo-relevance feedback application to imageclef 2006 photo retrieval[END_REF], Müller et al., 2010] have reported that the late fusion and the transmedia fusion techniques are performing better than the early fusion one, there is no fusion strategy which is optimal in a general way. The fusion strategy has to be selected according to the targeted task and the data structure [START_REF] Clinchant | Semantic combination of textual and visual information in multimedia retrieval[END_REF].

Discussion

The image annotation approaches based on information fusion are motivated by the use of the correlations between the different multimedia modalities in order to reduce the semantic gap problem. Indeed, the fusion of information stemming from multiple sources can reduce considerably the uncertainty of image annotation, and as a consequence, significant improvements on the accuracy of image annotation have been reported by these approaches. However, these approaches do not necessarily provide a reliable solution to solve the semantic gap problem in its entirety. Actually, these approaches do not allow establishing explicit links between image features and image semantics. Therefore, they only use implicit inference mechanisms for improving the image annotation results and remain closely sensitive to machine learning accuracy. The use of explicit semantic structures, such as semantic hierarchies and ontologies, seems then to be essential to tackle the semantic gap problem in an effective way.

Ontology-Driven Approaches for Image Annotation

An important issue to solve the semantic gap problem is to make use of explicit and formal methods to represent contextual knowledge. This will help taking into account general and specific context of the image, and allow reasoning in order to improve the image annotation. [START_REF] Kompatsiaris | Semantic Multimedia and Ontologies: Theory and Applications[END_REF] underlines that among the possible representations of knowledge in the multimedia domain, ontologies are the most useful, and have considerable advantages. Indeed, ontologies provide a formal framework that may contain explicit semantic definitions, which can be directly processed by a machine, and allow at the same time to derive implicit knowledge by automatic inference.

Ontology and Multimedia Ontologies

The concept of Ontology emerged in the early 1990's in the artificial intelligence and the knowledge engineering communities. The term Ontology was borrowed from philosophy where it means a theory about the nature of being, the study of the kinds of things that exist. Nowadays, ontologies have become the new standard for knowledge representation [START_REF] Horrocks | From SHIQ and RDF to OWL: the making of a Web ontology language[END_REF], Baader, 2011].

Ontologies are a formal, explicit specification of a shared conceptualization [Gruber, 1995]. "Formal" reflects that ontology is machine-readable and allows reasoning about its content from the human and the machine side. "Explicit" means that the type of concepts used, and the constraints on their use are explicitly defined. "Shared" refers to the common knowledge embodied in ontology. "Conceptualization" refers to the model obtained by abstracting some phenomena existing in the real world by identifying the relevant concepts of those phenomena. Thus, ontologies allow capturing the relevant knowledge of a domain, provide a common understanding of this domain knowledge, determine acknowledged vocabulary of this domain, and give the explicit definition of the vocabulary(terms) and the relations between these vocabularies in formal models at different levels [START_REF] Ren | Semantic-based image retrieval using fuzzy domain ontology[END_REF].

Recent advances in ontological engineering have motivated several work in the field of image retrieval. As a result, a considerable number of multimedia ontologies have been proposed in order to define standards for the description of low-level multimedia content [Hunter, 2001, Simou et al., 2005, Arndt et al., 2007, Dasiopoulou et al., 2010]. Other domain ontologies have been proposed to allow the semantic interpretation of images and reasoning over the extracted descriptions [START_REF] Bagdanov | Semantic annotation and retrieval of video events using multimedia ontologies[END_REF], Neumann and Möller, 2008, Peraldi et al., 2007, Hudelot et al., 2008, Hudelot et al., 2010].

Usually, ontologies were used in the image retrieval field to target the following goals:

1. A unified description of low level features: where ontologies are used to provide standards of description of low-level features -e.g. [START_REF] Simou | A visual descriptor ontology for multimedia reasoning[END_REF], Dasiopoulou et al., 2010].

2. Visual description ontology: where ontologies are used to represent the different types of relations among image features such as edges, lines and regione.g. [Maillot andThonnat, 2008, Yao et al., 2010]. Typically the use of these ontologies comes during the image analysis process, and targets to optimize or to reason on this task.

3. Semantic mapping: ontologies are used to help the mapping between the visual level and the semantic level of images. For instance, the use of semantic hierarchies to reduce the semantic gap, e.g. [Fan et al., 2008a[START_REF]Mining multilevel image semantics via hierarchical classification[END_REF].

4. Knowledge description: ontologies are used to model the concepts (objects) and relations among them. Typically, these are all approaches that use reasoning on concepts or on contextual information, i.e. after the image analysis process. These approaches most often tackle the problem of image annotation and interpretation -e.g. [START_REF] Hollink | Adding spatial semantics to image annotations[END_REF], Simou et al., 2008, Hudelot et al., 2008, Dasiopoulou et al., 2009, Hudelot et al., 2010, Hamadi et al., 2012].

In a general way, the use of ontologies contributes to improve the image retrieval by incorporating contextual knowledge. The integrated knowledge can help in various spots of the image retrieval process, such as: i) image analysis, ii) mapping visual Chapter 2. State of the Art on Image Annotation features into semantic concepts, iii) assigning a meaning to tags, iv) disambiguation, and v) annotation enrichment. However, since the major challenge with the semantic gap is to provide effective tools that allow the mapping between low-level features and semantic concepts, semantic mapping is almost getting all the lights and it has been one of the most active issues. Indeed, while common methods for image annotation are limited to provide a latent correlation between semantic and visual spaces, ontology-driven approaches can make explicit this relationship. Consequently, ontologies provide an effective way to map low-level features into semantic concepts by building rules that supply semantic association between image features and concepts. This is achieved while maintaining a semantic structure to this process which will allow reasoning in order to check the consistency of this mapping. This domain is still in its infancy, but early results seem promising.

Moreover, despite a widely collaborative annotation effort, and the large amount of available goundtruth, the provided annotation vocabulary can never reach the richness of human-know vocabularies. Besides, different persons may use different terms to describe/search a same image. To address this problem, some ontologies [Fellbaum, 1998, Liu and Singh, 2004, Naphade et al., 2006] were proposed/used to structure the annotation vocabulary and the concepts that can be used by users to express their needs in a multimedia information retrieval system. The use of these ontologies allows then to make more precise and consistent the description of multimedia content. Ontologies allow disambiguating the various interpretation of a multimedia content by providing a semantic structure for the annotation vocabulary.

With respect to the image annotation field, explicit knowledge structures (or ontologies under their general form) have been introduced under three different shapes 6for modeling contextual knowledge, as: i) Heavy-Weight Ontologies, ii) Light-Weight Ontologies, and iii) formal Ontologies. Figure 2.9 illustrates these categories and their underlying methods. In the following, we review the use of these explicit semantic structures for image annotation, and we discuss their roles in narrowing the semantic gap.

Light-Weight Ontologies

Heavy-Weight Ontologies

Ontologies Semantic Hierarchies Taxonomies Lattices Semantic Networks Formal Ontologies Heavy-Weight Ontologies are fully described ontologies, including concept definitions and relations. These ontologies make intensive use of axioms to model knowledge about concepts and to restrict domain semantics. Heavy-Weight Ontologies, tailored to the image semantics understanding, have been used to attach meaning to the produced annotations and to help extracting, querying, analyzing and interpreting these annotations.

For example, in M-OntoMat-Annotizer [START_REF] Petridis | MOntoMat-Annotizer: image annotation linking ontologies and multimedia low-level features[END_REF], low level MPEG-7 visual descriptions are linked to conventional Semantic Web ontologies and annotations. M-OntoMat-Annotizer is used in order to construct an ontology that includes prototypical instances of high-level domain concepts together with a formal specification of corresponding visual descriptors. Thus, it formalizes the interrelationships of high-level and low-level multimedia concept descriptions, allowing therefore for new kinds of multimedia content analysis and reasoning.

[ [START_REF] Dong | Multi-ontology based multimedia annotation for domain-specific information retrieval[END_REF] proposed a multi-ontology based multimedia annotation model. They proposed to integrate a domain-independent multimedia ontology with multiple domain ontologies in an effort to provide multiple domain specific views of multimedia content. Thus, accessing multimedia content can be less subjective to users background knowledge and their need of information. [START_REF] Hare | Bridging the semantic gap in multimedia information retrieval: Top-down and bottom-up approaches[END_REF] suggested to use an ontology as an extra level in between the search query and keywords. So when performing a concept-based search, the search engine automatically performs inference to find all narrower concepts of the query concept. However, this method focuses on the query understanding and does not take into account image content. [START_REF] Mezaris | An ontology approach to object-based image retrieval[END_REF] proposed an image retrieval methodology where low-level features are extracted from image regions, and mapped automatically to intermediate level descriptors called 'object ontology', which is used for the definition of high-level concepts. Nevertheless, the suggested 'object ontology' is purely visual as it defines a simple vocabulary to describe perceptual manifestation of semantics (objects or regions).

Discussion

The aforementioned approaches are interesting, in particular, since they tried to narrow the semantic gap by using ontologies as a mid-level between image features and the semantic concepts in a tentative to make these relationships explicit. However, these approaches have not provided a formalism allowing reasoning over these explicit relationships, which make them impractical for improving the image annotation. Moreover, as stated previously in this chapter, there is no guarantee that there is a direct relationship between image features and the semantic concepts, but it rather seems that, often, this relationship is implicit and inferred from the context (of the image or of the observer).

It is therefore more appropriate to provide ontologies that model the image context, and to build tools allowing reasoning on this knowledge from both sides, the Chapter 2. State of the Art on Image Annotation visual appearance of concepts and their conceptual definition. This will certainly help narrowing the semantic gap in an effective manner.

Light-Weight Ontologies (LWO)

Ontologies have varying degrees of expressiveness. In this dissertation, we refer to ontologies with restricted expressiveness as Light-Weight Ontologies. These are partially described ontologies, like taxonomies, thesauri, semantic hierarchies, lattices and semantic networks. With respect to image retrieval field, many approaches have proposed to use Light-Weight Ontologies in order to perform image classification, or to provide multiple levels of abstraction image annotation [START_REF] Marszalek | Semantic hierarchies for visual object recognition[END_REF], Wu et al., 2008, Tousch et al., 2008, Fan et al., 2009, Martinet et al., 2011, Wu et al., 2012].

Definition 8 (Light-Weight Ontologies). A Light-Weight Ontology is a directed graph whose nodes represent concepts. The links between the nodes indicate associations (or untyped relationships) between the corresponding concepts. These associations express semantic nearness [START_REF] Reimer | Learning a lightweight ontology for semantic retrieval in patientcentered information systems[END_REF].

Light-Weight Ontologies were used under different shapes in order to improve the image annotation. For instance, concept hierarchies [START_REF] Naphade | Large-scale concept ontology for multimedia[END_REF], Marszalek and Schmid, 2007, Deng et al., 2009], visual taxonomies [START_REF] Yao | I2T: Image parsing to text description[END_REF], Fei-Fei and Perona, 2005, Griffin and Perona, 2008, Bart et al., 2008], and semantic hierarchies [Fan et al., 2008a, Li et al., 2010] have been used for image annotation. These hierarchies7 are typically ontologies limited to the subsumption (is-a) relationship, i.e. they are a collection of classes ordered by the transitive closure of explicitly declared subclass or subtype relations. Being A a subclass of B, captures the fact that the state and the behavior of the elements of A are coherent with 'the intended meaning' of B (or 'the visual features' of B, depending on the nature of the hierarchy), while disregarding the additional features and functionalities that characterize the subclass [START_REF] Logozzo | Semantic hierarchy refactoring by abstract interpretation[END_REF]. One of our contributions in this dissertation deals with the building of semantic hierarchies dedicated to image annotation. Therefore, we discuss in more details the use of these hierarchies for image annotation in Section 3.2.

Other approaches have proposed to use semantic networks, built upon both conceptual and visual information, for image annotation [START_REF] Fan | Integrating visual and semantic contexts for topic network generation and word sense disambiguation[END_REF], Wu et al., 2008, Wu et al., 2012, Tousch et al., 2012]. 'Flickr distance' is proposed in [START_REF] Wu | Flickr distance[END_REF], which is a new semantic similarity measure between the concepts in the visual domain. A Visual Concept NETwork (VCNet) based on the Flickr distance is also proposed in [START_REF] Wu | Flickr distance[END_REF], Wu et al., 2012]. [START_REF] Fan | Integrating visual and semantic contexts for topic network generation and word sense disambiguation[END_REF] proposed an algorithm to integrate the visual similarity and the contextual similarity for a topic network generation. An image parsing to text description (I2T ) framework is proposed in [START_REF] Yao | I2T: Image parsing to text description[END_REF], which generates text descriptions for images and videos. I2T is mainly based on an And-or Graph for visual knowledge representation. Semantic lattices have also been explored in order to improve the image annotation [START_REF] Belkhatir | Integrating perceptual signal features within a multi-facetted conceptual model for automatic image retrieval[END_REF], Tousch et al., 2008, Martinet et al., 2011]. For instance, [START_REF] Tousch | Semantic lattices for multiple annotation of images[END_REF] used a handmade semantic lattice describing the "Car" domain with the aim of classifying images according to a trade-off between semantic precision and accuracy. [START_REF] Martinet | A relational vector space model using an advanced weighting scheme for image retrieval[END_REF] proposed to merge the vector space model of information retrieval with the conceptual graph (which has a lattice structure) formalism in order to provide a framework for relational information retrieval dedicated to images.

Discussion

Light-Weight Ontologies have shown to be very useful to narrow the semantic gap. Indeed, the aforementioned approaches have reported significant improvement in the image annotation accuracy. However, besides few work that addressed the problem of building semantic structures dedicated to image annotation, a basic problem with most of recent approaches is that they used either a specification based on textual information extracted from Wordnet, or a specification based on image features. Consequently, while these semantic structures are useful to provide a meaningful structure (organization) for image concepts, this organization does not necessarily reflect image semantics, since these structures were not originally built for this specific task. This is one of the fundamental assumptions of this thesis work and we will argue in more detail on this problem throughout this dissertation.

Moreover, most of recent work have attempted to use the incorporated knowledge (about image concepts) in these semantic structures in an implicit manner. Nevertheless, it is more convenient to use the explicit knowledge provided by these structures, and therefore, it is of prime importance to focus on this issue in order to exploit the strong inter-concepts correlation, and also the subsumption relationship that can help reasoning on the proper decision-making for image annotation.

Formal Ontologies

While ontologies often play a passive taxonomic role, some approaches consider ontologies as an active inference framework for image annotation and interpretation. These approaches focus on the use of formal semantics by constructing precise mathematical models of image semantics, i.e. explicit and formal modeling of knowledge about image context and the concepts.

Given the high expressivity and the well-defined inference services coming with them, Description logics appear today as an excellent candidate to support ontology representation and ontological reasoning. Description logics (DL) are a family of formal language for representing knowledge. They provide facilities for implementing knowledge bases, reasoning about their content, and manipulating them [START_REF] Baader | The Description Logic Handbook: Theory, Implementation, and Applications[END_REF].

Definition 9 (Description logics). Description logics (DLs) are a family of knowledge representation formalisms that represent the knowledge of an application domain (the "world") by first defining the relevant concepts of the domain (its terminology), and then using these concepts to specify properties of objects and individuals occurring in the domain (the world description) [START_REF] Baader | The Description Logic Handbook: Theory, Implementation, and Applications[END_REF].

As illustrated in Figure 2.10, a DL knowledge base usually consists of three components, the TBox, the RBox and the ABox. The TBox and the RBox introduce the terminology, i.e. the vocabulary of an application domain, while the ABox contains assertions about named individuals in terms of this vocabulary. These notions are introduced in details in Section 4.4. Formal ontologies8 based approaches have successfully managed to exploit some interesting reasoning properties in the context of high-level image interpretation. These approaches make intensive use of contextual knowledge and inference rules with the aim of performing semantic image analysis, annotation and/or interpretation, ensuring thus the acquisition of interpretations that can match human cognition. In order to achieve these inference tasks, these approaches are based on sets of objects (or concepts) and relationships between them (called roles in DL). Axioms, which are a set of appropriate statements, are used to capture the conditions that need to be met by coherent and consistent states of the domain (interpretations). Image annotation/interpretation tasks are therefore formalized as:

TBox

1. Deduction: where the interpretation is an instantiation of formal knowledge consistent with evidence about the real-world domain [START_REF] Hotz | Scene interpretation as a configuration task[END_REF], Hartz and Neumann, 2007, Hudelot et al., 2008, Dasiopoulou et al., 2009], or as 2. Abduction: where the interpretation is an instantiation of formal knowledge which allows to deduce the evidence [Shanahan, 2005, Town, 2006, Peraldi et al., 2007, Atif et al., 2011, Atif et al., 2013].

Deductive and abductive reasoning were introduced as inference standards, where for deductive reasoning: if Σ is a logical theory and α a set of facts, through deduction is verified whether ϕ is logically entailed, that is whether Σ, α |= ϕ. For abductive reasoning: given Σ and ϕ, abduction consists in finding an 'explanation' α so that the entailment Σ, α |= ϕ is true. For example, let us say Σ is (visual or contextual) contextual knowledge regarding concept "Car", α is (visual or contextual) information extracted from an image containing a "Car" and ϕ is an instance of concept "Car", deduction is then, given Σ, α |= ϕ.

For instance, in [START_REF] Hudelot | Fuzzy spatial relation ontology for image interpretation[END_REF], Hudelot et al., 2010] an ontology of spatial relations is proposed to facilitate image interpretation. Indeed, [START_REF] Hollink | Adding spatial semantics to image annotations[END_REF], Maillot et al., 2004, Hudelot et al., 2005] have reported that spatial relationships among objects and regions have appeared to be crucial in the concept detection process. [START_REF] Mylonas | Using visual context and region semantics for high-level concept detection[END_REF] proposed an approach based on a visual thesaurus and visual context to improve concept detection. The authors introduce local (topological and unified) context in the analysis, to refine the confidence values of regions before taking decision. [Town, 2006] proposes an iterative process where low-level detections (induction) are compared with high-level models to derive new hypotheses (deduction). These can in turn guide the search for evidence to confirm or reject the hypotheses on the basis of expectations defined over the lower level features. In [START_REF] Simou | Image indexing and retrieval using expressive fuzzy description logics[END_REF], a knowledge-assisted analysis architecture is proposed to perform the refinement of an initial set of over-segmented regions. They also used a fuzzy reasoning engine for the extraction of additional implicit knowledge and the improvement of region-based classification by incorporating spatial relations and neighborhood information. A comprehensive survey on the use of Description Logics for image annotation/interpretation is proposed in [START_REF] Neumann | On scene interpretation with description logics[END_REF], Möller and Neumann, 2008, Dasiopoulou and Kompatsiaris, 2010].

In this dissertation, we propose a new approach for building a formal multimedia ontology dedicated to image annotation and we also propose an approach for image annotation using ontological reasoning. Therefore, in order to highlight our contributions, we provide a more detailed discussion on related work in Section 4.2 and in Section 6.2.

Discussion

So far, we have seen that many recent work have proposed to use ontologies with the aim of improving the automatic image annotation, and knowledge based models have also been proposed in a tentative to achieve a consistent representation of image semantics. Specifically, formal ontologies are of great interest for image annotation, as they provide a formal framework containing explicit semantic definitions of concepts and their relations. These ontologies allow modeling contextual knowledge in a formal way, which allow deriving implicit knowledge by automatic inference.

However, most of current approaches for image annotation have not used all the expressiveness, nor the reasoning ability provided by ontologies. Most of them have proposed to use ontologies in order to define standards for the description of low-level multimedia content [START_REF] Simou | A visual descriptor ontology for multimedia reasoning[END_REF], Dasiopoulou et al., 2010], or for Chapter 2. State of the Art on Image Annotation providing semantic structures for the annotation vocabulary [START_REF] Wei | Ontology-enriched semantic space for video search[END_REF], or as a hierarchical image classification frameworks [START_REF] Marszalek | Semantic hierarchies for visual object recognition[END_REF], Fan et al., 2008a, Deng et al., 2009], or also to define the semantic relationships between image concepts [START_REF] Hollink | Adding spatial semantics to image annotations[END_REF], Hudelot et al., 2008]. Nevertheless, for the purpose of reducing the semantic gap in an effective manner, ontologies should be considered as an active inference framework for reasoning about image annotation and interpretation. Although interesting attempts have recently emerged [START_REF] Dasiopoulou | Using Fuzzy DLs to Enhance Semantic Image Analysis[END_REF], Straccia, 2009, Hudelot et al., 2010], ontology-driven approaches for semantic image annotation are still in their infancy, and significant efforts have to be done before achieving effective systems for image annotation. In particular, the automatic building of such ontologies is rarely tackled.

General Discussion

As previously introduced, automatic image annotation is still an important challenge despite more than a decade of research. Indeed, many recent work have addressed this issue and have proposed new approaches with the aim of narrowing the semantic gap problem. In order to summarize the advantages and limitations of existing approaches for image annotation, we propose in Table 2.2 a comparison between the different families of approaches. As shown in this table, text-based approaches for image annotation provide a rich description of image content which may include the visual dimension, but also, subjective, spatial, temporal and social ones. However, text-based approaches are sensitive to the surrounding context which may be irrelevant with respect to image content. In particular, these approaches do not consider image content and therefore do not guarantee a consistent annotation, nor allow to process raw images which are widely spread nowadays (e.g. in photo sharing services).

As regards of automatic image annotation approaches, significant solutions have been proposed to reduce partially the semantic gap problem thanks to machine learning algorithm. These approaches have successfully managed to provide methods allowing to link the visual features of image to semantic concepts. However, it seems that the only use of machine learning algorithms is not sufficient to solve the image annotation problem. Indeed, these approaches face significant problems when dealing with large image databases (large in terms of image and concept number). Moreover, automatic image annotation approaches are limited to detect perceptual (visual) manifestations of semantics, i.e. they do not provide an explicit link between image features and concepts and they only consider concepts as classes (without any associated semantics).

Finally, semantic image annotation approaches have proposed to use either implicit knowledge (as the correlation between image modalities) or explicit knowledge (using knowledge models) about image context in order to achieve relevant decisions on the image annotation. Specifically, ontology-driven approaches for image annotation have shown to be appropriate for modeling image semantics as they provide explicit relationships between image concepts, between concepts and their meaning and sometimes between image features and concepts. These explicit relationships

Approaches Pros Cons

Text-Based Approaches for Image Annotation

Text-Based Approaches

May describe the visual content of images, and also subjective, spatial, temporal and social dimensions.

Description closer to the human one.

✗ Can only process images with surrounding context, ✗ Sensitive to the surrounding context, ✗ Sensitive to the subjectivity of human perception, ✗ Do not take into account image content (may be incoherent w.r.t image content).

Content-Based Approaches for Image Annotation Automatic Image Annotation Approaches

Allow processing raw images without any surrounding information.

Retrieval is achieved using semantic concepts.

These concepts are extracted automatically from low-level features.

Sometimes description is less subjective than textual approaches.

Allow to narrow the semantic gap.

✗ Sometimes sensitive to image segmentation.

✗ Suffer from the scalability problem.

✗ Do not allow semantic interpretation of images. ✗ Do not adapt to the user background.

Semantic Image Annotation Approaches

May describe the visual content of images, and also subjective, spatial, temporal and social dimensions.

Allow processing raw images without any surrounding information.

Retrieval is achieved using abstract and semantic concepts.

Description less subjective to human perception.

Allow reasoning on image annotation and interpretation.

Allow to narrow the semantic gap. Scale well with large databases.

✗ Difficult to implement. ✗ Not well explored. ✗ The reasoning power has not yet showed its efficiency. ✗ Ontological reasoning may be subject to the scalability problem. are of great interest, since if they are used correctly could help to formalize the link between images and their semantic, and consequently allow reasoning on image annotation in order to achieve a consistent decision-making. However, the ontology building and ontological reasoning may be subject to the scalability problem. In Table 2.3 and in Figure 2.11 we propose a comparison between the different categories of approaches for image annotation. The performance evaluation of each approach is quite subjective to our standpoint about existing approaches in the image annotation field, since these were not developed following experiments. However, this quantification is motivated by the results and the discussions reported in several significant work in our domain, and represents our synthesis of the state of the art. The Comparison criteria were the following:

• Semantic level: visual content vs. semantic description.

• Reasoning capabilities: implicit knowledge vs. explicit knowledge.

• Scalability problem according to image number.

• Scalability problem according to concept number. As one can see, semantic image annotation approaches appear to be promising to help tackling the semantic gap problem in an efficient manner. Indeed, the semantic level provided by these approaches, and specifically by formal ontologies, is rich enough to match user expectations in an image retrieval system. Formal ontologies allow also to dispose of good reasoning capabilities, which is useful to reduce the uncertainty problem introduced by machine learning algorithms and to reason about the consistency of image annotation. However, as shown in Table 2.3, ontology-driven approaches may suffer from the scalability problem. Indeed, in these knowledge models, the representation of each single real world object is split into many axioms about concepts and roles, leading to an overall design that is very difficult to apprehend [START_REF] Spaccapietra | On spatial ontologies[END_REF]. Consequently, an important issue in these approaches is to provide methods allowing to build ontologies dedicated to image annotation in an automatic manner.

According to our review of the state of the art, we present in the following our general standpoint about existing approaches for semantic image annotation and the main issues that motivated our contributions. These issues will be discussed deeply in the different chapters of this dissertation.

• Existing knowledge models used for image annotation are based either on a visual or textual specification (e.g. based on WordNet). However, these knowledge models, are they representative enough of image semantics? Would we not need a more elaborated specification (for example, by combining visual and textual specification) for modeling the image semantics?

• Ontology-driven approaches seem promising for narrowing the semantic gap. However, knowledge representation in these models is a hard task9 and faces a major scalability problem. Therefore, should we not rather focus on providing methods that allow building ontologies (HWO, LWO or formal ontologies) in automatic manner in order to scale with a large number of concepts?

• Currently, concept hierarchies (and respectively visual taxonomies) are the most widely explored for semantic image annotation. These, constrain the reasoning to the inheritance relationships. Do we not need to enrich these structures with other semantic relationships as composition relationships, spatial, topological, etc., in order to benefit from a stronger reasoning power on contextual knowledge?

Conclusion

In this chapter, we have proposed a comprehensive survey of the recent work for image annotation. Our aim was not to provide an exhaustive survey of the state of the art approaches, nor to state that an approach is better than the others, but to introduce the different categories of approaches for image annotation and to underline the benefits and limits of each of them. This chapter also has highlighted the importance of contextual knowledge and explicit semantic structures for the image annotation problem.

Indeed, the use of explicit semantic structures, such as semantic hierarchies and ontologies, seems to be essential to improve the image annotation and in order to narrow the semantic gap. In this dissertation, we are interested in semantic image annotation. These approaches aim at improving the image annotation by the use of images content and structured knowledge models about image semantics. This involves first to capture and to model suitable knowledge models about image context, and subsequently to build effective tools using these knowledge models in order to improve the image annotation process. The overall goal remains to narrow the semantic and sensory gaps using the available visual features of images and relevant knowledge sources, ultimately to satiate the user. Specifically, our concerns in this dissertation are as follows. Firstly, we address the problem of building explicit knowledge models dedicated to image annotation. Subsequently, we focus on the use of these knowledge models in order to produce a semantically consistent image annotation.

Introduction

In this chapter, we propose a new approach for automatically building semantic hierarchies dedicated to image annotation. Our approach is based on a new image-semantic measure, named "Semantico-Visual Relatedness of Concepts" (SVRC ), which allows estimating the semantic similarity between image concepts. The proposed measure incorporates visual, conceptual and contextual information in order to provide a measure which is meaningful and representative of image semantics. We also propose a new methodology based on the previously proposed measure SVRC and on a new heuristic, named TRUST-ME, to connect the concepts with higher relatedness till the building of the final semantic hierarchy. The built hierarchy explicitly encodes, a general to specific, concept relationships, and therefore provides a semantic structure to concepts which facilitates the semantic interpretation of images. An evaluation of the effectiveness of the produced semantic hierarchy is presented in Chapter 5.

The rest of this chapter is structured as follows. In Section 3.2 we present the motivations of our proposal. We review some existing approaches and we emphasize their limitations. A discussion is also presented, where we provide a definition to what is meant by a semantic hierarchy suitable for image annotation. In Section 3.3 we introduce the proposed measure for computing the semantic relatedness between image concepts. Section 3.4 introduces the proposed rules for the building of semantic hierarchies dedicated to image annotation. Section 3.5 presents an application of the proposed method on the Pascal VOC'2010 dataset and illustrates the obtained hierarchy. Finally, the chapter is concluded in Section 3.6.

Motivation for Using Semantic Hierarchies

As stated in Chapter 2, current approaches for automatic image annotation face significant problems to narrow the semantic gap. Indeed, these approaches allow to adequately describe the visual content of images but are unable to extract image semantics like humans do, i.e. they are limited to describe the perceptual manifestation of image semantics. Moreover, they are subject to the scalability problem when dealing with broad content image databases [START_REF] Liu | A survey of content-based image retrieval with high-level semantics[END_REF], Deng et al., 2010].

A new trend to overcome the aforementioned problems is to use explicit semantic structures, such as semantic hierarchies and ontologies. Specifically, semantic hierarchies have shown to be very useful to narrow the semantic gap [START_REF] Deng | What does classifying more than 10,000 image categories tell us?[END_REF]. In particular, they provide a hierarchical semantic structure for image concepts that can be used as a framework for image classification. They also can be used as a formal framework for reasoning about the consistency of image annotation, and thus for reducing the uncertainty introduced by machine learning algorithms.

Several approaches have been proposed to use, or also to build, semantic hierarchies in order to improve image annotation. As introduced in Chapter 2, these approaches can be mainly categorized in three:

1. Language-based hierarchies: based on textual information1,2 [START_REF] Wei | Ontology-enriched semantic space for video search[END_REF], Marszalek and Schmid, 2007, Torralba et al., 2008],

Visual hierarchies (called also visual taxonomies):

based on lowlevel image features [START_REF] Marszalek | Constructing category hierarchies for visual recognition[END_REF], Griffin and Perona, 2008, Sivic et al., 2008, Bart et al., 2008], and 3. Semantic hierarchies: based on both textual and visual features [START_REF] Li | Building and using a semantivisual image hierarchy[END_REF], Fan et al., 2007].

Language-Based Hierarchies

Language-based hierarchies are concept hierarchies built using only conceptual information, i.e. based on a similarity measure between concepts calculated from textual data, or from a conceptual specification. These include commonsense ontologies, such as WordNet and Wikipedia, or any other concept hierarchy which does not take account of the visual information of images. Currently, many existing approaches are based on WordNet [Fellbaum, 1998] for extracting the concept hierarchy that will serve thereafter for image annotation. For instance, [START_REF] Marszalek | Semantic hierarchies for visual object recognition[END_REF] proposed a semantic hierarchy classifier based on WordNet. Their hierarchy is built by extracting the relevant subgraph of WordNet that links all the concepts of the annotation vocabulary. The structure of this hierarchy is then used to train a set of hierarchical classifiers in order to perform image annotation. [START_REF] Torralba | 80 million tiny images: A large data set for nonparametric object and scene recognition[END_REF] proposed a classification scheme using the Wordnet tree. Given a query image, they look for neighbors using some visual similarity measure. Then, each neighbor votes for its branch within the Wordnet tree. Votes are accumulated, from the entire sibling set, across a range of semantic levels. Classification is therefore performed by assigning to the query image the label with the most votes at the desired height within the tree, where the number of votes is acting as a measure of confidence in the decision. ImageNet is proposed in [START_REF] Deng | ImageNet: a large-scale hierarchical image database[END_REF], which is a large-scale ontology of images built upon the backbone of WordNet. Im-ageNet aims at populating the 80 000 synsets3 of WordNet with an average of 500-1000 manually selected images. [START_REF] Deng | What does classifying more than 10,000 image categories tell us?[END_REF] proposed a measure of similarity between categories based on WordNet, and claimed that the obtained results indicate a correlation between the structure of the concept hierarchy (of WordNet) and visual confusion between the categories. [Deng et al., 2011a] proposed a visual similarity function defined on semantic attributes of trained images using the WordNet hierarchy. The similarity between two concepts was defined according to the lowest common ancestor of these concepts. The reported results have shown that adding hierarchical knowledge has significantly increased retrieval performance.

Some other approaches have proposed to use the LSCOM4 ontology, or to automatically build a binary concept hierarchy using an agglomerative algorithm on concept vectors [START_REF] Wei | Ontology-enriched semantic space for video search[END_REF]. For instance, [START_REF] Naphade | Large-scale concept ontology for multimedia[END_REF] proposed LSCOM, a multimedia ontology which aims at designing a taxonomy with a coverage of around a 1000 concepts for broadcast news video retrieval. An Ontology-enriched Semantic Space (OSS) was built in [START_REF] Wei | Ontology-enriched semantic space for video search[END_REF] to ensure globally consistent comparison of semantic similarities.

While these hierarchies are useful to provide a meaningful structure (organization) for concepts, they ignore the visual information which is an important part of image semantics. Furthermore, this kind of hierarchies are usually deep, and may contain many intermediate concepts which are not necessarily relevant in the image domain. For instance, it is may be hard to discriminate, at the perceptual level, the images of the following classes: ruminant vs. non-ruminant animals, or carnivore vs. herbivore animals, etc. Moreover, as it will be shown in our experiments, the hierarchy depth has a significant impact on the accuracy of the hierarchical classifiers -cf. Section 5.6. Figure 3.1 illustrates a hierarchy of concepts that we built by the extraction of the relevant subgraph of WordNet which links the 20 concepts of the Pascal VOC'2010 dataset. The building of the hierarchy is achieved by connecting first the most related concepts according to the shortest path in WordNet, then by connecting their hypernyms until reaching the root of WordNet. A disambiguation step is performed previously in order to retrieve the good synset (sense) that corresponds to each word (this method is quite similar to [START_REF] Torralba | 80 million tiny images: A large data set for nonparametric object and scene recognition[END_REF], with the difference that they took the most common meaning of each word to transform the initial graph-structured relationships between words into a tree-structured one).

Chapter 3. Building Semantic Hierarchies Faithful to Image Semantics

! " # $ % & ! & ' & ( & ) ) ' & ( * # ) # )
)

) & & && + ( + ! + * & '

Visual Hierarchies

Some other approaches have proposed to use visual information in order to build visual hierarchies (called also visual taxonomies) for the end-task of image classification [START_REF] Fei-Fei | A bayesian hierarchical model for learning natural scene categories[END_REF], Sivic et al., 2008, Marszalek and Schmid, 2008, Bart et al., 2008, Gao and Koller, 2011]. For instance, [START_REF] Deng | Hierarchical semantic indexing for large scale image retrieval[END_REF]Perona, 2005, Griffin and[START_REF] Griffin | [END_REF] proposed to automatically build a visual taxonomy for image classification, which consists in a hierarchy of classifiers built directly from the confusion matrix. The authors suggested using this taxonomy to increase the classification speed instead of performing a multi-class classification on all the categories. [START_REF] Sivic | Unsupervised discovery of visual object class hierarchies[END_REF] proposed to group visual objects using a multilayer hierarchy tree that is based on common visual elements. The clustering is achieved by adapting to the visual domain, the generative Hierarchical Latent Dirichlet Allocation (hLDA) model [START_REF] Blei | Hierarchical topic models and the nested chinese restaurant process[END_REF]. [START_REF] Bart | Unsupervised learning of visual taxonomies[END_REF] proposed a Bayesian method to organize a collection of images into a tree shaped hierarchy. [Marszalek andSchmid, 2008, Gao and[START_REF] Gao | [END_REF]] adopted a relaxed hierar-chy structure, where a set of binary classifiers are organized in a tree or DAG (Directed Acyclic Graph) structure. [START_REF] Pujol | Discriminant ECOC: a heuristic method for application dependent design of error correcting output codes[END_REF]] used a greedy sequential forward floating search algorithm to find two subsets of classes with the maximum mutual information between the feature distributions.

Figure 3.2: Obtained result by the method of [START_REF] Sivic | Unsupervised discovery of visual object class hierarchies[END_REF] for the building of visual object class hierarchies. The illustrated hierarchy is built on a subset of the LabelMe dataset [START_REF] Russell | LabelMe: A database and web-based tool for image annotation[END_REF].

However, these visual hierarchies face a major problem which is the lack of expressiveness at the semantic level. Indeed, these hierarchies are not usable at the semantic level since they do not carry any semantics, and are uninterpretable as illustrated in Figure 3.2. Consequently, these hierarchies serve only as visual taxonomies, and can only be used for hierarchical image classification in order to improve the classification accuracy. Hence, a good direction for building meaningful and suitable semantic hierarchies for the purpose of image annotation would be to make use of both information sources, conceptual and visual.

Semantic Hierarchies

Although the two first categories of hierarchies have received more attention, they showed a limited success in their general usage. Indeed, the conceptual semantics is not always consistent with the perceptual (visual) semantics of images, and is then insufficient to build good semantic structures for the purpose of image annotation [START_REF] Wu | Flickr distance: A relationship measure for visual concepts[END_REF]. Whereas the perceptual semantics cannot lead by itself to have a meaningful semantic hierarchy (cf. Figure 3.2), i.e. it is difficult to interpret these hierarchies in higher levels of abstraction. Therefore, it seems mandatory to combine the both components of image semantics, i.e. perceptual and conceptual, in order to build semantic hierarchies suitable for image applications.

For instance, a semantic hierarchy based on a visual similarity (between images) and a contextual similarity (between their tags) is proposed in [Fan et al., 2007, Shen andFan, 2010]. [Fan et al., 2008a] proposed an approach for hierarchical concept learning, by incorporating concept ontology and multi-task learning in order to exploit the strong inter-concept correlations. [START_REF] Li | Building and using a semantivisual image hierarchy[END_REF] proposed a method based on visual features and image tags to automatically build a 'semantivisual ' image hierarchy.

To summarize, semantic hierarchies seem to have great potentials to improve the image annotation, mostly through their explicit representation of concepts relationships which may help understanding image semantics. However, to our knowledge, none of the existing approaches have used the reasoning power of these hierarchies in this context. Most of the proposals have focused on the problem of hierarchies building, or have proposed to use them as a hierarchical framework for image classification.

Discussion

As we have seen in the previous sections, many approaches have proposed to use WordNet in order to extract/build a hierarchical structure that will serve thereafter for hierarchical image classification. However, the use of WordNet in this way implicitly assumes that conceptual and visual information are tightly correlated. While this assumption might be true for many concepts, it does not hold for all the concepts of WordNet, and respectively for a large annotation vocabulary. Indeed, we believe that WordNet is not necessarily appropriate for modeling image semantics. The organization of concepts in WordNet follows a psycholinguistic structure, which may be useful for reasoning about the concepts and understanding their conceptual meaning, but it is limited and inefficient for reasoning about the image context or its content. More precisely, the distances between related concepts in WordNet do not necessarily reflect an appropriate semantic measure for annotating images or reasoning about them, i.e. the distances between concepts is not proportional to their semantic relatedness with respect to the image domain. Although [START_REF] Torralba | 80 million tiny images: A large data set for nonparametric object and scene recognition[END_REF], Deng et al., 2010, Deselaers and Ferrari, 2011] assert have find a surprisingly strong correlation between purely linguistic metrics and the performance of visual classification algorithms, we will argue in the rest of this chapter that their statement does not always hold, specifically in broad domain applications. For instance, let us consider the example depicted in Figure 3.3. For fairness, we used the same conceptual (linguistic) measure than [START_REF] Deng | What does classifying more than 10,000 image categories tell us?[END_REF], i.e. Word-Net density h(c i , c j ), defined as the height of the lowest common ancestor of two concepts c i and c j . According to the WordNet density measure: h(Human, Dolphin) = 5, h(Shark, Human) = 5 and h(Shark, Dolphin) = 7. This implies that concept "Dolphin" is closer to "Human" than to "Shark", which is consistent from a biological point of view since "Dolphin" and "Human" are mammal while "Shark" is not. However, in the image domain it is more accurate to have higher similarity between concepts "Shark" and "Dolphin" as they live in the same environment, share similar visual features, and for sure share a higher value in the confusion matrix. Therefore, a suitable semantic hierarchy should represent this information or allow deducing it in order to help understanding the image semantics. To sum up , we propose in Table 3.1 a comparison of the different types of hierarchies used for annotating images.

Proposed Measure: Semantico-Visual Relatedness of Concepts (SVRC )

Based on the previous discussion, we define the following assumptions which motivated the design of our approach:

Assumption 1. A suitable semantic hierarchy for image annotation should:

A1-model image context, and consequently incorporates all aspect of image semantics, i.e. at least the visual and conceptual information between image concepts.

A2-reflect image semantics, i.e. the organization of concepts into the hierarchy and their semantic relatedness reflect image semantics -cf. Chapter2 -Section 2.3.

A3-allow grouping visually similar concepts in order to obtain better performance of classifiers.

Following the above assumptions, we propose in the remaining of this chapter a new method for building semantic hierarchies dedicated to image annotation. Our approach is based on a new measure to estimate the semantic relatedness between concepts, which is more faithful to image semantics since it is based on its different modalities. As illustrated in Figure 3.4, this measure, named SVRC, is based on 1) a visual similarity which represents the visual correspondence between concepts, 2) a conceptual similarity which defines a relatedness measure between target concepts, based on the concept definition in WordNet, and 3) a contextual similarity which measures the distributional similarity between each pair of concepts. SVRC is then used in TRUST-ME, a set of heuristic rules that allow deciding about the likelihood of the semantic relatedness between concepts, and help building the final semantic hierarchy.

Problem Formalization

In the following we introduce a formal description of our problem.

Given:

• DB, a training image database consisting of a set of pairs image/textual annotation , i.e.

DB = {[i 1 , A 1 ], [i 2 , A 2 ], • • • , [i L , A L ]}
, where:

-

I = i 1 , i 2 , • • • , i L is the set of all images in DB,
-L is the number of images in the database.

-C = c 1 , c 2 , • • • , c
N is the annotation vocabulary used for annotating images in I,

-N is the size of the annotation vocabulary.

-A i is a textual annotation consisting of a set of concepts {c j ∈ C, j = 1..n i i } associated with a given image i i ∈ DB.

• CO, a generic commonsense ontology containing N ′ concepts (C ), such that C ⊆ C . For this work, we used WordNet as a commonsense ontology.

Our objective is to build a semantic hierarchy (SH), consisting of a set of |C|+|C ′ | concepts (s.t. C ∪ C ′ ⊆ C , and C ′ could be probably the empty set), dedicated to this specific annotation problem, i.e. dependent on the initial annotation vocabulary. The built semantic hierarchy should consider the previously defined assumptions 1. The proposed approach consists therefore in identifying C ′ new concepts that link (and subsume) all the concepts of C in a hierarchical structure that accurately represents image semantics. Figure 3.5 illustrates the architecture of our approach for building semantic hierarchies dedicated to image annotation.

Annotations of Images

Training Dataset Data Processing Information Analysis

Building Semantic Hierarchies

Commonsense Knowledge Base

Conceptual information

Figure 3.5: From image data to structured knowledge models: Architecture of our approach for building semantic hierarchies dedicated to image annotation.

Visual Similarity Between Concepts

The visual similarity between concepts allows estimating the visual correlation between two concepts. Many recent approaches have proposed to measure this similarity using:

• the confusion matrix [START_REF] Griffin | Learning and using taxonomies for fast visual categorization[END_REF], Bengio et al., 2010, Deng et al., 2010],

• Kullback-Leibler (KL) divergence [START_REF]Mining multilevel image semantics via hierarchical classification[END_REF], or its variant Jensen-Shannon (JS) divergence [START_REF] Wu | Flickr distance[END_REF],

• the common features shared across the classes [START_REF] Torralba | Sharing visual features for multiclass and multiview object detection[END_REF], or using

• canonical correlation analysis between the image sets of targeted concepts [START_REF] Fan | Integrating visual and semantic contexts for topic network generation and word sense disambiguation[END_REF].

In our approach, we use a simple but effective method for estimating the visual correlation between concepts. Indeed, we propose to compute for each visual concept a centroid assumed to be representative of its visual appearance. These centroids are computed using the set of support-vectors defining each SVM classifier associated with a given concept. Therefore, the visual similarity between two given concepts is computed as the inverse Euclidean distance between their centroids. In the following, we detail the proposed visual similarity.

Let x v i be any visual representation of an image i i (a visual feature vector), we train for each concept c j a classifier that can associate this concept with its visual features. For this, we use N binary Support Vector Machines (SVM) [START_REF] Cortes | Support-vector networks[END_REF] (One-Versus-All) with a decision function G(x v i ):

G(x v i ) = k α k y k K(x v k , x v i ) + b (3.1)
where N = |C| is the size of the annotation vocabulary, K(x v k , x v i ) is the value of a kernel function for the training sample x v k and the test sample After several tests performed on the training samples in order to find the kernel function which gives the best result for defining our classifiers, we decided to use a radial basis function kernel:

x v i , y k ∈ {1, -1} is the class label of x v k , α k is
K(x v k , x v i ) = exp x v k -x v i 2 σ 2 (3.2)
Now, given these N trained SVM where the inputs were the visual features of images and the outputs were the concepts c i ∈ C, we want to define a centroid ϑ(c i ) for each concept class c i that best represents it. These centroids should then minimize the sum of squares within each set S i as illustrated in Figure 3.6:

argmin S N i=1 x v j ∈S i x v j -µ i 2 (3.3)
where S i is the set of support vectors of class c i , S = {S 1 , S 2 , • • • , S N }, and µ i is the arithmetic mean of vectors in S i .

δ j

Data Centroids

Figure 3.6: Objective: for each concept c i ∈ C, find a centroid ϑ(c i ) that minimizes δ j (δ j is the intra-class variance) for all x v j ∈ S i .

Our objective being to estimate a distance between these concepts c i ∈ C in order to assess their visual similarities, we compute the centroid ϑ(c i ) of each visual concept c i as follows:

ϑ(c i ) = 1 |S i | x j ∈S i x v j (3.4)
The visual similarity between two concepts c i and c j , is then inversely proportional to the distance between their visual features (centroids) ϑ(c i ) and ϑ(c j ):

ϕ(c i , c j ) = 1 1 + d(ϑ(c i ), ϑ(c j )) (3.5)
where d(ϑ(c i ), ϑ(c j )) is the Euclidean distance between ϑ(c i ) and ϑ(c j ).

Conceptual Similarity

Conceptual similarity reflects the semantic relatedness between two concepts from a linguistic and/or a taxonomic point of view. Several conceptual similarity measures have been proposed [Resnik, 1995, Banerjee and Pedersen, 2003, Budanitsky and Hirst, 2006, Popescu and Grefenstette, 2011]. Most of them are based on a lexical resource, such as WordNet [Fellbaum, 1998]. A first family of approaches is based on the structure of this external resource (often used as a semantic network or a directed graph), and the similarity between concepts is computed according to the distances of the paths connecting them in this structure [START_REF] Budanitsky | Evaluating WordNet-based measures of lexical semantic relatedness[END_REF]. However, as introduced in Section 3.2.4, the structure of these resources does not necessarily reflect image semantics, and therefore such measures do not seem suited to our problem. An alternative approach to measure the semantic relatedness between concepts is to use their provided definitions. With respect to the WordNet resource, these definitions are known as glosses5 and are provided by the synsets 3 associated with each concept. For example, [START_REF] Banerjee | Extended gloss overlaps as a measure of semantic relatedness[END_REF] proposed a measure of semantic relatedness between concepts that is based on the number of shared words (overlaps) in their definitions (glosses).

In this work we used the gloss vector relatedness measure proposed by [START_REF] Patwardhan | Using WordNet-based context vectors to estimate the semantic relatedness of concepts[END_REF], in which they suggest to use "second order" cooccurrence vector of glosses rather than matching words that co-occur in it. Specifically, in a first step a word space of size P is built by taking all the significant words used to define all synsets of WordNet. Thereby, each concept c i is represented by a context vector -→ w c i of size P, where each n th element of this vector represents the number of occurrences of n th word in the word space in the gloss of c i . The semantic relatedness of two concept c i and c j is therefore measured using the cosine similarity between -→ w c i and -→ w c j :

η(c i , c j ) = - → w c i • - → w c j | - → w c i || - → w c j | (3.6)
Some concepts definitions in WordNet are very concise which could make this measure unreliable. Consequently, in the same spirit as [START_REF] Patwardhan | Using WordNet-based context vectors to estimate the semantic relatedness of concepts[END_REF], we propose to extend the glosses of the target concepts with the glosses of their adjacent concepts (located in their immediate neighborhood). Hence, for each concept c i , we define Ψ c i as the set of all adjacent glosses connected to c i , i.e. (Ψ c i ={gloss(c i ), gloss(hyponyms(c i )), gloss(meronyms(c i )), etc.}). Then, each element (gloss), denoted x, of Ψ c i is represented by -→ w x as explained above. The similarity measure between two concepts c i and c j is therefore defined as the sum of the individual Chapter 3. Building Semantic Hierarchies Faithful to Image Semantics cosines of the corresponding gloss vectors:

θ(c i , c j ) = 1 |Ψ c i | x∈Ψc i ,y∈Ψc j - → w x • - → w y | - → w x || - → w y | , where|Ψ c i | = |Ψ c j |. (3.7)
Finally, each concept c i ∈ C may be associated to one or several synsets in Word-Net. Usually, these synsets have different meanings, and thus differ from each other in their definitions (glosses) and in their positions in the hierarchy of WordNet. A disambiguation step is then mandatory in order to associate to each concept c i ∈ C the good synset with respect to its semantics. For instance, the similarity between "Mouse" (Animal) and "Keyboard" (device) widely differs from the one of "Mouse" (device) and "Keyboard" (device). In our approach, the disambiguation is performed as follows. First, we compute the conceptual similarity between the different senses (synsets) of c i and c j . The maximum likelihood of these similarities is then used to identify the most likely meaning of these two concepts, i.e. disambiguate c i and c j . Consequently, the conceptual similarity is calculated as following:

π(c i , c j ) = argmax δ i ∈s(c i ),δ j ∈s(c j ) θ(δ i , δ j ) (3.8)
where s(c x ) is "the set of all synsets that can be associated to the meanings of c x ".

Contextual Similarity

It is intuitively clear that if two concepts are similar or related, it is likely that their role in the world will be similar, and thus their context of occurrence will be equivalent (i.e. they tend to occur in similar contexts, for some definition of context).

The information related to the context of appearance of concepts, called contextual information, is used to connect concepts that often appear together in images although semantically distant from the taxonomic point of view. Moreover, this contextual information can also help to infer higher-level knowledge from images. For example, if a photo contains "Sea" and "Sand", it is likely that the scene depicted in this photo is the one of "Beach". It is therefore important to measure the contextual similarity between concepts. However, unlike the visual and the conceptual similarity, the contextual similarity is a corpus-dependent measure, and more precisely depends on the distribution of concepts in the corpus. Therefore, it is important to ensure an equivalent distribution of concepts within the training set, or to normalize the produced measure in order to perform a good contextual measure. Otherwise this measure will be biased from the outset and will be unreliable.

In our approach, we define the contextual similarity between two concepts c i and c j as the Pointwise Mutual Information [START_REF] Church | Word association norms, mutual information, and lexicography[END_REF]] (PMI), denoted ρ(c i , c j ), and computed as follows:

ρ(c i , c j ) = log P (c i , c j ) P (c i )P (c j ) (3.9)

where: P (c i ) is the probability of occurrence of c i , and P (c i , c j ) is the joint probability of c i and c j . These probabilities are estimated by computing the frequency of occurrence and co-occurrence of concepts c i and c j in the database.

Given N the total number of concepts in the database, L the total number of images, n i the number of images annotated by c i (occurrence frequency of c i ) and n ij the number of images co-annotated by c i and c j , the above probabilities can be estimated by:

P (c i ) = n i L P (c i , c j ) = n ij L
As a consequence:

ρ(c i , c j ) = log L * n ij n i * n j (3.10)
ρ(c i , c j ) quantifies the amount of information shared between the two concepts c i and c j . Thus, if c i and c j are independent concepts, then P (c i , c j ) = P (c i ) • P (c j ) and therefore ρ(c i , c j ) = log 1 = 0. ρ(c i , c j ) can be negative if c i and c j are negatively correlated. Otherwise, ρ(c i , c j ) > 0 and quantifies the degree of dependence between these two concepts.

In this work, we only want to measure the positive dependence between the concepts and therefore we set negative values of ρ(c i , c j ) to 0. Finally, to normalize the contextual similarity between two concepts c i and c j into [0,1], we compute it in our approach by: γ(c i , c j ) = ρ(c i , c j ) log[max(P (c i ), P (c j ))]

(3.11)

Computation of the Semantico-Visual Relatedness of Concepts

For two given concepts c i and c j , their similarity measures: visual ϕ(c i , c j ), conceptual π(c i , c j ) and contextual γ(c i , c j ) are first normalized into the same interval using the Min-Max Normalization [START_REF] Jain | Score normalization in multimodal biometric systems[END_REF]. Then, the Semantico-Visual Relatedness φ(c i , c j ) of these concepts c i and c j is computed as:

φ(c i , c j ) = ω 1 • ϕ(c i , c j ) + ω 2 • π(c i , c j ) + ω 3 • γ(c i , c j ) (3.12)
where 3 i=1 ω i = 1. ϕ(c i , c j ), π(c i , c j ), and γ(c i , c j ) are the normalized visual similarity, conceptual similarity and contextual similarity.

The choice of the weights ω i is very important. According to the target application, some would prefer to build a domain-specific hierarchy (that best represents a specific-domain or corpus), and can therefore assign a higher weight to the contextual similarity (ω 3 ր). Others would be conducted to build a generic hierarchy, and will therefore assign a higher weight to the conceptual similarity (ω 2 ր). However, if the purpose of the semantic hierarchy is rather to build a hierarchical framework to image classification, it may be advantageous to assign a higher weight to the visual similarity (ω 1 ր).

The building of semantic hierarchies using our approach remains flexible, depending on the sought image semantics and the targeted application. However, the recommended method to find the best weights (w i , i ∈ [1, 3]) needed to compute the SVRC measure is to use cross-validation. Indeed, using the cross-validation, it is possible to assess the impact of weights distribution as a function of the system performance. Thus, finding the best weights to achieve the best performance of the system. More details on the impact of these weights on our "Semantico-Visual" measure, and the choice of the weighting factor by cross-validation are proposed in Section 3.5.2.

Rules for the Hierarchy Building

Once we have estimated the semantic relatedness between each pair of concepts, it is important to group them in a more comprehensive hierarchy despite the uncertainty introduced by semantic similarity measurements. In the following, we propose a heuristic named TRUST-ME, that allows to infer hypernym relationships between concepts, and to bring together these various concepts in a hierarchical structure.

Let us define the following functions to understand the reasoning rules we used for the building of our hierarchy:

• Closest(c i ) returns the closest concept to c i according to the SVRC measure:

Closest(c i ) = argmax c k ∈C\{c i } φ(c i , c k ) (3.13)
• LCS(c i , c j ) allows to find the Least Common Subsumer of c i and c j in WordNet:

LCS(c i , c j ) = argmax c k ∈{H(c i )∩H(c j )} len(c k , root) (3.14)
where H(c i ) is a function allowing to find the set of all hypernyms of c i in WordNet, root is the root node of WordNet, and len(c x , root) returns the length of the shortest path in WordNet between the concept c x and the root.

• Hits 3 (c i ) is a function returning the 3 closest concepts to c i within the meaning of Closest(c i ).

Basically, TRUST-ME consists of three rules which are based on the SVRC measure and on the reasoning about the Least Common Subsumer (LCS) to select the concepts to be connected to each other. These rules are illustrated and executed in the order described in Figure 3.7.

Rule 1.

The first rule checks whether a concept c i is classified as the closest relative to more than one concept, i.e. (Closest(c j ) = c i ), ∀j ∈ {1, 2, • • • }. In this case, if these concepts {c j , ∀j ∈ {1, 2, • • • }} are reciprocally in Hits 3 (c i ), then according to their LCS they will be connected either directly to their LCS or in a two levels structure as illustrated in Figure 3.7(a).

Rule 2.

In the second rule, if (Closest(c i ) = c j ) and (Closest(c j ) = c i ) (can also be written as Closest(Closest(c i )) = c i ), then c i and c j are actually related and are connected to their LCS -cf. Figure 3.7(b).

Rule 3.

The third rule covers the case when (Closest(c i ) = c j ) and (Closest(c j ) = c k ). In this case, if c i ∈ Hits 3 (c j ) and c j ∈ Hits 3 (c k ), then according to the LCS of c i , c j and c k , these concepts will be connected either directly to their LCS or in a tow levels structure as illustrated in Figure 3.7(c).

The building of the semantic hierarchy is a bottom-up process (starts from the leaf concept nodes), and uses an iterative algorithm until reaching the root node. Given a set of concepts associated to a set of images in a dataset, our method compute the SVRC (φ(c i , c j )) between all pairs of concepts, then links most related concepts to each other while respecting the defined rules in TRUST-ME -cf. Algorithm 1. Thus, we obtain a new set of concepts in a higher level resulted by the linked concepts in the lower level. We iterate the process until all the concepts are linked to a root node. In Section 3.5, we illustrated some semantic hierarchies built using our method, in particular on VOC'2010 dataset.

The complexity of our method for building hierarchies is of O(N 2 ), where N is the size of the initial annotation vocabulary. Indeed, our method requires two steps: i) computing the similarity between each pair of concepts, which is performed in O(N 2 ), and ii) connecting each pair of concepts to the least common subsumer using the rules illustrated in Figure 3.7, which is performed in the worst-case in O(N log(N )).

Experimental Results

As part of this chapter, we will only illustrate the obtained hierarchies using the above described method on the Pascal VOC'2010 dataset [START_REF] Everingham | The PASCAL Visual Object Classes Challenge 2010[END_REF], and the impact of the weighting factors on the consistency of the produced hierarchies. We also evaluate the distribution of the proposed similarities on the different image modalities, i.e. visual, conceptual and contextual similarity. An experimental evaluation of the produced hierarchies within a framework of hierarchical image classification is performed in Chapter 5.

Visual Representation of Images

To compute the visual similarity of concepts, we used in our approach the Bag-of-Features (BoF) model, also known as bag-of-visual words, which is a widely known method [START_REF] Li | A bayesian hierarchical model for learning natural scene categories[END_REF]. The BoF model has shown excellent performances and became one of the most widely used techniques for image classification and object recognition. The BoF model consists of three steps: feature detection, feature description and the codebook generation. The feature detection process is performed by the extraction of several local patches (or regions), which will be considered as visual words. In our approach, the used BoF model is built as follows. Lowe's DoG Detector [Lowe, 1999] is used for detecting a set of salient image regions. A signature of these regions is then computed using SIFT descriptor [Lowe, 1999]. The SIFT descriptor has proofed to be invariant to intensity, rotation, scale and affine variations to some extent, which makes it very robust to feature description. This descriptor converts each patch to 128-dimensional vector describing gradient magnitudes and orientations around the keypoint. Each image in the database is then represented as a collection of non-ordered vectors of the same dimension (128 for SIFT).

Afterwards, given the collection of detected patches (or regions) from the training dataset of all categories, we generate a codebook of size K = 1000 by performing the k-means algorithm. Since the K-means algorithm is sensitive to initialization, we control clusters after each iteration to ensure that we do not have empty ones. Empty clusters are again randomly initialized. Finally, the generated codebook is a set of features assumed to be representative of all images features. Thus, each patch (detected region) in an image is mapped to the most similar visual word in the codebook through a KD-Tree. Each image is then represented by a histogram of D visual words, where each bin in the histogram correspond to the occurrence number of a visual word in that image.

Impact of Weighting on the Building of Hierarchies

This work aims at building meaningful semantic hierarchies for the purpose of image annotation. Thus, for our experiments we set the weighting factors in an experimental way as follows: ω 1 = 0.4, ω 2 = 0.3, and ω 3 = 0.3. Indeed, we used cross-validation in order to retrieve the best set of weights that allows to maximize the accuracy of hierarchical image classification. The adjustment of weighting was performed using a step of 0.1. In fact, during our experiments we observed that the use of a lower step for adjusting weights (△ ω i < 0.1) does not have a significant impact on the produced hierarchy, but the counterpart is an execution time which grows exponentially. We also underline that the accuracy of image classification depends mostly on the structure of the hierarchy and not on the used weights.

Furthermore, our experimentations on the impact of weights (ω i ) showed also that the visual similarity is more representative of concepts similarity, as it will be illustrated with the produced hierarchies in Figure 3.8, Figure 3.11 and Figure 3.10, and with the heat maps in Figure 3.12.

In order to illustrate our approach, we show in Figure 3.8 the built semantic hierarchy using the weighting factors described above, and in Figure 3.9 the buit binary hierarchy based on our semantic-visual relatedness of concepts (SVRC) and the second rule of TRUST-ME. The semantic hierarchy is built under the shape of a taxonomy, i.e. a concept c i may have several hyponyms: hyponyms(c i ) = {c f , 0 ≤ f ≤ |C|}. The binary hierarchy is built by restricting the number of child nodes of each concept to 2, i.e. hyponyms(c i ) = {c f , 0 ≤ f ≤ 2}. As can be seen in these two figures, the structure of the hierarchy and the relationships between concepts seem to be coherent at both, conceptual and visual point of view. Figure .3.11 illustrates the built hierarchy on Pascal VOC'2010 dataset, using a similarity measure based on the shortest path in WordNet between the considered concepts. Figure .3.10 illustrates the built hierarchy on the same dataset by using only the visual similarity and our heuristic TRUST-ME. Compared to the produced semantic hierarchy in Figure 3.8, these hierarchies seem to be less consistent and less efficient. Unfortunately, to our knowledge there is no method currently available that would allow to compare these hierarchies in terms of structures nor in terms of informative modelling. However, we can easily note some inconsistency with the hierarchies built using only the conceptual similarity (respectively the visual similarity), as for example:

• in Figure 3.11, "Bicycle" was considered as a hyponym of "Container" instead of "Vehicle" (both are hypernyms of "Bicycle" in WordNet). Indeed, according to the shortest path in Wordnet, the distance between "Bicycle" and "Bottle" is less than the distance between "Bicycle" and "Motorbike" (respectively less than the distance to all the other concepts of Pascal VOC'2010). Consequently, "Bicycle" and "Bottle" have been considered as very close and have shared "container" as hypernym, which is senseless with respect to the context of Pascal VOC dataset. An explanation to this result is expressed in Definition 10.

• in Figure 3.10, the concepts "Car" and "Train" are considered as closer to "Tv-Monitor" and "Chair" than to "Motorbike", "Boat" and "Aeroplane", which is not coherent. Also, "Car" and "Train" are considered as hyponyms of "Instrumentality" instead of "Conveyance", which could make sense in some context, but in the context of VOC'2010 dataset we believe that "Conveyance" sounds better.

Definition 10 (Conceptual Semantics). The precise semantic meaning of a concept can only be understood in a specific-context6 , and therefore, it is difficult to gain high accuracy without modeling this specific-context. Consequently, conceptual similarity is context-sensitive. In Figure 3.12 we illustrate the semantic affinity matrix of the VOC'2010 concepts according to the different image modalities, i.e. visual, conceptual, contextual and semantic modality. For each modality, the semantic affinity matrix (also called distance matrix) between concepts is represented as a heat map 7 . This choice is motivated by the easiness provided by heat maps to compare the distribution of individuals. We can see in this figure that the distribution of correlation between each pair of concepts is widely different according to the image modality, and respectively the used measure. This makes sense if we consider the problems related to each modality. Remember, the visual similarity reflects only the perceptual similarity between con-cepts, the contextual similarity is a "corpus-dependent" measure (i.e. depends on the concept distribution in the corpus), and the conceptual similarity is context sensitive (cf. Definition 10). This observation has motivated our choice for integrating the different image modalities in order to decrease the limitation of each of them.

According to the obtained results in Figure 3.12, we make the statement that the visual similarity, the conceptual similarity and the contextual one are not always correlated and have different distributions with respect to the targeted semantic or visual space, partly due to the aforementioned problems. Therefore, we do not agree with the statements of [START_REF] Deselaers | Visual and semantic similarity in ImageNet[END_REF] who reported that: i) the visual variability within a category grows with its semantic domain and ii) visual similarity grows with semantic similarity. At least, we believe that these statements cannot be generalized to any concept.

Finally, we can see in Figure 3.12 that, with respect to each similarity measure, there are some relevant results and many non-relevant ones with respect to image semantics (image context). In Table 3 colored in red irrelevant results and in green sufficiently relevant results8 . We can see for instance, that according to the conceptual similarity Closest(Horse)=Sofa, or also according to the visual similarity Closest(Dog)=Car. For cons, as illustrated in Figure 3.12(d), the combination of the different measures, i.e. visual, conceptual and contextual, allows usually recovering from these inconsistencies. Our heuristic TRUST-ME is supposed to make the final adjustment in order to achieve a more reliable decision on the relatedness of the concepts, and to link together in a semantic hierarchy the concepts that are semantically related.

Conclusion

In this chapter we proposed a new approach to automatically build a suitable semantic hierarchy for image annotation/classification. Our approach is based on a new measure of image-semantic relatedness, named "Semantico-Visual Relatedness of Concepts" (SVRC ), which takes into account the visual similarity and also the conceptual and the contextual ones. As illustrated in our experiments, SVRC provides a measure that is more faithful to the image semantics, and consequently allows to dispose of more meaningful and consistent semantic hierarchies with respect to image context. A new heuristic, named TRUST-ME, is also proposed for reasoning about concepts relatedness, and to link together in a semantic hierarchy the concepts that are semantically related. Our experiments showed that the built semantic hierarchy using our semantico-visual relatedness measure (SVRC) and the proposed heuristic (TRUST-ME) is more coherent compared with the hierarchies produced using only visual information, and only conceptual information. An experimental evaluation of the produced hierarchies within a framework of hierarchical image classification is proposed in Chapter 5. Despite the improvement achieved using semantic hierarchies as a hierarchical framework for image classification, there is still a gap between the low-level visual features and the high-level concepts, i.e. the semantic gap. Therefore, there is a need to make use of the explicit knowledge of these hierarchies by supplying a formal framework to reason about the coherence of extracted information from images. In Chapter 4 we propose a new approach to build multimedia ontologies that serve as a framework for reasoning about annotation consistency, and thus allow an effective use of explicit knowledge extracted from images.

Introduction

In Chapter 3, we have proposed a new method for building semantic hierarchies suitable for image annotation. As aforementioned, semantic hierarchies enable only the description of the subsumption relationships between concepts, which is actually insufficient for modeling the rich semantics of multimedia content and reasoning about it. Indeed, more sophisticated knowledge structures, i.e. richer in terms of semantic relationships, are required in order to deal with big data currently available everywhere, and in order to provide efficient systems for storing, indexing and retrieving in this amount of data while satisfying semantic user expectations.

This chapter proposes an approach to build an ontology of spatial and contextual information suitable for reasoning about the coherence of image annotation. Our approach uses firstly the visual and conceptual information in order to build a semantic hierarchy that will serve as a backbone of our multimedia ontology. Contextual and spatial information about image concepts are then computed and incorporated in the ontology to model other semantic relationships between concepts. We also have chosen to use fuzzy description logics as a formalism to represent our ontology in order to take into account the uncertainty and the imprecision of these kinds of information, but also in order to enable formal reasoning as it will be introduced in Chapter 6.

The rest of this chapter is structured as follows. In Section 4.2, we highlight our motivations for proposing a new approach for building multimedia ontologies dedicated to image annotation. Section 4.3 presents an overview of the proposed approach. Section 4.4 introduces the proposed formalism for our multimedia ontology, and the set of axioms and inferences rules allowing to perform the reasoning tasks. Section 4.5 introduces the design of our multimedia ontology as a knowledge base, i.e. the construction of the TBox, RBox, and ABox. In Section 4.6, we illustrates the obtained multimedia ontology on the Pascal VOC dataset. A discussion about the proposed approach and the usefulness of our multimedia ontology for computer vision tasks is presented in Section 4.7. The chapter is concluded in Section 4.8.

Context and Motivations

Context and General Problems

As introduced in Chapter 2, most approaches for image annotation rely on machine learning techniques to provide a mapping function that allows classifying images in semantic classes using their visual features [START_REF] Barnard | Matching words and pictures[END_REF], Lavrenko et al., 2003, Carneiro et al., 2007]. However, these approaches face the scalability problem when dealing with broad content image databases [START_REF] Liu | A survey of content-based image retrieval with high-level semantics[END_REF], i.e. their performances decrease significantly when the concept number is high and depend on the targeted datasets as well [START_REF] Hauptmann | How many high-level concepts will fill the semantic gap in news video retrieval?[END_REF]]. Yet, more and more concept classes are introduced for annotating multimedia content in order to enrich the description of images and to satisfy user expectations in a multimedia retrieval system. Consequently, current techniques are struggling to scale up. Therefore, the only use of machine learning seems to be insufficient to solve the problem of image annotation. Firstly, because of the lack of a computer (or statistical) model that allows to model the correlation between the low-level features of images and the semantic concepts. Secondly, because it seems that there is a lack of coincidence between the high-level concepts and the low-level features, and that image semantics is not always correlated with the visual appearance. Therefore other alternatives need to be explored in order to improve existing approaches. In particular, some recent work proposed to use explicit semantic structures, such as semantic hierarchies or ontologies to improve image annotation [START_REF] Dasiopoulou | Applying fuzzy DLs in the extraction of image semantics[END_REF], Straccia, 2010, Wu et al., 2012, Dong et al., 2012].

Indeed, ontologies defined as a formal, explicit specification of a shared conceptualization [Gruber, 1995] have shown to be very useful to narrow the semantic gap. They allow identifying, in a formal way, the dependency relationships between the different concepts, and therefore provide a valuable information source for many problems. Moreover, ontological reasoning can also be used to formulate image interpretation tasks [Hudelot, 2005]. For instance, in [START_REF] Dasiopoulou | Using Fuzzy DLs to Enhance Semantic Image Analysis[END_REF], the authors proposed a framework for the extraction of enhanced image descriptions based on an initial set of graded annotations, generated through generic image analysis techniques. Explicit semantics, represented by ontologies, have also been intensely used in the field of image and video indexing and retrieval [START_REF] Hudelot | Symbol grounding for semantic image interpretation: From image data to semantics[END_REF], Kompatsiaris and Hobson, 2008, Hudelot et al., 2008]. In most of these approaches, only the descriptive part of ontologies is used as a common multi-level language to describe image content [START_REF] Simou | Image indexing and retrieval using expressive fuzzy description logics[END_REF], or more recently as semantic concept networks to refine image annotation [START_REF] Wei | Ontology-enriched semantic space for video search[END_REF], Fan et al., 2008a, Wu et al., 2012], or to perform image classification [START_REF] Marszalek | Semantic hierarchies for visual object recognition[END_REF], Torralba et al., 2008, Dong et al., 2012].

However, much remains to be done in order to achieve more expressive ontologies of image semantics. In the following we describe the noted problems.

GP1 Firstly, most of existing approaches for building multimedia ontologies start from an existing specification (defined by an expert, or inferred from a generic commonsense ontology, etc.) of a domain in order to design their ontologies. However, any given specification, as well-defined it may be, remains incomplete, subjective and subject to many inconsistencies. Indeed, many assumptions about the concepts, their properties and relationships must be done in order to achieve a given specification, which finally do not hold in the real world. These approaches remain useful, but it is more interesting to try to solve the problem in a more generic way by building automatically knowledge models using data mining techniques.

GP2 Secondly, most of approaches for building multimedia ontologies are based either on a conceptual specification [START_REF] Jaimes | Semi-automatic, datadriven construction of multimedia ontologies[END_REF], Hoogs et al., 2003, Snoek et al., 2007], or on a visual one [START_REF] Mezaris | An ontology approach to object-based image retrieval[END_REF], Maillot et al., 2004, Yao et al., 2010]. However, as previously pointed out in Chapter 3, conceptual semantics/specification is not always consistent with the perceptual semantics of images (cf. Figure 3.12), and is then insufficient to build ontologies dedicated to image annotation. In the other hand, the perceptual (visual) semantics does not in itself lead to significant semantic structures. Therefore, these approaches do not allow to model image semantics accurately.

GP3 Finally, many of these approaches are limited to provide a formalism allowing to use ontologies as a repository for storing knowledge about multimedia content. However, since these approaches have not addressed the problem of reasoning about this knowledge, the effectiveness of stored knowledge has to be proved.

Motivations for Building Ontologies as a Knowledge Base

According to [START_REF] Spaccapietra | On spatial ontologies[END_REF], ontologies can be classified as taxonomic ontologies, descriptive ontologies, or as knowledge bases, according to the way they are designed. Early efforts on building ontologies have focused on taxonomic ontologies, i.e. ontologies acting as thesauri or sophisticated dictionaries. These approaches have focused on defining the used terms in a given domain, and the organization of these terms into generalization/specialization hierarchies, enriched by semantic links (relationships) commonly used in linguistics (e.g., synonymy, antonymy, meronymy, etc.). WordNet [Fellbaum, 1998] and LSCOM [START_REF] Naphade | Large-scale concept ontology for multimedia[END_REF] are two examples of taxonomic ontologies. Some other approaches addressing the ontology building problem, have focused on supporting richer ontology models that enable sharing more complex information. They tend to enrich the description of the concepts semantics by associating to each concept a structured description of its properties and its relationships with the others. These ontologies, called descriptive ontologies, allow modeling some given domain and the appropriate terms to talk about its concepts. DOLCE1 is an example of such ontologies [START_REF] Gangemi | Sweetening ontologies with DOLCE[END_REF].

The final category of approaches have proposed to build ontologies as a knowledge bases. These ontologies consist of conceptual knowledge, i.e. description of concepts and their relations, and knowledge about the instances. Indeed, in this dissertation we refer to an ontology with associated instances as a knowledge base.

With respect to multimedia domain, a core problem with most of existing approaches for ontologies building is that they are limited to propose taxonomic or descriptive ontologies, i.e. they have only focused on describing the concepts (of a given domain) and their relationships. Even the supplied rules2 and axioms3 by these approaches are only used to provide a better description of the concepts and their roles. However, an efficient ontology would be rather built as a knowledge base, i.e. by modeling conceptual knowledge about the concepts and their relationships, while including instances and inference rules allowing reasoning on them in an effective manner.

Such ontologies allow terminological reasoning (on concepts and inter-concepts relationships) and assertional reasoning (on instances). Within this meaning, we are interested in this chapter on building multimedia ontologies as a knowledge base. Specifically, we propose an approach that aims at modeling image semantics through the representation of the semantic relationships between concepts, while including valuable knowledge about the image context under the shape of instances of these concepts and relationships. These instances are automatically recovered from image databases, which allows achieving a more representative knowledge base of the image semantics (and respectively the image context).

Motivations for Building Fuzzy Formal Ontologies

This work aims at building (automatically) a multimedia ontology which models knowledge about image semantics, and which will subsequently allow performing reasoning tasks in order to achieve a semantically consistent image annotation. With respect to the knowledge engineering domain, the reasoning tasks require to dispose firstly of a formal ontology.

In this regard, Description Logics (DLs) appeared to us as an excellent candidate to support ontology representation and ontological reasoning. Indeed, as introduced in Section 2.5.2.4, description logics are a family of formal languages for knowledge representation (respectively ontologies representation) and reasoning about them. The goal of reasoning, within this context, is to infer implicitly represented knowledge from the knowledge that is explicitly contained in the knowledge base.

However, despite the rich expressiveness of DLs, they lack the ability to deal with vague and uncertain information which is very common in multimedia content/annotation [START_REF] Simou | Image indexing and retrieval using expressive fuzzy description logics[END_REF]. Indeed, as often stressed in this dissertation, multimedia applications, and specifically image annotation, face a major problem of uncertainty introduced by machine learning algorithms, but also the semantic gap problem, the subjectivity of human perception, etc.

To cope with the problem of uncertainty of image annotation, we have chosen to use Fuzzy-DLs as a formalism for representing our multimedia ontology. Indeed, in DLs a statement s is either true or false (s ∈ {0, 1}), whereas in fuzzy DL, s ∈ [0, 1] is the degree of truth of the statement. Consequently, the value of s is used to model the uncertainty of an assertion. Thereby, we propose in this chapter a new approach for building a fuzzy formal ontology dedicated to image annotation. The proposed ontology is built using a highly expressive formalism, i.e. f -SROIQ(D), which provides many axioms for building the ontology and reasoning about its instances. The formalism of our multimedia ontology is introduced in Section 4.4.2.

However, it is important to note that DL strength is on the theoretical level. From a practical standpoint, DL ontologies face significant problems and their effectiveness is still to be proved. For instance: DLP1 Ontology modeling in DL is not at all an intuitive task. The representation of each single real world object is split into many axioms about concepts and roles, leading to an overall design that is very difficult to apprehend [START_REF] Spaccapietra | On spatial ontologies[END_REF].

DLP2 This makes the design by humans of a well-defined ontology a big challenge, with no guarantee of success (scalability problem of ontology building).

DLP3 Querying functionality are limited in Description logics, i.e. a more expressive ontology query language is needed.

DLP4 Scalability of DL reasoners, when dealing with very large knowledge bases, is not demonstrated .

Discussion

In this dissertation, we propose to go deeper in the use of ontologies in order to improve image annotation. Our objective is twofold. We first propose an approach to automatically build a fuzzy multimedia ontology suitable for image annotation. Our multimedia ontology includes visual, conceptual, contextual and spatial knowledge. Indeed, spatial and contextual knowledge are two valuable sources of information for image annotation and interpretation [START_REF] Hollink | Adding spatial semantics to image annotations[END_REF], Hudelot et al., 2008, Wu et al., 2008]. Secondly, we propose in Chapter 6 a generic approach for image annotation, combining both machine learning techniques, such as hierarchical image classification, and fuzzy ontological reasoning. The proposed approach uses explicit knowledge stored in our multimedia ontology, and ontological reasoning on the produced image annotations by machine learning techniques in order to lift the ambiguity of these annotations and to achieve a better accuracy on the description of images.

In this chapter, we propose a new approach for building multimedia ontologies, while trying to handle some of the aforementioned issues. Specifically, our main contributions can be summarised as follows:

1. Our approach allows for building multimedia ontologies as a knowledge base, and consequently allows for terminological reasoning and assertional reasoning.

Therefore, we answer the problem stated in Section 4.2.2.

2. In our approach, we use data mining techniques in order to automatically discover from image databases the adequate specification (or ontology) for representing image content. This specification is assumed to be faithful to data semantics. Thus, we also show that ontologies are not necessarily limited to be defined by humans. Therefore, we reduce the following problems: DLP1, DLP2, GP1 (introduced above).

3. Our approach uses different modalities of the image semantics in order to build an ontology that best represents the data semantics. Thus, we solve the problem GP2.

4. Our framework allows both, multimedia ontologies building and ontological reasoning in order to produce a semantically consistent image annotation. The obtained results have shown a significant improvement in the accuracy of image annotation. Thus, our approach allows to answer to the limitation GP3 stated before.

5. In recent work, the uncertainty was about the truth-degree of a given assertion with respect to the real world facts, e.g. how much it is true that the earth is round, or how much it is true that the left hemisphere is on the left of the right hemisphere (depending on the actual angle between them). In our approach, every assertion is subject to uncertainty with the respect to image semantics, e.g. how much it is true that a mouse appears on the right of a keyboard (given an image database and according to the appearance context).

Overview of our Approach for Building Multimedia Ontologies

We propose in the rest of this chapter, a new approach for building a fuzzy multimedia ontology dedicated to image annotation. Our ontology incorporates several types of knowledge about image semantics. Furthermore, this knowledge is extracted automatically from image databases using data mining techniques, which make it faithful to image semantics. In our approach, three steps are required for knowledge discovery from image databases:

1. Processing the set of images in the training dataset to discover useful knowledge about the image domain (i.e. perceptual semantics), such as the visual similarity between concepts.

Chapter 4. Building Fuzzy Multimedia Ontologies for Image Annotation 3. Query a commonsense knowledge base to gather precise information about the semantics of image concepts, and in order to link the initial concepts to their hypernyms using the method proposed in Chapter 3.

Thereafter, the building of our multimedia ontology is fully automatically performed, i.e. without any human intervention, using the formalism described in Section 4.4.2, and the extracted knowledge from image databases which will be detailed in the following.

Problem Formalization.

Given:

• DB, an image database consisting of a set of pairs image/textual annotation , i.e.

DB = {[i 1 , A 1 ], [i 2 , A 2 ], • • • , [i L , A L ]}
, where:

-

I = i 1 , i 2 , • • • , i L is the set of all images in DB,
-L is the number of images in the database.

-C = c 1 , c 2 , • • • , c
N is the annotation vocabulary used for annotating images in I,

-N is the size of the annotation vocabulary.

-A i is a textual annotation consisting of: * the set of concepts {c j ∈ C, j = 1..n i i } associated with a given image i i ∈ DB, * the spatial location of each concept c j in the image i i given by its minimum bounding box defined as (c j xmin , c j ymin , c jxmax , c jymax ), where c j xmin and c j ymin are the coordinates of the low left corner of the bounding box (and respectively c jxmax and c jymax are the coordinates of the upper right corner of the bounding box).

• CO, a generic commonsense ontology containing N ′ concepts (C ), such that C ⊆ C . For this work, we used WordNet as a commonsense ontology.

Our objective is to build a multimedia ontology, consisting of a set of andC ′ could be probably the empty set), dedicated to this specific annotation problem, i.e. dependent on the initial annotation vocabulary. This ontology should not only incorporate the subsumption relationships between the different concepts, but also richer semantic relations, such as contextual and spatial relationships. Figure 4.1 illustrates the architecture of our approach for building multimedia ontologies.

|C| + |C ′ | concepts (s.t. C ∪ C ′ ⊆ C ,
As introduced in Section 4.2, current knowledge-driven approaches for semantic image annotation have many limitations. Firstly, they were limited to propose either taxonomic ontologies or descriptive ontologies. Secondly, most of them are based on either a conceptual specification or on a visual one, which make them unsuitable for image annotation. Furthermore, most of these approaches have used a handmade built ontology, which raises the problem of scalability of the building of these knowledge structures.

In order to answer these specific limitations, we propose in this work to build our ontology as a knowledge base that contains explicit and structured knowledge about image context. So that the structure of our ontology is representative of the image semantics, we propose to use a Semantico-Visual specification (which incorporates the visual and conceptual semantics of image concepts) for designing our ontology.

In addition, we propose to build our multimedia ontology in an automatic manner, and based on mining image databases to gather valuable information about image context. Thus, the depicted knowledge within our ontology is faithful to image semantics. Finally, the proposed ontology is built using a highly expressive formalism (Fuzzy OWL2-DL), which allows a good interaction with it, i.e. a good querying and reasoning capabilities. Our belief is that such formal ontology will allow to perform reasoning tasks on the image annotation and therefore to achieve an effective decision-making toward that goal.

The design of our ontology, and of a well defined formal ontology in a general way, passes exclusively through the following main steps which will be detailed in the remaining of this chapter:

⋆ Defining the DL formalism of the proposed ontology, i.e. the expressiveness of the ontology.

⋆ Defining the set of axioms and inferences rules allowing to perform the reasoning tasks on the proposed ontology.

⋆ Defining the main concepts of the ontology.

⋆ Defining the RBox, i.e. definition of the key roles (relationships between concepts) and their properties.

⋆ Defining the TBox, i.e. definition of the subsumption hierarchy, and consequently the subsumption relationships between the ontology concepts.

⋆ Defining the ABox, i.e. the instances of concepts and the relations between them with respect to the roles defined in the RBox.

Formalism of our Multimedia Ontology

Preliminaries

The Web Ontology Language (OWL) is the current standard language for creating ontologies. It allows describing a domain in terms of: concepts (or classes), roles (or properties), individuals and axioms. Concepts (C ) are a set of objects, individuals (I ) are instances of concepts in C, roles are binary relationships between individuals in I, whereas axioms describe how these concepts, individuals, roles, etc. should be interpreted. Three sublanguages of OWL can be used: OWL-Full which is the most expressive language but reasoning within it is undecidable, OWL-Lite which has the lowest complexity but fewer constructs, and OWL-DL which has a good balance/tradeoff between expressiveness and reasoning complexity [Bobillo andStraccia, 2009, Bobillo andStraccia, 2011].

In our approach, in order to ensure a high expressiveness with a decidable reasoning for our ontology, we used OWL 2 DL as a language for designing our ontology. Indeed, OWL 2 DL is more expressive than OWL-DL, i.e. includes more axioms. Concretely, we have implemented a framework using the OWL API4 [START_REF] Horridge | The owl api: A java api for owl ontologies[END_REF], which supports OWL 2 since it last version. The reasoning tasks about concepts, roles and individuals are also performed using our framework, which is based on the FaCT++ reasoner and extending it with the axioms illustrated in Table 4.1 to support the Fuzzy Description Logics (Fuzzy DL). Initially, FaCT++ supports the SROIQ(D) logic (i.e. the DL for OWL2 ontology). However, our framework supports the fuzzy logic f -SROIQ(D) thanks to the extension we have made.

Description Logics (DLs) are a family of logics for representing structured knowledge. Fuzzy DLs extend classical DLs by allowing to deal with fuzzy/imprecise concepts [Straccia, 2001]. Indeed, in fuzzy logics a statement is no longer true or false, but is changed in a fuzzy statement signifying that it has a degree of truth α ∈ [0, 1].

Fuzzy Set Preliminaries.

In a formal way, let X be a set of elements. A fuzzy set A over a countable crisp set X is characterized by a membership function µ A : X → [0, 1] (or A(x) ∈ [0, 1]), assigning a membership degree A(x) to each element x in X. A(x) gives an estimation of the belonging of x to A. In fuzzy logics, the degree of membership A(x) is regarded as the degree of truth of the statement "x is A". Accordingly, a concept C is interpreted in fuzzy DL as a fuzzy set, and thus concepts become imprecise. For instance, the statement a : C (a is an instance of concept C) will have a truth-value in [0,1] given by its membership degree denoted C I (a). A fuzzy relation R over two countable crisp sets X and Y is a function

R : X × Y → [0, 1]. R is reflexive iff for all x ∈ X, R(x, x) = 1 holds, while R is symmetric iff for all x, y ∈ X, R(x, y) = R(y, x) holds. R is said functional iff R is a partial function R : X × Y → {0, 1}
such that for each x ∈ X there is unique y ∈ X where R(x, y) is defined.

Expressiveness of our Ontology

As introduced in the previous section, for the concern of providing a highly expressive multimedia ontology with a decidable reasoning, we used the fuzzy DL f -SROIQ(D) for designing our ontology. Based on the work of [Straccia, 2006, Stoilos and[START_REF] Stoilos | [END_REF], we introduce in the following the specific formalism (constructors and axioms) used for defining our multimedia ontology.

The f -SROIQ(D) is a fuzzy extension of the SROIQ(D) DL, which provide both a set of constructors allowing the construction of new concepts and roles. The f -SROIQ(D) includes ALC standard constructors (i.e. negation ¬, conjunction ⊓, disjunction ⊔, full existential quantification ∃, and value restriction ∀) extended with transitive roles (S), complex role axioms (R), nominals (O), inverse roles (I), and qualified number restrictions (Q). (D) indicates support for (fuzzy) concrete domains, i.e. datatype properties, data values or data types.

Fuzzy concrete domain. A fuzzy concrete domain is a pair ∆ D , Φ D , where ∆ D is an interpretation domain and Φ D is the set of fuzzy domain predicates d with a predefined arity n and an interpretation d D : ∆ n D → [0, 1] [Straccia, 2012]. In f -SROIQ(D), concepts (denoted C or D) and roles (R) can be built inductively from atomic concepts (A), atomic roles (R A ), top concept ⊤, bottom concept ⊥, named individuals (o i ), simple roles S, and universal role U . Simple roles S are inductively defined: i) R A is simple if it does not occur on the right side of a Role Inclusion Axioms (RIA), ii) R -is simple if R is, iii) if R occurs on the right side of a RIA, R is simple if, for each w ⊑ R ⊲ α , w = S for a simple role S.

Fuzzy Concepts. Under f -SROIQ(D), a fuzzy concept is defined by the following assertions5 :

C → ⊤ | ⊥ | A | C 1 ⊓ C 2 | C 1 ⊔ C 2 | ¬C | ∃R.C | ∃T.d | ∀R.C | ∀T.d | (≥ m S.C) | (≥ m T.d) | (≤ n S.C) | (≤ n T.d) | {o 1 , . . . , o n } D → d | ¬d
For more details about the semantics of these assertions cf. Table 4.1 -Constructor C1-C16.

Fuzzy KB. A f -SROIQ(D) knowledge base (denoted KB) is a triple (T ,R,A) where T is a fuzzy Terminological Box (TBox ), R is a regular fuzzy Role Box (RBox ), and A is a fuzzy Assertional Box (ABox ) containing statements about individuals. The TBox and RBox contain general knowledge about the domain application.

C Constructor Syntax Semantics Atomic concept A A I (a) ∈ [0, 1] Top ⊤ ⊤ I (a) = 1 Bottom ⊥ ⊥ I (a) = 0 Conjunction C ⊓ D (C ⊓ D) I (a) = C I (a) ⊗ D I (a) Disjunction C ⊔ D (C ⊔ D) I (a) = C I (a) ⊕ D I (a) Negation ¬C (¬C) I (a) = ⊖C I (a) Existential restriction ∃R.C (∃R.C) I (a) = sup b∈∆ I {R I (a, b) ⊗ C I (b)} ∃T.d (∃T.d) I (a) = sup v∈∆ D {T I (a, v) ⊗ d D (v)} Universal restriction ∀R.C (∀R.C) I (a) = inf b∈∆ I {R I (a, b) → C I (b)} ∀T.d (∀T.d) I (a) = inf v∈∆ D {T I (a, v) → d D (v)} At-least restriction ≥ m S.C (≥ m S.C) I (a) = sup b1,...bm∈∆ I (⊗ m i=1 {S I (a, b i ) ⊗ C I (b i )}) ⊗ (⊗ j<k {b j = b k }) ≥ m T.d (≥ m T.d) I (a) = sup v1,...vm∈∆ D (⊗ m i=1 {T I (a, v i ) ⊗ d D (v i )}) ⊗ (⊗ j<k {v j = v k }) At-most restriction ≤ n S.C (≤ n S.C) I (a) = inf b1,...b n+1 ∈∆ I (⊗ n+1 i=1 {S I (a , b i ) ⊗ C I (b i )}) → (⊕ j<k {b j = b k }) ≤ n T.d (≤ n T.d) I (a) = inf v1,...v n+1 ∈∆ D (⊗ n+1 i=1 {T I (a, v i ) ⊗ d D (v i )}) → (⊕ j<k {v j = v k }) Local reflexivity ∃S.Self (∃S.Self ) I (a) = S I (a, a) Fuzzy nominals m i=1 {(o i , α i )} {(o 1 , α 1 ), . . . , (o m , α m )} I (a) = sup i|a∈{o I i } α i Atomic role R A R I A (a, b) ∈ [0, 1] Universal role U U I (a, b) = 1 Inverse role R - ∀a, b ∈ ∆ I , (R -) I (a, b) = R I (b, a) Concrete role T T I (a, v) ∈ [0, 1] A Axiom Syntax Semantics Concept assertion a : C ⊲⊳ α C I (a I ) ⊲⊳ α Role assertion (a : b) : R ⊲⊳ α R I (a I , b I ) ⊲⊳ α Concrete role assertion (a : b) : T ⊲⊳ α T I (a I , v D ) ⊲⊳ α Equality assertion a = b a I = b I Inequality assertion a = b a I = b I Subsumption C ⊑ D ⊲ α inf a∈∆ I {C I (a) → D I (a)} ⊲ α Concept definition C ≡ D ∀a ∈ ∆ I , C I (a) = D I (a) Role inclusion axioms R 1 R 2 • • • R n ⊑ R ⊲ α sup b 1 ...b n+1 ∈∆ I [R I 1 (b 1 , b 2 ), . . . , R I n (b n , b n+1 )] → R I (b 1 , b n+1 ) ⊲ α Disjoint role dis(S 1 , S 2 ) ∀a, b ∈ ∆ I , S I 1 (a, b) ⊗ S I 2 (a, b) = 0 Symmetric role sym(R) ∀a, b ∈ ∆ I , R I (a, b) = R I (b, a) Reflexive role ref (R) ∀a ∈ ∆ I , R I (a, a) = 1 Transitive role trans(R) ∀a, b ∈ ∆ I , R I (a, b) ≥ sup c∈∆ I R I (a, c) ⊗ R I (c
∆ I are abstract individuals, v ∈ ∆ D is a concrete individual, n, m are natural numbers (n ≥ 0, m > 0), α ∈ [0, 1] is the truth degree of a statement, ⊲∈ {>, ≥}, ⊲⊳∈ {>, <, ≥, ≤}.
Chapter 4. Building Fuzzy Multimedia Ontologies for Image Annotation Fuzzy ABox.

The fuzzy ABox consists of a finite set of fuzzy concept and fuzzy role assertion axioms. Typically, these assertions include: concept assertion ( a : C ⊲⊳ α )6 , role assertion ( (a : b) : R ⊲⊳ α ), concrete role assertion ( (a : b) : T ⊲⊳ α ), equality assertion ( a = b ), and inequality assertion ( a = b ). The semantics of these assertions is defined in Table 4.1 -Axioms A1-A5.

Fuzzy TBox.

The fuzzy TBox is a finite set of General Concept Inclusions (GCIs) constrained with a truth-value and of the form C ⊑ D ⊲ α between two f -SROIQ(D) concepts C and D. Concept equivalence C ≡ D can be captured by two inclusions C ⊑ D and D ⊑ C -(cf. Table 4.1 -Axioms A6-A7).

Fuzzy RBox.

The fuzzy RBox consists of a finite set of role axioms which are illustrated in Table 4.1 -Axioms A8-A14. These include: role inclusion axioms, disjoint role, symmetric role, reflexive role, transitive role, irreflexive role, and asymmetric role.

Owing to the specific motivations discussed in Section 4.4.3, we have defined the fuzzy operators used in Table 4.1 as follows:

1. product t-norm: a ⊗ b = a * b. 2. product t-conorm: a ⊕ b = a + b -a * b.
3. Łukasiewicz negation: ⊖α = 1α.

Gödel implication (for GCIs and RIAs

): α → β = 1 if α ≤ β, β otherwise.

KD implication (for other constructors

): α → β = max(1 -α, β).
The Semantics of f -SROIQ(D) is defined in terms of fuzzy interpretations [Straccia, 2001].

Fuzzy interpretation.

A fuzzy interpretation is a pair I = (∆ I , • I ) where ∆ I is a non-empty set of objects (called the domain) and • I is a fuzzy interpretation function, which maps:

• a concept name C onto a function C I : ∆ I → [0, 1], • a role name R onto a function R I : ∆ I × ∆ I → [0, 1],
• an individual name a onto an element a I ∈ ∆ I ,

• a concrete individual v onto an element v D ∈ ∆ D , • a concrete role T onto a function T I : ∆ I × ∆ D → [0, 1],
• a concrete feature t onto a partial function t

I : ∆ I × ∆ D → {0, 1} Satisfiability.
Finally, a fuzzy interpretation I satisfies an f -SROIQ(D) knowledge base KB = (T ,R,A) if it satisfies all axioms of T , R and A. I is then called a model of KB, written: I |= KB.

Ontology-Based Reasoning

General automatic reasoning tasks on ontologies include concept consistency, concept subsumption to build inferred concepts taxonomy, instance classification and retrieval, parent and children concept determination, and answering queries over ontology classes and instances [START_REF] Baader | The Description Logic Handbook: Theory, Implementation, and Applications[END_REF]. These reasoning tasks are induced by inferring logical consequences from a set of asserted facts or axioms.

Logical consequence. A fuzzy axiom τ is a logical consequence of a knowledge base KB, denoted KB |= τ if f every witnessed model of KB satisfies τ .

Given a KB and an axiom τ of the form C ⊑ D , a : C or (a, b) : R , it is possible to compute the best explanation of a given statement (probably, about an image) as the τ 's best entailment degree (bed). The bed problem can be solved by determining the greatest lower bound (glb) [Straccia, 2001].

Greatest lower bound. The greatest lower bound of τ with respect to a fuzzy KB is:

glb(KB, τ ) = sup{n | KB |= τ ≥ n }, where sup ∅ = 0 (4.1)
Example 1 (Greatest lower bound). For instance, given KB = { (a, b) : R, 0.5 , b : C, 0.9 }, the greatest lower bound that a is an instance of a concept which is in relation R with concept C is:

glb(KB, a : ∃R.C) = 0.45
Best satisfiability degree. The best satisfiability degree (bsd) of a concept C with respect to a fuzzy KB is defined as:

bsd(KB, C) = sup I|=KB sup x∈∆ I {C I (x)} (4.2)
The best satisfiability degree consists in determining the maximal degree of truth that the concept C may have over all individuals x ∈ ∆ I , among all models I of the KB.

According to our specific context, and in order to achieve an efficient reasoning (and subsequently an accurate decision) on the best explanation of a given image, it is important to compute a membership degree for this explanation which reflects the likelihood of conjunction of all independent events composing it. The product logic makes possible to dispose of this desirable property for the t-norm. This assumption has motivated our choice for the product t-norm and the product t-conorm as fuzzy operators of our ontology -cf. Section 4.4.2. For instance, let us consider the following example where we want to compute the membership of an image i to the class BeachImage:

Example 2 (Product semantics and Zadeh semantics). KB = { i : Image, 1 , i : ∃depicts.Sea, α 1 , i : ∃depicts.Sand, α 2 , i : ∃depicts.Sky, α 3 } BeachImage ≡ Image ⊓ ∃depicts.Sea ⊓ ∃depicts.Sand ⊓ ∃depicts.Sky

Building the Semantic Hierarchy and Definition of the TBox

The subsumption hierarchy (respectively the subsumption relationships) is a fundamental component of ontologies. It acts as a backbone of the produced ontology, where the subsumption roles allow defining the inheritance of properties from the parent (subsuming) concepts to the child (subsumed) concepts. Thus, any statement that is true (with an α degree) for a parent concept is also necessarily true (with at least an α degree) for all of its subsumed (child) concepts. Consequently, these subsumption relationships allow defining the Terminological Box of ontologies.

In order to define the TBox of our multimedia ontology, we propose to build a subsumption hierarchy where leaf nodes are the initial concepts of the considered dataset (c j ∈ C), and mid-level nodes are the concepts discovered by a variant of the approach proposed in Chapter 3. Indeed, we propose in this work to automatically build the semantic hierarchy using a Semantico-Visual similarity computed between concepts. The proposed Semantico-Visual similarity incorporates: i) a visual similarity which represents the visual affinity between concepts, and ii) a conceptual similarity which defines a relatedness measure between target concepts based on their definitions in WordNet.

For more information about how to compute these (visual and conceptual) similarities, the reader is suggested to refer to Section 3.3.2 and 3.3.3.

Afterwards, the building of the subsumption hierarchy is bottom-up, and is based on the set of heuristic rules defined in Section 3.4 in order to link together the concepts that are semantically most related, with respect to the previously computed similarity. Consequently, the building of the subsumption hierarchy consists in identifying |C ′ | new concepts that link all the concepts of C in a hierarchical structure that best represents image semantics.

Once the subsumption hierarchy is built, and therefore we dispose of the set of concepts (C ∪ C ′ ) of our multimedia ontology, the subsumption relationships between all these concepts are added to our ontology according the hierarchy structure. This is achieved automatically using the axiom A6 illustrated in Table 4.1.

Example 3 (Definition of the TBox ).

Sof a ⊑ Seat ≥ 1 Chair ⊑ Seat ≥ 1 Seat ⊑ F urniture ≥ 1 • • • 4.5.

Definition of the ABox

Subsequent to the building of the semantic hierarchy that will be used as the backbone of our ontology, information about the context of images is added to our ontology in order to design a much representative knowledge base about image semantics. This information, mainly consisting of contextual and spatial relationships between image concepts, will form the ABox of our ontology and will serve for reasoning about image annotation. Moreover, our intent is to design a fuzzy multimedia ontology and to represent the uncertainty about the concepts relationships. consequently, we introduce in the following how to compute the truth-degree for each of the proposed roles (relationships).

Contextual Relationships

As previously introduced in Section 3.3.4, contextual information is of great interest to help understanding the image semantics. A simple form of contextual information is the co-occurrence frequency of a pair of concepts. For example, it is intuitively clear that if two concepts are similar or related, it is likely that their role in the world will be similar, and thus their context of occurrence will be equivalent (i.e. they tend to occur in similar contexts, for some definition of context). For instance, a photo containing "Television" and "Sofa" depicts usually a "Living-room" scene. Nevertheless, as aforementioned contextual similarity is a 'corpus-dependent' measure, i.e. depends on the concepts distribution in the dataset. It is therefore important to normalize the measures based on contextual information.

In our approach, we defined three contextual relations that we estimated important in order to reason about image annotations. These are: CON = {"has-Frequency", "hasAppearedWith", "isAnnotatedBy"}. However, nothing prevents the enrichment of our ontology with other contextual relationships in the future. The proposed relations (∈ CON ) are detailed bellow.

Let us consider an image database DB, where:

• L is the number of images in the database,

• N is the size of the annotation vocabulary,

• n i is the number of images annotated by c i (occurrence frequency of c i ), and

• n ij the number of images co-annotated by c i et c j .

Our objective is to estimate P (c i ) as the probability of occurrence of a given concept c i (and respectively P (c i , c j ) as the joint probability of c i and c j ) in DB. These probabilities can be easily estimated by:

P (c i ) = n i L (4.3) P (c i , c j ) = n ij L (4.4)
Based on these probabilities, we define the concept frequency relationship as the concrete feature: hasF requency : ∆ I * ∆ D → {0, 1}, where ∆ I = C and ∆ D = [0, 1] are the interpretation domains. This concrete feature associates to each concept c i ∈ C a fuzzy degree corresponding to its occurrence frequency in DB:

µ hasFrequency(c i ) = P (c i ) (4.5)
We also define the contextual relationship 'hasAppearedWith' as the fuzzy role hasAppearedW ith : ∆ I * ∆ I → [0, 1], where ∆ I = C. The membership degree of this relationship is computed using the Normalized Pointwise Mutual Information (NPMI). To this purpose, the Pointwise Mutual Information ρ(c i , c j ) is firstly computed for all pairs of concept c i , c j ∈ C as follows:

ρ(c i , c j ) = log P (c i , c j ) P (c i )P (c j ) = log L * n ij n i * n j (4.6)
As previously introduced in Section 3.3.4, ρ(c i , c j ) quantifies the amount of information shared between the two concepts c i and c j . In this work, we only want to estimate the positive correlation between two given concepts and therefore we set the negative values of ρ(c i , c j ) to 0. Moreover, in order to normalize into [0,1] the membership degree of the fuzzy role 'hasAppearedWith' between two concepts c i and c j , we compute it in our approach as:

µ hasAppearedWith(c i ,c j ) = ρ(c i , c j ) -log[max(P (c i ), P (c j ))] (4.7)
Example 4 (Contextual Relationship: 'hasAppearedWith').

a : T v_M onitor ≥ 1 b : Sof a ≥ 1 (a : b) : hasAppearedW ith ≥ 0.26 • • •
Finally, we define the fuzzy role 'isAnnotatedBy' as a relationship between instances of concepts "Image" and "Annotation", i.e. isAnnotatedBy : ∆ I * ∆ I → [0, 1], where ∆ I = {Image, Annotation}. This relationship is intended to represent the probability of finding an image in DB annotated by a set of concepts (Annotation j = c 1 , c 2 , • • • , c Λ ), or inversely, the likeliness that a given annotation 'Annotation j ' is associated with an image i i ∈ I. To this end, all possible annotations in DB are extracted and are added to our ontology as subconcepts of concept "Annotation". For instance, we illustrated in Example 5 some inputs of the added assertions to our ABox :

The confidence value of this relationship is computed as follows:

µ isAnnotatedBy(Image 1 ,Annotation j ) = n Annotation j L (4.8)
where

Annotation j = c 1 , c 2 , • • • , c
Λ is a textual annotation used for annotating a set of images in DB, n Annotation j is the number of images annotated by Annotation j , and L = |I| is the total number of images in DB.

Example 5 (Contextual Relationship: 'isAnnotatedBy').

Annotation 1 ≡ Bus ⊓ M otorbike ⊓ P erson Annotation 1 ⊑ Annotation ≥ 1 Annotation 2 ≡ Aeroplane ⊓ Car ⊓ P erson Annotation 2 ⊑ Annotation ≥ 1 Annotation 3 ≡ Dining_T able ⊓ Chair ⊓ Bottle ⊓ Dog Annotation 3 ⊑ Annotation ≥ 1 a : Image ≥ 1 b : Annotation 1 ≥ 1 (a : b) : isAnnotatedBy ≥ 0.004823 • • • 4.5.4.

Spatial Relationships

Spatial information is a valuable source for the understanding of image semantics. The spatial arrangement of objects provides an important information for the recognition and interpretation tasks, and allows to solve the ambiguity between objects having a similar appearance [Bloch, 2005]. For instance, using object detectors if one have detected in an image that "Sky" has appeared bellow "Sea", we can easily fix this prediction using spatial information because any well defined knowledge base (KB) would allow to detect and correct this inconsistency.

In our approach, eight spatial relationships are used in order to define the directional positions and distances between concepts. The directional relationships are defined as follows: DIR = {"hasAppearedAbove", "hasAppearedBelow", "has-AppearedLeftOf", "hasAppearedRightOf", "hasAppearedAlignedWith"}, such as ∀X ∈ DIR, X :

∆ I * ∆ I → [0, 1], with ∆ I = C.
The relationships in DIR are derived from the following primitives 'left', 'right', 'above', 'below ' and 'aligned ', which are computed according to the angle between the segment joining two points 'a' and 'b' (where 'a' and 'b' are the centroids of two given objects in a given image) and the x-axis of the image -cf. Figure 4.3. This angle, denoted θ(a, b), takes values in [-π, π] which constitutes the domain of definition of these primitives. They are then computed using cos 2 θ and sin 2 θ, and are functions from [-π, π] into {0, 1}. Thus, any of the previous primitives can be computed by an angle α with the x-axis as illustrated in Figure 4.4.

Regarding the primitive 'aligned ', it takes 1 when θ ∈ [-π/6, π/6] ∪ [5π/6, -5π/6] and 0 otherwise. A comprehensive survey about spatial relationships for image processing can be found in [Bloch, 1999, Bloch, 2005]. The confidence value of a given directional relationship is finally computed as follows:

µ X (c i ,c j ) = ♯ of instances where X (c i , c j ) n ij (4.9)
where c i , c j ∈ C, and X is a directional relationship, i.e. X ∈ DIR.

In our approach, the distance relationships are defined as DIS = {"hasAppeared-CloseTo", "hasAppearedFarFrom"}, such as ∀χ ∈ DIS, χ : ∆ I * ∆ I → [0, 1], with ∆ I = C. These distance relationships are computed according to the Euclidean distance on the considered objects.

To this purpose, let us consider, in a given image, two objects O and P defined by their centroids (x 1 , y 1 ) and (x 2 , y 2 ), and their bounding box (O xmin , O xmax , O ymin , O ymax ) and (P xmin , P xmax , P ymin , P ymax ). We define then the Chapter 4. Building Fuzzy Multimedia Ontologies for Image Annotation following primitives:

distance(O, P ) = (x 1 -x 2 ) 2 + (y 1 -y 2 ) 2 (4.10) size(O) = (O xmax -O xmin ) 2 + (O ymax -O ymin ) 2 (4.11) close(O, P ) = 1 if distance(O, P ) < 2(size(O) + size(P )) 0 otherwise (4.12) f arf rom(O, P ) = 1 if distance(O, P ) ≥ 2(size(O) + size(P )) 0 otherwise (4.13)
Using the previous primitives, the distance relationships can easily be computed by the following equation:

µ χ(c i ,c j ) = ♯ of instances where χ(c i , c j ) n ij (4.14)
where c i , c j ∈ C, and χ is a distance relationship, i.e. χ ∈ DIS.

Experiments

In order to illustrate our approach, we propose in the following to test our proposal on the Pascal VOC dataset. As part of this chapter, we only present the built multimedia ontology on the considered dataset. An empirical evaluation of the effectiveness of the proposed multimedia ontology within an image annotation framework is proposed in Chapter 6. Figure 4.5 illustrates the built semantic hierarchy on the Pascal VOC'2010 dataset. Indeed, in Section 4.5.3 we proposed to build a semantic hierarchy using a variant (based on the visual and conceptual similarities) of our approach introduced in Chapter 3. This Semantic hierarchy allowed to define the subsumption relationships between image concepts. We can observe that the produced hierarchy is a N-ary tree like-structure, where leaf nodes are the concepts in C. Mid-level concepts are automatically recovered from WordNet based on our method described in Section 3.4. We can also observe that the connected concepts share strong visual and semantic similarity, which justifies the choice of this method in our approach. We therefore concur with the assumption that a suitable semantic hierarchy for representing image semantics should incorporate visual and conceptual (semantic) modalities during the building process -cf. Assumption 1. Figure 4.6 illustrates the built ontology on Pascal VOC dataset and the used roles for defining the concepts relationships. Full arrows represent the subsumption relationships between the ontology concepts. Dashed arrows represent the fuzzy roles used for defining the contextual and spatial relationships between concepts. For the clarity of the illustration we restricted the Annotation j concept number to 4 and we did not displayed the instances (individuals).

Discussion and Usage Scenarios

The proposed methodology for building multimedia ontologies is original, and is useful for the modeling and the understanding of image semantics, i.e. identify and formalize the semantic relationships between image concepts. Indeed, the representation of our concepts and their semantic relationships are automatically extracted from image datasets, which provides an efficient modeling of image semantics and allows for extending our ontology at any time by mining new image datasets. Efficient modeling of image semantics means here: less sensitive to the subjectivity of human perception and less sensitive to the semantic gap.

Regarding the usefulness of our multimedia ontology for computer vision tasks, we propose in the following some usage scenarios. Let us consider an expressive amount of multimedia content, it is possible to extend our approach in order to model (or to learn), in a simple way, complex concepts by the mining of this multimedia content. For instance, let us suppose that we dispose of a well annotated image database and which is representative of the scenes from real life. It is obvious that when we find a 'Computer monitor ' in a given image, it is very likely to find a 'Mouse' and a 'Keyboard ', and thus, these concepts will share a high co-occurrence confidence score. One can therefore use our proposed approach to define complex concepts, which are not previously included in the annotation vocabulary, based on the fuzzy role 'hasAppearedWith' and the co-occurrence confidence score. Specifically, if the context of appearance of a set of concepts is sufficiently high (greater than a predefined threshold), therefore using their definition in WordNet we can find the common concept that connects them, and consequently define automatically this (complex) concept. To illustrate this proposal, here are some examples of defined concepts by the above described method:

Example 7 (Scenario 1: Defining complex concepts).

Sitting_room ≡ Sof a ⊓ T able ⊓ T elevision Beach ≡ Sea ⊓ Sand ⊓ Sky ⊓ ∃hasAppearedAbove(Sea, Sand)⊓ ∃hasAppearedBellow(Sea, Sky) Computer ≡ Screen ⊓ Keyboard ⊓ M ouse ⊓ ∃hasAppearedAbove(Screen, Keyboard) ⊓ ∃hasAppearedRightOf (M ouse, Keyboard)
Another usage scenario consists in a knowledge-driven approach for image annotation using object detection. Indeed, as introduced in Chapter 2 -Section 2.4.1, one popular technique for identifying and localizing objects in an image is by the use of sliding-window object detection. It consists in defining a fixed-size rectangular window and applying a classifier to the sub-image defined by the window. The classifier extracts image features from within the window and returns the probability that the window bounds a particular object. The process is repeated on successively scaled copies of the image so that objects can be detected at any size. So, let us suppose that one dispose of a multimedia database annotated with an average of 3000 concepts, as for instance the SUN database [START_REF] Xiao | SUN database: Large-scale scene recognition from abbey to zoo[END_REF]. Thus, we will dispose of 3000 object detectors that will be performed on all images of the database and at different scales, which is computationally very expensive. The complexity of this task can be decreased significantly by the use of our multimedia ontology and the scenario defined in the following.

Example 8 (Scenario 2: A knowledge-driven approach for object detection.). Given a previously unseen image:

1. Apply progressively the detectors of the most frequent concepts (w.r.t 'hasFrequency' concrete feature) in KB, until a first concept c i ∈ C is detected.

2. Query the ontology (KB) for the most likely concept (c j ∈ C) to appear with c i and its spatial location.

3. Apply the detector for c j by delimiting the retrieving space according to the predicted spatial location. If it fails go to 2, else go to 4.

4. Query the ontology for candidate textual annotations with respect to the already detected concepts and their locations.

5. According to the decreasing confidence scores of these annotations, apply the detectors for the concepts of the selected annotation. If all concepts of the considered annotation are detected go to 6, else go to 4 (to select another annotation consistent w.r.t the already detected concepts).

6. Stop the processing and return the object detection result (i.e., the set of detected concepts and their spatial location) for the input image.

This usage scenario allows reducing significantly the complexity of the object detection process. In order to perform image annotation, it requires performing much less detectors than the classical approach and targeting the detection zone according to the already detected concepts. Thus, it is clear that the proposed ontology is useful to effectively manage image processing tasks, and to efficiently perform image annotation. These usage scenarios will be addressed in our future work.

Conclusion

In this chapter, we proposed a new approach to automatically build a fuzzy multimedia ontology dedicated to image annotation and interpretation. In our approach, visual and conceptual information are used to build a semantic hierarchy faithful to image semantics, and which will serves as a backbone of our ontology. The ontology is thereafter enriched with contextual and spatial information. Fuzzy description logics are used as a formalism to represent our ontology and to deal with the uncertainty and the imprecision of concept relationships. Some usage scenarios are then proposed to show the usefulness of the proposed ontology.

For illustrating the efficiency of the proposed multimedia ontology in an experimental way, we propose in Chapter 6 a new method for image annotation. Our approach is based on the hierarchical image classification and a multi-stage reasoning framework (based on our multimedia ontology) for reasoning about the consistency of the produced annotation. An empirical evaluation of our approach on the Pascal VOC'2010 dataset is also proposed.

Introduction

As introduced in Chapter 3, semantic hierarchies have been introduced recently to improve image annotation. They were mostly used as a framework for hierarchical image classification, and thus to improve the classifiers accuracy and to reduce the complexity of managing large scale data.

This chapter aims primarily to experimentally validate the proposed approach in Chapter 3, which allows building semantic hierarchies dedicated to image annotation. We propose therefore to investigate the contribution of semantic hierarchies, and specifically our proposed approach for building hierarchies, for hierarchical image classification. Thus, we propose first a method based on the hierarchy structure to train efficiently hierarchical classifiers. Our method, named One-Versus-Opposite-Nodes, allows decomposing the problem into several independent tasks and therefore scales well with large databases. We also propose two methods for computing a hierarchical decision function that serves to annotate previously unseen images. The former is performed by a top-down classifiers voting, while the second is based on a bottom-up score fusion. The experiments on Pascal VOC'2010 dataset showed that our methods improve well the accuracy of image annotation.

The rest of this chapter is structured as follows. Section 5.2 presents the motivations of our proposal. In Section 5.3, we present an overview of the proposed approach for hierarchical image classification. Section 5.4 introduces the proposed method for training the hierarchical classifiers based on the semantic hierarchy structure. In Section 5.5, we present the proposed hierarchical decision functions for computing the belonging of a given test image to the classes from the annotation vocabulary. Section 5.6 reports our experimental results on Pascal VOC'2010 dataset. The chapter is concluded in Section 5.7.

Context and Motivations

In the last decade, many approaches have considered image annotation as an image classification problem and very often as a multi-class classification problem. As introduced by [Marszalek andSchmid, 2008, Cevikalp, 2010], multi-class classification problem is often handled by the combination of binary SVM classifiers. The traditional combination strategies, which do not exploit any hierarchical structure, are the One-Versus-All (OVA) strategy (through competition) and the One-Versus-One (OVO) strategy (through voting). However, these strategies do not scale well with a large number of classes, since they only offer linear (OVA) or square (OVO) complexity depending on the class number [Marszalek andSchmid, 2008, Bengio et al., 2010].

As discussed in [START_REF] Deng | What does classifying more than 10,000 image categories tell us?[END_REF], to cope with a large semantic space, i.e. a large number of concept categories, the classification process can be improved by the use of semantic or visual hierarchies. This assumption has motivated several recent work [START_REF] Marszalek | Semantic hierarchies for visual object recognition[END_REF], Griffin and Perona, 2008, Fan et al., 2007, Fan et al., 2008a, Cevikalp, 2010, Li et al., 2010]. These approaches can be classified into tops-down methods or bottom up methods according to the way the hierarchy is built. In the top-down approach, the class hierarchy is built by the recursive partitioning of the set of classes [START_REF] Griffin | Learning and using taxonomies for fast visual categorization[END_REF], Marszalek and Schmid, 2008, Cevikalp, 2010, Gao and Koller, 2011]. In the bottomup approach, the class hierarchy is built by the agglomerative clustering of the classes [START_REF] Marszalek | Semantic hierarchies for visual object recognition[END_REF], Fan et al., 2008a, Li et al., 2010, Bengio et al., 2010, Deng et al., 2011b]. These hierarchies are thereafter combined with a set of binary classifiers in order to reduce the complexity of the classification problem. Indeed, this usually results in a logarithmic complexity, which is of interest, especially when dealing with a large number of classes. We refer to these as hierarchical image classification approaches.

Recently, two main directions have been explored for computing a decision function for hierarchical image classification: i) using Decision Directed Acyclic Graphs (DDAG) [START_REF] Platt | Large margin DAG for multiclass classification[END_REF], Marszalek and Schmid, 2008, Gao and Koller, 2011], and ii) using Binary Hierarchical Decision Trees (BHDT ) [START_REF] Marszalek | Semantic hierarchies for visual object recognition[END_REF], Griffin and Perona, 2008, Cevikalp, 2010].

Given, C = c 1 , c 2 , • • • , c N the annotation vocabulary of the image database, where N = |C|, the DDAG based approaches train N (N -1)/2 binary classifiers and use a Directed Acyclic Graph (DAG) to decide about the belonging of an image i to a class c j ∈ C. These methods allow at each node in a distance d from the rooted DAG to eliminate d candidate classes from C, resulting in a N -1 decision nodes to be evaluated for the labeling of a test sample -cf. Figure 5.1(a).

On the other side, BHDT based approaches build and use hierarchies as a binary tree, i.e. data are divided hierarchically into two subsets until each subset consists of only one class. Data partition is often achieved using a clustering algorithm. Thus, one SVM is trained for each node of the tree, resulting in a log 2 N SVM runs to label a test sample -cf. Figure 5.1(b). BHDT based approaches target to optimize the efficiency of SVM classifiers by reducing the unnecessary comparisons while maintaining a high classification accuracy [Cevikalp, 2010].

The aforementioned approaches (BHDT and DDAG based methods) focus on optimizing the classification accuracy and do not model in anyway the image semantics. Although they allow achieving a higher image classification accuracy compared to traditional approaches, they constrain the used hierarchies to binary structures, resulting in a significant deadlock when the concept number is large. For instance, the method of [START_REF] Marszalek | Semantic hierarchies for visual object recognition[END_REF] is in a deadlock when the concept number exceeds 34, since their method requires N -1 intermediate concepts extracted from [START_REF] Platt | Large margin DAG for multiclass classification[END_REF]. 5.1(b) illustrates a simple BHDT with four classes: {1,2,3,4} [START_REF] Griffin | Learning and using taxonomies for fast visual categorization[END_REF].

WordNet according to the hypernymy relationship (N being the number of initial concepts), and knowing that WordNet depth is limited to 17 levels. Moreover, the image semantics does not correspond necessarily to binary items, and many concepts do not have binary opposites1 . For example, animal or plant species have no binary opposites. Also, data is by nature more complex than binary items. For instance, animals can be classified as 'Herbivores', 'Carnivores' or also 'Omnivores' (both herbivore and carnivore). Consequently, binary structures are not always faithful to data semantics and can lead to inconsistencies in representing knowledge about images or a given domain.

Some other approaches have proposed to use n-ary tree-based models to address the large-scale image classification problem [START_REF] Zweig | Exploiting object hierarchy: Combining models from different category levels[END_REF], Bengio et al., 2010, Deng et al., 2011b]. In particular, label embedding trees proposed by [START_REF] Bengio | Label embedding trees for large multi-class tasks[END_REF] have been shown to achieve high performance on the image classification problem (outperforms other tree-based or embedding approaches). Nevertheless, the learning of the tree structure is based on the confusion matrix and involves the training of OVA classifiers for all of the N classes of the annotation vocabulary. Moreover, their method may lead to an unbalanced tree structure. In order to overcome these problems, [Deng et al., 2011b] have proposed some improvements to the original method of [START_REF] Bengio | Label embedding trees for large multi-class tasks[END_REF]. However, in these approaches, the tree structure over the set of classes is motivated by the reduction of the misclassification loss of the tree. Consequently, the resulted tree structure does not explicitly carries any semantic information and thus, does not fall in the main scope of our researches -cf. Section 3.2.2. Hierarchical topic models have also been applied for image classification [START_REF] Sivic | Unsupervised discovery of visual object class hierarchies[END_REF], Li et al., 2010]. But, as stated above, in this chapter we are interested in the SVM based approaches for hierarchical image classification.

Alternative approaches have emerged recently and have proposed the use of semantic relationships between concepts for the building or the use of hierarchies [START_REF] Fan | Hierarchical classification for automatic image annotation[END_REF], Fan et al., 2008a, Torralba et al., 2008, Maillot and Thonnat, 2008, Deng et al., 2009] 2 . For instance, [Fan et al., 2008a] proposed to incorporate concept ontology and a multi-task learning algorithm for hierarchical concept learning. A hierarchical boosting algorithm is also proposed to learn their ensemble classifiers hierarchically. The labeling of a new sample is obtained by a voting procedure at all levels of the hierarchy, i.e. |C + C ′ | SVM runs are necessary for labeling new images (where C ′ is the intermediate concept nodes in the hierarchy). In [START_REF] Deng | ImageNet: a large-scale hierarchical image database[END_REF], a 'tree-max classifier' based on ImageNet hierarchy is proposed. The authors proposed to train a classifier at each node of the ImageNet tree. Then, a decision function is computed according to a target class and all its child nodes in order to decide whether an image contains a given object class or not.

To sum up, we illustrate in Figure 5.2 and Table 5.1 the different strategies for hierarchical image classification described above, and we give some information about their complexity according to the number of classes. Our concern in this chapter is to survey how to draw profits from semantic hierarchies within the context of image classification frameworks. We therefore propose an approach for hierarchical image classification that scales well with the number of classes, and that is less sensitive to unbalanced data.

Multi-Class

Chapter 5. Hierarchical Image Classification using Semantic Hierarchies

Training Labeling

OVO O(N 2 ) O(N 2 ) OVA O(N ) O(N ) DDAG O(N 2 ) O(N ) BHDT O(N ) O(log 2 N ) N-ary Trees O(N ) O(log 2 N ) Semantic Hierarchies O(N ) O(log 2 N )
Table 5.1: Complexity of existing approaches for image classification.

Overview of our Approach

In the following, we propose an approach for training hierarchical classifiers in an effective manner. Our approach relies on the structure of the semantic hierarchy in order to train a set of classifiers used for image classification. Subsequently, we propose two methods for computing a decision function in order to achieve image classification. The first one is a bottom-up approach, and performs the hierarchical image classification by score fusion. Fusion is achieved by the spreading of scores starting from leaf nodes until reaching the root node. The second is a top-down approach and is performed by classifiers voting. Starting from the root node and according to the classifier votes, the hierarchy is traversed until reaching the leaf nodes. The proposed approach for hierarchical classification is independent from the visual representation of images.

For the building of the semantic hierarchy, we rely on the proposed method in Chapter 3. As aforementioned, the building of the hierarchy is based on a semanticovisual relatedness measure which incorporates: i) visual information, ii) conceptual information, and iii) contextual information about image concepts. Thus, the built hierarchy is suitable for image classification as it combines the different modalities of images. Figure 5.3 illustrates the semantic hierarchy that we will use for the remaining of our experiments. This hierarchy is a N-ary tree like structure where leaf nodes are the initial concepts. Indeed, one of our motivations for using this method is that the semantics of data (and specifically images) is much more complex than binary items.

Proposed Approach for Training Hierarchical Classifiers

Based on the hierarchy structure and the subsumption relationships, we propose in the following to train several classifiers that represent the same concept at different levels of abstraction. These classifiers are consistent with each other since they are linked by subsumption relationships, and thus represent the same information with different levels of details. Therefore, the results of these classifiers can be merged in order to achieve relevant decision on the membership of an image to a given class. For instance, according to the semantic hierarchy illustrated in Figure 5.3, a "Cow" ⊑3 "Bovid" ⊑ "Vertebrate" ⊑ "Organism". Therefore, a good classification system should be able when it recognizes that an image contains a "Cow", to say that it also contains all the concepts that are subsumers of this concept, i.e. "Bovid", "Vertebrate" and "Organism". Reciprocally, if an image does not contains all the subsumers of a given concept, then it should not be annotated by this concept.

Concretely, given a semantic hierarchy, a classifier for each concept node of the hierarchy is trained by performing a One-Versus-Opposite-Nodes (OVON) SVM. Indeed, in order to propose a method that scales well with large image databases, a good strategy would be to decompose the problem into several independent tasks based on the hierarchy structure. Thus, instead of considering all images of a given database for training the classifiers, we only consider the images of a given target concept and its sibling concept nodes. This is similar to cut the target concept node from its upper part of the hierarchy and to focus only on its children and sibling concepts. Therefore, for training the classifier of a given target concept, we took as positive samples all images associated with its children leaf nodes. Negative samples are all images of leaf nodes of its sibling concepts as illustrated in Figure 5.4. So, if an image is annotated by "Cow" it will also serves to train the classifiers for "Bovid", "Vertebrate" and "Organism".

The proposed approach is of interest, especially in our case where we only dispose of the images of leaf concept nodes. In particular, this approach allows recognizing other (abstract) concepts that were not initially included in the annotation vocabulary (allows extending the annotation vocabulary of the image database). Furthermore, decomposing the image classification problem into such a complementary and independent sub-tasks will allow a better scalability, since only few classes will be considered for each iteration and with more balanced data (the ratio of positive/negative samples is closer to 0.5).

Proposed Methods for Computing the Decision Function

5.5.1 Bottom-Up Score Fusion (BUSF)

The Bottom-Up Score Fusion (BUSF) method is based on the fusion of classifier scores in order to achieve the final decision about the belonging of an input image to one or more classes from the annotation vocabulary. The classifiers whose scores will be merged must belong to a given path in the semantic hierarchy. The details of this method are as follows.

Starting from the leaf concept nodes and following the subsumption relationships, we compute the average confidence scores of all paths in the hierarchy. The decision function is then computed according to the sign of this average score, i.e. when the sign is positive for a given leaf concept then the image is annotated by it and all its subsuming concepts. A practical standpoint of this method is that the classification results of these SVM are independent. Therefore, in order to reduce the complexity of this method it is possible to run all SVM classifiers together and to compute the membership degree of an image to all classes (c j ∈ C) in one time. Subsequently, as illustrated in Figure 5.5, the decision function can be computed easily for all leaf concept-nodes according to the hierarchy structure. Thus, the complexity for labeling a given image is ≤ (2N -1).

' 4 C ' C 5 ' C 7 ' C 6 ' C 10 ' C 9 ' C 8 C ' C 1 ' C 2 ' C 3 1 C 2 C 3 C 4 C 5 C 6 C 7 C 8 C 9 C ÷ ÷ ø ö ç ç è ae = å ÎS c v i c v i c l l j x G S sign x f ) ( 1 ) (
i i , f c 5 (x v i ) = 1 3 * (G c 5 (x v i ) + G c ′ 7 (x v i ) + G c ′ 2 (x v i ))
, where G c 5 (x v i ) is the decision function of the classifier associated with c 5 , and so on. If f c 5 (x v i ) > 0, then the image i i is annotated by c 5 , c ′ 7 , c ′ 2 .

Concretely, let x v i be any visual representation of an image i i (a visual feature vector), a classifier is trained for each concept class c j in the hierarchy using our method One-Versus-Opposite-Nodes (OVON). |C| + |C ′ | binary support vector machines are then trained with a decision function:

G c l (x v i ) = k α k y k K(x v k , x v i ) + b (5.1)
where Radial Basis Function (RBF) kernels are used for training our SVM:

G c l (x v i ) is the decision function for the concept c l ∈ {C ∪ C ′ }, K(x v k , x v i )
K(x v k , x v i ) = exp x v k -x v i 2 σ 2 (5.2)
Finally, the hierarchical decision function for calculating the membership degree of an image i i to a concept class c j is computed as follows:

f c j (x v i ) = sign 1 |S| c l ∈S G c l (x v i ) (5.3)
where S is the set of subsumers of c j . G c l (x v i ) is the decision function of the classifier associated with the concept c l .

From a statistical standpoint, the final decision function f c j (x v i ) is computed by achieving n measures of the same event (n = |S| is the hierarchy depth). Thus, the uncertainty about f c j (x v i ) can be calculated as a function of the standard deviation [Vehkalahti, 2008]:

σ fc j (x v i ) = σ √ n (5.4)
where σ is the standard deviation of {f c j (x v i ), c j ∈ S}. Therefore, the final decision function is √ n times more accurate than the one obtained from a single classifier.

Consequently, one should expect better results using this method compared with flat classification.

Top-Down Classifiers Voting (TDCV)

The Top-Down Classifiers Voting (TDCV) method aims at decomposing the image classification problem into several complementary sub-tasks. It consists in building several classifiers that are able to discriminate one class from the others under a given parent node. Thus, to reach the final decision about the membership of an image to a given class, it is essential to descend the hierarchy according to the classifier decisions (votes). If a classifier in a given level of the hierarchy has responded negatively, then the subtree which it is the root is no longer explored. Otherwise, the hierarchy is explored until reaching one or more leaf concepts. Algorithm 2 illustrates the top-down image classification process using this method. Starting from the root node, the decision functions of subsequent level nodes are evaluated. Concept nodes with a positive confidence value are recursively explored until reaching leaf concepts. Several paths in the hierarchy can be simultaneously explored, and thus a test image can be associated to many classes as illustrated in Figure 5.6. For a given test image, if none of the leaf nodes of the explored paths is reached, then the image is annotated with the leaf concept having the higher confidence value.

Algorithm 2: Top-Down Classifiers Voting method Input: SH: a semantic hierarchy, i i : a test image, x v i : a visual representation of the image i i , {G c l (x v )}: the set of decision functions of the classifiers of concepts c l ∈ {C ∪ C ′ }. Result:

A i = {c j | c j ∈ {C ∪ C ′ }, j ≥ 0}: predicted annotation for the image i i . begin A i ← ∅ Ω ← { c j | ∀c j ⊑ ⊤, ∄ c k s.t. c j ⊑ c k , c k ⊑ ⊤} // (i.e., Ω ← immediate children concepts of ⊤) while (|Ω| ≥ 1) do Υ ← ∅ foreach (c l ∈ Ω) do if (G c l (x v i ) > 0) then Υ ← Υ + c l end end if (|Υ| = 0) then Υ ← argmax c l ∈Ω G c l (x v i ) end A i ← A i ∪ Υ Ω ← { c j | ∀c l ∈ Υ, c j ⊑ c l , ∄ c k s.t. c j ⊑ c k , c k ⊑ c l } // (i.e.
, Ω ← immediate children of concepts in Υ) end return A i end * ⊤: stands for the root of the semantic hierarchy (SH).

The TDCV method is efficient in terms of complexity since it requires to train less than 2N -1 classifiers for hierarchical classification, and to evaluate less than log 2 N decision nodes for labeling a test image -cf. Table 5.2. However, TDCV method is sensitive to the initial classification since classifiers at the subsequent levels cannot recover from the misclassification of a test image that may occur in a higher concept level. Thus, the classification error can be propagated to the leaf nodes. Nevertheless, as it will be demonstrated in Figure 5.9, the average precision is strongly high for the nodes in highest levels of the hierarchy, and therefore errors propagation is small. This average precision is high because the concept classes in higher levels of the hierarchy are sufficiently visually different, and therefore it is easier to find a boundary that separates these classes. Moreover, the more one moves up the hierarchy, classes become more balanced, i.e. more positive and negative examples for these classes.

' 4 C ' C 5 ' C 7 ' C 6 ' C 10 ' C 9 ' C 8 C ' C 1 ' C 2 ' C 3 1 C 2 C 3 C 4 C 5 C 6 C 7 C 8 C 9 C 0 ) ( < i c f 0 ) ( > i c f

Experimental Results

Visual Representations of Images

In order to describe image features, we used in this work the Bag-of-Features (BoF) representation (also known as Bag-of-Visual-Word (BoVW)) which is a widely known method [START_REF] Li | A bayesian hierarchical model for learning natural scene categories[END_REF]. The BoF model has shown excellent performances and became one of the most widely used techniques for image classification and object recognition. In our approach, image features are described as follows: Lowe's DoG Detector [Lowe, 1999] is used for detecting a set of salient image regions. A signature of these regions is then computed using SIFT descriptor [Lowe, 1999]. Afterwards, given the collection of detected region from the training set of all categories, we generate a codebook of size K = 1000 by performing a k-means algorithm. Thus, each detected region in an image is mapped to the most similar visual word in the codebook through a KD-Tree. Each image is then represented by a histogram of K visual words, where each bin in the histogram corresponds to the occurrence number of a visual word in that image.

Experimental Setup

Our experiments are performed on Pascal VOC'2010 dataset [START_REF] Everingham | The PASCAL Visual Object Classes Challenge 2010[END_REF]. This dataset contains 4998 training images and 5105 validation images which are used for our experiments. Each image may belong to one or more of the 20 existing classes. Since we do not disposed of the test set, we used the training set for training our concept classifiers and the validation set for evaluating the proposed methods.

The proposed methods for hierarchical image classification are compared to the following methods:

• Hierarchical classification method using the hierarchy illustrated in Figure 5.3 and OVA classifiers.

• Flat classification method described below.

• The H-SVM method proposed by [START_REF] Marszalek | Semantic hierarchies for visual object recognition[END_REF].

• A baseline method.

To perform a fair comparison, we used the same visual representation of images for all of these methods, i.e. Bag-of-Features representation. The flat classification is performed using |C| support vector machines One-Versus-All, where the inputs are the Bag-of-Features representation of images and the outputs are the desired SVM responses for each image (1 or -1). We used a k-fold cross-validation to overcome the unbalanced data problem, taking at each fold as many positive as negative images. Hierarchical image classification with One-Versus-All classifiers is performed by training a set of (|C| + |C ′ |) hierarchical classifiers consistent with the structure of the hierarchy illustrated in Figure 5.3. The baseline method is built by taking the average submission results to VOC'2010 challenge. In the following, the evaluations are performed using the recall/precision curves and Average Precision scores (AP).

Experiments

In Figure 5.7, we compared our method One-Versus-Opposite-Nodes (OVON) for training hierarchical classifiers to the One-Versus-All (OVA) one. OVON performs a better result than the OVA classifiers, with an average precision of 63.25% versus 56.42% for the OVA hierarchical classification. We can also observe that our method OVON performs better on all concept classes, except for the two concepts "Train" and "Bus". As can be seen in Figure 5.3, these two concepts are at the lowest level of the hierarchy and do not share any further sibling concepts. Consequently, both concepts lack of sufficient positive/negative samples for training efficiently their classifiers (90 images for Bus and 113 images for Train). In the other hand, the OVA method has benefited from cross-validation (Repeated random sub-sampling validation). But despite this, we can observe that our method is better when considering all concept classes, since it decomposes the problem of classifiers training according to the hierarchy structure. Thus, the classification problem is simplified since we are looking for a boundary that separates fewer concept classes, which is therefore easier to found. In Figure 5.8, we compare our methods for hierarchical image classification: Bottom-Up Score Fusion (BUSF) and Top-Down Classifiers Voting (TDCV) to i) a baseline method introduced in the beginning of this section, ii) a flat classification method previously introduced, and iii) the H-SVM method of [START_REF] Marszalek | Semantic hierarchies for visual object recognition[END_REF]. Indeed, the authors proposed a method, called Hierarchy of SVM (H-SVM ), based on WordNet for building a semantic hierarchy which is subsequently used for hierarchical image classification. In order to compare our methods to H-SVM, we built a hierarchy of concepts from WordNet using VOC'2010 dataset. The building of the hierarchy, as well as the training of hierarchical classifiers are performed in the same manner described in [START_REF] Marszalek | Semantic hierarchies for visual object recognition[END_REF].

As one can observe in Figure 5.8, our methods achieve a higher average precision than the flat classification method with a gain of +26.8% for the BUSF method and a gain of +16.04% for the TDCV method. Compared to the baseline method our approaches are slightly better. This can be explained by the efficient image features used in the submission of VOC challenge, and the basic one used in our approach. Moreover, we recall that we used only the half of the training set since we did not dispose of the test set used in the challenge. We also included in our evaluations the images marked as difficult, which are ignored in the challenge because they are considered as difficult to recognize. Thus, the obtained results are still promising and could be improved by incorporating more sophisticated image descriptors, as for instance SIFT + HOG, or SIFT + color/texture descriptors. A comparison of our methods for hierar- chical image classification to the H-SVM method of [START_REF] Marszalek | Semantic hierarchies for visual object recognition[END_REF] is also illustrated in Figure 5.8. Our method BUSF achieves a higher average precision than the others with a gain of +8.99% compared to the TDCV method and a gain of +10.76% compared to the method H-SVM. The average precision for these methods was as follows: 60.6% for the BUSF method, 51.61% for the TDCV method, and 49.84% for the method H-SVM. We can therefore conclude that there is a significant improvement in performance with the use of the proposed hierarchy and our methods for hierarchical image classification. Figure 5.9 illustrates the average precision of the different concepts at the intermediate levels of the hierarchy. As shown in this figure, the classifiers accuracy decreases as we go deeper in the hierarchy. This is a logical result since the concept classes in higher levels of the hierarchy are sufficiently visually different, i.e. it is easier to find a boundary that separates these classes. The training data for these concepts is also more balanced. For instance, the ratio of positive/negative samples in VOC'2010 dataset is about 5%. OVON method allows overcoming this problem as it decomposes image classification into several sub-tasks. The ratio of positive/negative samples is 35.6% for OVON, i.e. the classes are more balanced and there is no need for techniques as over-sampling or under-sampling to recover the problem of unbalanced data. In addition, the results reported in Figure 5.9 demonstrate the effectiveness of our method for building semantic hierarchies (presented in Chapter 3) compared to the methods using textual knowledge for building hierarchies. Indeed, Figure 3.1 illustrates a hierarchy inferred from textual knowledge, i.e. built by extracting the relevant graph in WordNet linking all concepts of VOC'2010. The depth of this textual hierarchy is about 17 levels. So, as proved by the results shown in Figure 5.9, one can expect a significant decrease of the hierarchical classification results as we go deeper in the hierarchy. Therefore, the average precision for the leaf nodes (concepts) should be less relevant compared to our proposed method. Finally, we illustrate in Table 5.2 the complexity of our methods for hierarchical image classification using the semantic hierarchy shown in Figure 5.3. Our methods show a better time-complexity for training classifiers compared to the approaches based on Decision Directed Acyclic Graphs, and the ones based on Binary Hierarchical Decision Trees. For image annotation (labeling new images), our method BUSF has showed a higher complexity compared to the others, but in the other hand offers a better accuracy (average precision) compared to these methods. In terms of complexity, our method TDCV is more efficient than all others for the labeling of new images, but does not perform as well as the BUSF method in terms of accuracy.

Conclusion

As previously introduced in this chapter, hierarchical image classification was often considered as a binary classification problem, probably because of the lack of methods for automatically building semantic hierarchies for computer vision tasks. In Chapter 3, we have proposed a new approach for building semantic hierarchies dedicated to image annotation. Therefore, we have proposed in this chapter to investigate the contributions of such (n-ary like structure) hierarchies for hierarchical image classification.

In this chapter, we proposed an approach for hierarchical image classification using semantic hierarchies. Our approach is based on the structure of the semantic hierarchy to efficiently train hierarchical classifiers, i.e. draws benefits from the hierarchy structure to decompose the training process into several independent and complementary sub-tasks. Thus, it allows to gain in efficiency and in complexity as it was shown by the obtained results. We have also proposed two methods for computing a hierarchical decision function serving to annotate previously unseen images. The former is achieved by a top-down classifiers voting, while the second is based on a bottom-up score fusion. Compared to other approaches, our methods have achieved higher accuracy on Pascal VOC'2010 dataset. Specifically, the bottom-up score fusion has shown significant improvement in the classification accuracy, as it allows to reduce the inconsistency about classifier decisions by the fusion of scores of the classifiers belonging to a same path in the hierarchy.

Introduction

In Chapter 4 we have proposed an approach to automatically build a fuzzy multimedia ontology dedicated to image annotation. In our approach, visual and conceptual information were used to build a semantic hierarchy considered as a backbone of our ontology, and used to infer the subsumption relationships between the ontology concepts. Contextual and spatial information were thereafter added to our multimedia ontology in order to achieve a more expressive knowledge base of image semantics. Fuzzy description logics were used as formalism for representing our ontology. The choice of this formalism was motivated by its ability to represent the uncertainty and imprecision of these kinds of information, and also because it enables formal reasoning on concepts.

In this chapter, we propose a knowledge-based multi-stage reasoning framework for image annotation. Our framework uses the proposed multimedia ontology (in Chapter 4) in order to refine the image annotation, and to achieve a more consistent textual description of images. The reasoning tasks are achieved using fuzzy description logics reasoning. Finally, an empirical evaluation of our approach is performed on Pascal VOC'2009 and Pascal VOC'2010 datasets. The obtained results have shown a significant improvement on the average precision results.

The rest of this chapter is structured as follows. In Section 6.2 we introduce the motivations of our approach. Section 6.3 presents a global overview of the proposed framework. In Section 6.4, we introduce the proposed method for reasoning about image annotation in order to achieve a semantically consistent image annotation. Indeed, Subsection 6.4.1 introduces the proposed method for hierarchical image classification, Subsection 6.4.2 introduces the proposed method for reasoning about image annotation using the subsumption relationships, Subsection 6.4.3 introduces the reasoning process using contextual knowledge and Subsection 6.4.4 introduces the reasoning process using spatial information. Section 6.5 presents the experimental results obtained using the proposed framework for image annotation on the test datasets. This chapter is concluded in Section 6.6.

Context and Motivations

Despite significant progress shown by statistical approaches for images annotation, the semantic gap problem is still an open issue for image annotation and interpretation. In this context, several approaches have proposed recently to improve these tasks by the use of explicit knowledge.

A first category of approaches have proposed to use semantic hierarchies for image annotation and classification [START_REF] Marszalek | Semantic hierarchies for visual object recognition[END_REF], Fan et al., 2008a, Deng et al., 2009]. However, most of these approaches use the semantic hierarchies to reduce the complexity of the classification problem, or as a framework for hierarchical image classification. To our knowledge, none has used the semantic structure of these hierarchies (i.e. the inherent semantic relationships of concepts within these hierarchies). Consequently, only a limited improvement in the classification results was shown by these approaches.

Other approaches have proposed to use multimedia ontologies in order to define a standard for the description of low-level multimedia content [START_REF] Simou | A visual descriptor ontology for multimedia reasoning[END_REF], Dasiopoulou et al., 2010], or to use them as a semantic repository for storing knowledge about image domain [START_REF] Simou | Image indexing and retrieval using expressive fuzzy description logics[END_REF], or to use them as a framework for the semantic interpretation of multimedia content and reasoning over the extracted descriptions [START_REF] Hollink | Adding spatial semantics to image annotations[END_REF], Hudelot et al., 2008, Dasiopoulou et al., 2008, Hudelot et al., 2010]. Indeed, ontologies allow modeling many valuable semantic relations between concepts which are missing in the semantic hierarchy models, as for instance the contextual and the spatial relationships. These relationships have been proved to be of prime importance for image annotation [START_REF] Hollink | Adding spatial semantics to image annotations[END_REF], Hudelot et al., 2008, Hudelot et al., 2010, Straccia, 2010]. The reasoning power of ontological models has also been used for semantic image interpretation. In [START_REF] Dasiopoulou | Using Fuzzy DLs to Enhance Semantic Image Analysis[END_REF], Hudelot et al., 2008, Hudelot et al., 2010], formal models of domain application knowledge are used through fuzzy description logics in order to help and to guide the semantic image analysis. [START_REF] Dasiopoulou | Using Fuzzy DLs to Enhance Semantic Image Analysis[END_REF], Dasiopoulou et al., 2009] have proposed an approach for improving image annotation using ontological reasoning on the outputs of statistical concept classifiers. Nevertheless, their reasoning tasks were limited to the inconsistency checking with respect to the TBox, i.e. using only the subsumption relationships between concepts.

However, many problems were not raised by these approaches, or sometimes partially addressed. Firstly, it is widely accepted now that it is not possible to define a standard for the description of low-level multimedia content, since this domain is witnessing a fast evolution and it is in a constant improvement process. Secondly, many of these approaches are limited to provide a formalism allowing to use ontologies as a repository for storing knowledge about multimedia content. However, these approaches have not addressed the problem of reasoning about this knowledge, or when it is done, it is not performed on representative datasets of current real life applications. Consequently, the effectiveness of the stored knowledge and the proposed knowledge models have to be proved. The final category of approaches dealing with reasoning over the extracted descriptions can be qualified as "light". Indeed, these approaches have not addressed the problem of building ontologies dedicated to image annotation/interpretation, and they are usually limited to use hand-built knowledge models. Moreover, they propose reasoning scenarios to deal with the problem of images annotation, but these scenarios are not tested on real data applications, and consequently their effectiveness is not assessed. However, this application domain, i.e. knowledge-driven approaches for image annotation, is still in its infancy and we believe that these approaches will be effectively improved in the near future.

In this dissertation, we proposed a tentative to reduce the above limitations of current approaches, i.e.:

1. Building multimedia ontologies dedicated to image annotation (proposed in Chapter 4). Our approach rely on mining image databases in order to achieve a representative knowledge base of image semantics (and respectively of a given application domain).

2. The use of the reasoning power of ontologies in order to produce a semantically consistent image annotation.

3. To our knowledge, this is the first attempt integrating such a complex reasoning process, i.e using subsumption relationships, and also contextual and spatial relationships.

4. Illustrate and test our approach on a public dataset assumed to be enough representative of real current applications.

To sum up, in Chapters 3 and 4, we have proposed new approaches to build explicit structured knowledge models about image semantics. In this chapter, we propose to use these built knowledge models in a framework for reasoning over the outputs of machine learning algorithms. Our objective is to provide a semantically consistent image annotation while handling the imprecision of machine learning approaches.

Overview of the Proposed Framework

Image classification is one of the most widely used technique for image annotation. It consists in performing several binary SVM classifiers on an input image to find to which classes it belongs to. The annotation of an image depends therefore on the classifier outputs, i.e. an image is annotated by a concept c i ∈ C if the output of the classifier associated to c i is positive. Usually, such a process involves considerable uncertainty because of the errors introduced by the machine learning algorithms. However, this uncertainty can be reduced using reasoning over the produced image annotation. For instance, it is most often easy to compute a confidence score (membership value) for the classification of an image to a given class. Such information is valuable and can be of great importance to improve image classification accuracy. For instance, one can improve image annotation in a post-classification process based on these confidence scores and a knowledge source, such as an ontology which models images context. In that way, this uncertainty is used itself as a knowledge source in order to achieve a better decision-making on the image annotation.

• Thereafter, the consistency of the set of concepts from the candidate annotation is checked with respect to the contextual relationships and our fuzzy DL reasoner. Inconsistent concepts are again removed from the candidate annotation of the input image.

• Finally, the consistency of the candidate annotation is checked with respect to the spatial information, and the final (candidate) annotation is associated with the input image. This final annotation is supposed to be semantically consistent.

Proposed Method: Multi-Stage Reasoning Framework for Image Annotation

In the following, we focus on the proposed knowledge-based framework for image annotation. We will detail the different components of our framework, as they are shown in Figure 6.2.

Hierarchical Image Classification

Reasoning on the annotation using the subsumption relationships

Reasoning on the annotation using contextual information

Input Image

Multimedia Ontology

Fuzzy DL Reasoner

Detected concepts and their confidence scores

A set of concepts

Reasoning on the annotation using spatial information

Relevant Image Annotation

A set of concepts Given:

• DB, an image database consisting of a set of pairs image/textual annotation , i.e.

DB = {[i 1 , A 1 ], [i 2 , A 2 ], • • • , [i L , A L ]}
, where:

-

I = i 1 , i 2 , • • • , i L is the set of all images in DB,
-L is the number of images in the database.

-

C = c 1 , c 2 , • • • , c
N is the annotation vocabulary used for annotating images in I,

-N is the size of the annotation vocabulary.

-A i is a textual annotation consisting of a set of concepts {c j ∈ C, j = 1..n i i } associated with a given image i i ∈ DB.

⇒ DB is only used for training the hierarchical classifiers.

• I ′ , a test set consisting of a set of previously unseen images, i.e.

I ′ = {i ′ 1 , i ′ 2 , • • • , i ′ L ′ }, with L ′
is the number of images in the test set.

• KB = T ,R,A , a fuzzy multimedia knowledge base consisting of:

-A set of concepts C ∪ C ′ , where ∀c i ∈ C, ∃c j ∈ C ′ s.t c i ⊑ c j .
-A set of roles representing the contextual and spatial relationships between all concept (c i , c j ∈ C) (as introduced in Chapter 4).

Our objective is, given a previously unseen image i ′ i ∈ I ′ , to provide a semantically consistent annotation for i ′ i .

Hierarchical Image Classification

As stated above, our multi-stage reasoning framework for image annotation relies, in a first step, on hierarchical image classification. This chapter does not focus on the effectiveness of the hierarchical classification method. Thus, for the classification of previously unseen images, we may suppose that a classifier for each concept (c i ∈ C ∪ C ′ ) of the semantic hierarchy is already trained by any given hierarchical classification method.

In Chapter 5, we have proposed an approach for performing hierarchical image classification. However, in the context of this chapter, we are interested in the use the subsumption relationships within a framework allowing explicit reasoning tasks on the semantic consistency of the classifier outputs (cf. Section 6.4.2). Therefore, for image classification, we simply used a hierarchical One-Versus-All SVM classifiers instead of using the method proposed in Chapter 5.

Consequently, in this work, the semantic hierarchy is only used to recover the set of positive and negative images for training the classifiers in the different layers.

Moreover, the decision function of each classifier is independent from its subsumed (child) and subsuming (parent) concept nodes. Concretely, given a semantic hierarchy, a classifier for each concept node of the hierarchy is trained by performing a One-Versus-All (OVA) Support Vector Machines. Therefore, for training the classifier of a target concept node, we took as positive samples all images associated with its children leaf nodes. The negative samples are all the other images of the database. So, let x v i be any visual representation of an image i i ∈ I (a visual feature vector), we train for each concept class (c j ∈ C ∪ C ′ ) in the hierarchy a classifier that can associate c j with its visual features. This is achieved by the use of |C| + |C ′ | binary SVM OVA, with a decision function:

G(x v i ) = k α k y k K(x v k , x v i ) + b (6.1) where K(x v k , x v i ) is
K(x v k , x v i ) = exp x v k -x v i 2 σ 2 (6.2)

Reasoning on Image Annotation using the Subsumption Hierarchy

Based on the classifiers outputs and the subsumption relationships, we propose in the following to check the consistency of candidate concepts. So, let us consider a previously unseen image i ′ i ∈ I ′ . Performing a hierarchical image classification on i ′ i produces an output P which consists of a set of candidate concepts {c j ∈ C∪C ′ , j = 1..n i ′ i } and their confidence values {α j , j = 1..n i ′ i }, i.e. P = (c 0 , α 0 ), (c 1 , α 1 ), • • • (c m , α m ) as illustrated in Figure 6.3. Subsequently, these concepts and their confidence scores are transformed into fuzzy description logics assertions. In order to do so, we first normalize into [0, 1] the outputs {α j , j = 1..n i ′ i } of the SVM classifiers by assigning zero to negative values and performing min-max normalization on the positive values. Thereafter, the consistency of each concept c j ∈ C is checked using the subsumption relationships and our fuzzy DL reasoner. Inconsistent concepts are removed from the candidate annotation.

To sum up, our objective is to check the consistency of a candidate concept c j ∈ C to a given image i ′ i using the subsumption relationships, and consequently the set of its hypernym concepts {c k ∈ C ′ | c j : C > 0, c k : D > 0, C ⊑ D > 0}. Therefore, the reasoning process can be formulated using conjunctive queries as follows:

valid(c j ) ← P(c j ) > 0 ∧ c j : C > 0 ∧ c k : D > 0 ∧ C ⊑ D > 0 ∧ valid(c k ) valid(⊤) = 1
where ⊤ is the root of the ontology, and P(c j ) represents the confidence score of the concept c j given by α j .

In DL, given an abstract individual 'a' (an instance of a given candidate concept), the consistency checking of concept inclusions is performed as follows. For C ⊑ D, we compute the greatest lower bound glb(KB, C ⊑ D) using Axiom A6 in Table 4.1, i.e. as the minimal value of x such that KB = T , R, A ∪ { a : C, α 1 } ∪ { a : D, α 2 } is satisfiable under the constraints expressing that α 1 → α 2 ≤ x, with α 1 and α 2 ∈ [0, 1]. This process is then iterated until the root of the ontology is reached. Thus, we come up with the following hierarchy:

C 1 ⊑ C 2 ≥ x 1 , C 2 ⊑ C 3 ≥ x 2 , • • • , C n ⊑ ⊤ ≥ 1.
Thereafter, a confidence score for the considered candidate concept is computed as follows:

bed(KB, a :

V alidCC) = x 1 ⊗ x 2 ⊗ • • • ⊗ 1 = x 1 * x 2 * • • • * 1 (6.3)
where V alidCC stands for a Valid Candidate Concept, which is a concept defined to regroup all the consistent candidate concepts. Finally, all candidate concepts with a confidence score equal to zero are removed from the annotation of image i ′ i . For instance, let us consider the first example in Figure 6.3. The image classification algorithm has detected "Motorbike" as a candidate concept (among others) for the considered image. However, according to the subsumption hierarchy (cf. Figure 4.5) a "Motorbike" ⊑ "Wheeled_vehicle" ⊑ "Conveyance", etc., and therefore the classifiers should also have detected these concepts to stay coherent. The consistency checking of the concept "Motorbike" is performed according to the previously described procedure, and thus this concept is removed from the list of candidates since bed(KB, M otorbike : V alidCC) = 0.

Example 9 (Consistency checking of concept "Motorbike").

KB = T , R, A ∪ { a : M otorbike ≥ 0.262 } ∪ { a : W eeled_vehicule ≥ 0 } ∪ { a : Conveyance ≥ 0 } ∪ { a : Abstraction ≥ 0.109 } ∪ { a : Concept ≥ 1 } bed(KB, M otorbike : V alidCC) = 0.262 ⊗ 0 ⊗ 0 ⊗ 0.109 ⊗ 1 = 0

Reasoning on Image Annotation using Image Context

As aforementioned, contextual information can provide valuable information for the understanding of image context or to reason about the consistency of a given annotation. For instance, it is evident that an image which contains the set of concepts {"Aeroplane", "Person", "Car"} represents a scene of an airport tarmac, and not the one of a flying plane. And conversely, it is obvious that an image that contains "Dining_table" and "Sofa" should not contain "Boat" or "Bus". Thus, contextual information, if processed, can be helpful to check the consistency of image annotations.

Using our multimedia ontology, it is easy to recover contextual information about images. Consequently, we propose in the following to use this information to recover Chapter 6. Multi-Stage Reasoning Framework for Image Annotation from our ontology all consistent annotations with respect to contextual information, and to compute the best explanation of a considered image. Specifically, the fuzzy role "isAnnotatedBy" allows predicting a confidence score (based on contextual information) for a given set of candidate concepts. Given a Candidate Annotation CA j = c 1 , c 2 , • • • , c m , and a target image i ′ i ∈ I ′ , a confidence score is computed to estimate the correlation likelihood between CA j and i ′ i . This confidence score increases according to the likeliness of the candidate annotation CA j , or it is equal to 0 when the annotation is not valid.

Given an image i ′ i and P

′ : (c 0 , α 0 ), (c 1 , α 1 ), • • • (c m , α m ) , m = |P ′ |,
a set of valid candidate concepts with respect to the subsumption relationships, we build first the set of candidate annotation (CA j , j ∈ 1..|combinaisons|) by taking all the possible combination of the concepts in P ′ . A confidence score for each valid candidate annotation (ValidCA) is then computed. For instance, let us assume that we dispose of one candidate annotation consisting of 3 concepts. Its confidence score is computed as follows:

Example 10 (Reasoning using image context).

P ′ : (c 1 , α 1 ), (c 2 , α 2 ), (c 3 , α 3 ) , (classifier outputs) c 1 : C 1 ≥ α 1 , c 2 : C 2 ≥ α 2 , c 3 : C 3 ≥ α 3 CA ≡ C 1 ⊓ C 2 ⊓ C 3 b : CA ≥ α b , s.t. α b = α 1 ⊗ α 2 ⊗ α 3 KB = T , R, A ∪ { a : Image ≥ α a } ∪ { b : CA ≥ α b }
(a, b) : isAnnotatedBy ≥ α r , is already stored in the KB during the ontology building process, where α r = µ isAnnotatedBy(a,b) (cf. Equation 4.8).

Therefore, according to Equation 4.1, the correlation likelihood between a candidate annotation CA and a given image i ′ i can be computed as follows:

glb(KB, a : ∃ isAnnotatedBy.CA) = α b ⊗α r = (α 1 ⊗α 2 ⊗α 3 )⊗ µ isAnnotatedBy(a,b) (6.4) then, V alidCA ≡ ∃ isAnnotatedBy.CA (6.5)

Finally, the best explanation (bex) of i ′ i is retrieved as the ValidCA having the maximum correlation likelihood among all the others. This explanation is computed as follows: bex(KB, V alidCA) = { a, r |r = bed(KB, a : V alidCA)} (6.6) For instance, let us consider the first example in Figure 6.3. We show below some cases of DL reasoning using the contextual information: detected concepts, and therefore the reasoning capabilities using spatial information are limited in the current approach. However, we propose in the following a simple but effective usage scenario that relies on the spatial arrangement of the currently detected concepts in order to provide a semantically consistent image annotation -cf. Algorithm 3.

Given an image i ′ i ∈ I ′ and P ′′ : (c 0 , α 0 ), (c 1 , α 1 ), • • • (c m , α m ) , m = |P ′′ |, a set of a valid candidate concepts with respect to the subsumption relationships and contextual information. We propose first to query the ontology in order to retrieve all possible spatial arrangement of all pairs of concepts (c j , c k ) ∈ P ′′ , and to recover the confidence score of each of these spatial arrangements. A score can then be computed as the maximum likelihood of all spatial arrangements of these concepts to find the best explanation of i ′ i . Algorithm 3 details the different steps of this method. Reasoning on spatial information should also allow to provide a good image interpretation. For instance, computing the maximum spatial arrangement likelihood allows to retrieve the likeliness of spatial arrangement of each detected concept in a given image. This will allow for example, to provide a textual description of a given image in the following way: "This picture depicts a cat sitting on a table in a living room. There is a table, a sofa and a television in the living room."

It is easy to implement such a system for image interpretation once we dispose of the information about detected concepts and their spatial location [START_REF] Gupta | From image annotation to image description[END_REF]. We will address the implementation of such a system in our future work.

Experiments

For evaluating our multi-stage reasoning framework for image annotation, we used two public datasets: Pascal VOC'2009 [START_REF] Everingham | The PASCAL Visual Object Classes Challenge 2009[END_REF], and Pascal VOC'2010 [START_REF] Everingham | The PASCAL Visual Object Classes Challenge 2010[END_REF] datasets. These datasets contain about 11 000 images annotated with 20 predefined concepts. Each image is annotated by one or more concepts (multi-labeled images). A fair comparison of our proposal to existing methods for image annotation is also performed on these datasets. in the following, we first introduce the used method for visual representation of images, then we present the obtained experimental results.

Visual Representation of Images

The Bag-of-Features (BoF) representation, also known as Bag-of-Visual-Word (BoVW), is used in this work to describe image features. The BoF model has shown excellent performances and became one of the most widely used model for image classification and object recognition [START_REF] Li | A bayesian hierarchical model for learning natural scene categories[END_REF]. In our approach, image features are described as follows: Lowe's DoG Detector [Lowe, 1999] is used for detecting a set of salient image regions. A signature of these regions is then computed using SIFT descriptor [Lowe, 1999]. Afterwards, given the collection of detected region from the training set of all categories, we generate a codebook of size K = 1000 by performing the k-means algorithm. Thus, each detected region in an image is mapped to the most similar visual word in the codebook through a KD-Tree. Each image is then represented by a histogram of K visual words, where each bin in the histogram corresponds to the occurrence number of a visual word in that image.

Evaluation

Our experiments are performed on Pascal VOC'2009 and Pascal VOC'2010 datasets. Since we do not disposed of the test set, we used the training set for training our concept classifiers and the validation set for evaluating our approach. In Figure 6.4, we compare our framework for image annotation to the following methods:

• A flat classification method.

• A hierarchical classification method using the hierarchy illustrated in Figure 4.5 and OVA classifiers.

• A baseline method.

The baseline method is built by taking the average submission results to Pascal VOC'2010 challenge. The flat classification is performed by using |C| SVM One-Versus-All (OVA), where the inputs are the BoF representation of images and the outputs are the desired SVM responses for each image (1 or -1). We used crossvalidation to overcome the unbalanced data problem, taking at each fold as many As illustrated in Figure 6.4, our method for image annotation performs better results than the other ones, with an average precision of 66.49% and a gain of +8.6% comparing to the baseline method, a gain of +14.8% comparing to the hierarchical classification method and a gain of +32.6% comparing to the flat classification method. These results confirm the effectiveness of the proposed approach, and the importance of contextual and spatial information for improving image annotation. These improvements could be further significant when using a dataset containing much more multi-labeled images. Indeed, in the Pascal VOC dataset the proportion of images labeled with more than 2 concepts is small comparing to the total number of images [START_REF] Choi | Exploiting hierarchical context on a large database of object categories[END_REF].

In table 6.1, we compare our multi-stage reasoning framework for image annotation to the method of [START_REF] Zhou | Image classification using super-vector coding of local image descriptors[END_REF] on Pascal VOC'2009 dataset. [START_REF] Zhou | Image classification using super-vector coding of local image descriptors[END_REF] proposed a method for image classification using local visual descriptors and their spatial coordinates. Their method consists in performing first a nonlinear feature transformation on local appearance descriptor, termed as super-vector, which exploits the residual vector information obtained from the vector quantization (VQ). These de-Proposed Method (AP) [START_REF] Zhou | Image classification using super-vector coding of local image descriptors[END_REF] scriptors are then aggregated to form image-level feature vector. The image-level feature vector is finally fed into a classifier to perform image classification. As illustrated in table 6.1, our method performs better than the one proposed by [START_REF] Zhou | Image classification using super-vector coding of local image descriptors[END_REF], and achieves a gain of +3.41%. This result is promising especially because we did use only the half of the training set for training our classifiers and the other images for evaluating our approach, since we did not dispose of the testing set. We also wish to recall that we have included in our evaluation the images and the concepts marked as difficult, which are ignored in the challenge because they are considered as difficult to recognize. For instance, in the third example of Figure 6.3, we can easily observe a "Dining_table" in the illustrated image. However, "Dining_table" is marked as difficult in the ground-truth of this image in the VOC'2010 challenge, and thus it will not count for computing the average precision of this concept. In our evaluation, we included these concepts, i.e. if they are not detected they will count as false negative. Furthermore, the scope of our paper was to study the potential of adding contextual and spatial information into the image annotation process through the use of ontology and ontological reasoning. Thus, we have focused our contribution on these points and we did not seek to implement a more efficient image descriptor since this is not the aim of our paper. Accordingly, the obtained results can be further improved by incorporating other image features for example. In Figure 6.5, we compare our framework for image annotation to the following methods: Bottom-Up Score Fusion (BUSF ) and Top-Down Classifiers Voting (TDCV ) which were proposed in Chapter 5, and Hierarchy of SVM (H-SVM ) method of [START_REF] Marszalek | Semantic hierarchies for visual object recognition[END_REF]. As it can be seen on this figure, our multi-stage reasoning framework for image annotation outperforms on all classes comparing to the other ones. Please note that this comparison was performed using the same dataset, i.e. Pascal VOC'2010, the same training/validation sets from the VOC'2010 dataset, and the same visual representation of images -cf. Section 6.5.1. Therefore, it is clear that the proposed approach allows achieving a significant improvement for image annotation, and that the contextual and spatial information are of great importance to improve the accuracy of image annotation.

Finally, we want to highlight that some images in the VOC dataset are badly annotated. For instance in the third example of Figure 6.3, we can distinguish a bottle partially hidden by a vase, and a potted flower in the background of the image. However, these concepts (i.e. "Bottle" and "Potted_plant") are missing in the ground-truth of this image. Thus, despite that our method succeeded to recognize these concepts, they counted as a false positive detection in the evaluation of our method since they are missing in the ground-truth. For the second example of 6.3, our method has detected the concept "Dining_table" which is absent from the ground-truth. However, the image depicts indeed a "coffee table" and therefore our prediction is semantically relevant, especially since the annotation vocabulary does not provide concepts such as "Table " and "Coffee_Table". In Figure 6.6, we illustrate another image which is badly annotated in the dataset. Indeed, the groundtruth of this image contains only the concept "Person". However, the image depicts much more concepts: a bottle, chairs, tables, and screens. Our method has detected these concepts, but according to the ground-truth these detections counted as false positives.

Conclusion

In Chapter 4 we have proposed an approach to automatically build a fuzzy multimedia ontology suitable for image annotation and interpretation. Our multimedia ontology incorporates visual, conceptual, contextual and spatial knowledge in order to effectively model image semantics. Based on our multimedia ontology, we proposed in this chapter a new method for image annotation based on hierarchical image classification and a multi-stage reasoning framework for checking the consistency of the produced annotation. The proposed framework uses different knowledge about image concepts in order to remove the uncertainty about image annotation introduced during the image classification process. These knowledge include: subsumption rela-tionships, contextual relationships and spatial relationships. The reasoning tasks are achieved in Fuzzy Description Logics (Fuzzy-DLs). An empirical evaluation of our approach is performed on Pascal VOC'2009 and Pascal VOC'2010 datasets. The obtained results have shown a significant improvement on the average precision results, thus demonstrating the effectiveness of the proposed method.

As an extension to this work, we propose in the near future to test our approach using more robust image descriptors, and using a larger image datasets. Furthermore, a good direction to enhance our proposal is to implement a system that provides a human-like descriptions of images, similar to the work proposed by [START_REF] Gupta | From image annotation to image description[END_REF]. Indeed, the authors have shown in their paper that it is possible to automatically generate a description of an image based on its annotation. We also plan to implement and test the usage scenarios described in Chapter 4 -Section 4.7.

Chapter 7

Conclusions and Future Research Directions

Image annotation represents a major economic issue and involves several applications, including but not limited to: social networks, the Web, medical imaging, biocomputing, remote sensing, news, image and video monitoring and information retrieval. Recently, many advances have been made to reduce the semantic gap problem, and to improve automatic image annotation systems. These advances include computer vision techniques (such as image features, image descriptors, sparse representation, etc.) and artificial intelligence (such as machine learning, data clustering, knowledge engineering, etc.). However, automatic image annotation is still an open issue and does not seem to be resolved in the next few years. Indeed, current systems for image annotation face significant problems, including the semantic gap, the scalability problem, the subjectivity of image semantics, the lack of robustness of current computer vision techniques, and more.

In this dissertation, we address the problem of automatic image annotation from the perspective of building and using explicit semantic models, such as semantic hierarchies and ontologies. Our overall goal is to narrow the semantic gap and to achieve a semantically consistent image annotation. Indeed, recent efforts on knowledge modeling and knowledge-driven approaches for image annotation have shown that it is possible to reduce the semantic gap. Our concern in this thesis was therefore to demonstrate this assumption, and to show how it would be possible to use these explicit and structured knowledge models to improve image annotation.

The remaining of this chapter is structured as follows. The first part reviews the different contributions proposed in this PhD work and their significance. Specifically, we focus on how these contributions can improve existing systems for image annotation and what issues could be answered/narrowed using the proposed knowledge models. In the second part, we present our perspectives to improve the proposed approaches and our future research directions.

Contributions and their Significance

As aforementioned, the aims of this thesis are twofold. As a first step, we have focused our contributions on providing effective tools for the automatic building of structured and explicit knowledge models dedicated to image annotation. This is intended to propose knowledge models that are representative of image semantics, and allow modeling image contexts in an accurate way. Subsequently, we have concentrated our contributions on the effective use of this knowledge in order to achieve a more accurate image annotation. Our final purpose is to narrow the semantic gap, i.e. decrease the gap between the visual content of images and their meaning (image semantics). Therefore, the achieved contributions in this dissertation can be categorized in two main topics: 1) the automatic building of explicit and structured multimedia knowledge models, 2) the use of structured knowledge models to improve image annotation, and specifically, the use of formal fuzzy reasoning in order to reduce the uncertainty about image annotation. The detailed set of accomplishments are introduced in the following.

A Thorough Study of State-of-the-art

We proposed in this dissertation a thorough survey of the state-of-the-art regarding image annotation approaches. A special attention was given to knowledge-driven approaches for semantic image annotation. Firstly, we have summarized the proposals of current approaches, and then we pointed out what is missing in these approaches and which directions could be borrowed to narrow the semantic gap. Because of the huge number of existing methods for image annotation, the proposed state of the art is not exhaustive, and was intended only to show the limits of current approaches. Therefore, we have focused on the set of methods that we have considered as relevant with respect to the topic of our work. We finally have proposed a tentative to classify the different knowledge-based approaches for image annotation according to the level of expressiveness of the knowledge models [START_REF] Bannour | Towards ontologies for image interpretation and annotation[END_REF].

Significance:

⇒ The proposed survey of the state-of-the-art on semantic image annotation is focused on knowledge-based approaches for image annotation. This survey was the starting point of the directions taken in this PhD work, and the proposed contributions. It has also served to explain our standpoints about several issues related to image annotation, image retrieval, and terminologies that follow from these topics.

Building Semantic Hierarchies Faithful to Image Semantics

The first main contribution of this dissertation deals with the automatic building of semantic hierarchies for the purpose of image annotation. Indeed, according to our assumptions in the Chapter 3 -Section 3.2.4, a suitable semantic hierarchy for image annotation should take account of the visual information and also the conceptual and the contextual ones. Consequently, we have proposed a new image-semantic measure, named 'Semantico-Visual Relatedness of Concepts' (SVRC), which allows to estimate the semantic similarity between concepts with respect to these three kinds of information. This proposed measure incorporates visual, conceptual and contextual information in order to provide a measure which is more meaningful and more representative of image semantics. Subsequently, we have proposed a new methodology to automatically build semantic hierarchies suitable for image annotation. The building is based on the previously proposed measure (SVRC ) and on a set of simple but effective rules, named TRUST-ME, in order to connect the concepts with higher relatedness till the building of the final hierarchy [Bannour and Hudelot, 2012a[START_REF] Bannour | Combinaison d'information visuelle, conceptuelle, et contextuelle pour la construction automatique de hiérarchies sémantiques adaptées à l'annotation d'images[END_REF], Bannour and Hudelot, 2013b].

Significance:

⇒ The proposed measure, SVRC, is representative of image semantics as it incorporates the different modalities of images. Indeed, our experiments have shown that image modalities have different distributions in the semantic space, and their combination can lead to a more meaningful correlation between concepts in the image domain.

⇒ The built semantic hierarchy is faithful to image semantics, since it is a based on the previously proposed measure (i.e. SVRC ) and a set of rules allowing to link the concepts sharing a higher "semantico-visual" relatedness.

⇒ The semantico-visual relatedness measure, and consequently the built hierarchy, can adapt to an application domain and even to the user background since it incorporates a contextual similarity which is domain dependant.

Building Fuzzy Multimedia Ontologies for Image Annotation

A major issue with semantic hierarchies is that they are limited to describe the subsumption relationships between concepts, i.e. only the 'is-a' relationship is used in these semantic structures. In this dissertation, we proposed to go further in the modeling of image-concepts relationships, and therefore the modeling of image semantics. Consequently, we proposed a new approach for building an ontology of spatial and contextual information, suitable for reasoning about the coherence of image annotation. In our approach, visual and conceptual information were used to build a semantic hierarchy that will serve as a backbone of our ontology. Contextual and spatial information about image concepts were thereafter incorporated in the ontology to model other semantic relationships between these concepts. Fuzzy description logics were used as a formalism to represent our ontology and the inherent uncertainty and imprecision of this kind of information. The proposed ontology was automatically built by mining image databases and the provided annotations, which ⇒ Performing image classification using our framework allows a multi-level of abstraction image annotation, i.e. the target image is annotated by a set of concepts (from specific to generic) of a given path in the hierarchy.

Multi-Stage Reasoning Framework for Image Annotation

Finally, we proposed a framework for image annotation based on hierarchical image classification and a multi-stage reasoning framework for reasoning about the consistency of the produced annotation. In this approach, fuzzy ontological-reasoning is used in order to achieve a relevant decision on the belonging of a given image to the set of concept classes. An empirical evaluation of our approach on Pascal VOC'2009 and Pascal VOC'2010 datasets has shown a significant improvement on the average precision results. Our approach is of interest since it illustrates the reasoning power of multimedia ontologies and their effectiveness for achieving good decisions on image annotation [Bannour and Hudelot, 2013a].

Significance:

⇒ Previous methods dealing with multimedia ontologies are only providing some formalism to use ontologies as a repository for storing knowledge about image content. Few attempts have considered the problem of reasoning about this knowledge, but on limited data and without considering real applications. Thus, the effectiveness of the stored knowledge has to be proved. In this dissertation, we address deeply the reasoning process to show the usefulness of multimedia ontologies and the effectiveness of the stored knowledge about images.

⇒ Our framework has allowed solving many issues of image annotation. Indeed, image annotation is a difficult task because of the uncertainty introduced by the statistical learning algorithms, the scalability problem, and the dependency on the accuracy of the ground truth of the training dataset. Our approach uses explicit knowledge in order to reduce this uncertainty by supplying a formal framework to reason about the consistency of extracted information from images.

Future Research Directions

Many contributions were proposed in this dissertation to deal with the image annotation problem. These contributions are in no way complete solutions, and could be improved in several manners. Indeed, as aforementioned in Chapter 2, the semantic gap problem is an open-ended problem and seems to not be solved in the near future. Moreover, automatic image annotation is a difficult task because of the uncertainty introduced by machine learning algorithms. Thus, the topic of image annotation is witnessing an incremental improvement, and many efforts are still needed before Chapter 7. Conclusions and Future Research Directions achieving efficient systems dealing with this problem. In the following, we propose potential directions that can be explored further.

• Our major concern in this dissertation was to provide knowledge-driven models for improving image annotation accuracy. Thus, in this work we have not sought to implement more robust/sophisticated image descriptors when performing our evaluation. It is therefore of interest to see how far the proposed approaches can be used to push the state-of-the-art performance by using one of the most sophisticated classification (or detection) method.

• In order to evaluate the robustness of our approaches with large scale data, it is important to test our proposals on larger image datasets, i.e. including thousands of concepts and millions of images. This will allow for instance, to see if the proposed methods for building semantic hierarchies and hierarchical image classification are still efficient with such voluminous data. It is also widely known that DL reasoners suffer from the scalability problem, and it is important to survey this limit on real data applications.

• As an extension to our proposal in Chapter 4 dealing with the building of multimedia ontologies, we propose in the near future to implement and to test the usage scenarios described in Section 4.7. Indeed, two methods were proposed: the first allowing to learn very complex concepts by mining multimedia datasets, while the second allows reducing significantly the complexity of object detection process (specifically when dealing with a large annotation vocabulary).

• Implementing a system that provides a human-like descriptions of images. Similar works were proposed by [Kulkarni et al., 2011, Gupta and[START_REF] Gupta | [END_REF]. Indeed, the authors have shown in their papers that it is possible to automatically generate a textual description of images based on their annotations. Our framework for image annotation, proposed in Chapter 6, allows to dispose of meaningful attributes about concepts relationships. Therefore, performing such an approach for image description on the outputs of our framework will allow achieving an efficient description of images, i.e. a detailed description of image content through the description of image concepts and concept relationships.

  of image retrieval systems. . . . . . . . . . . . . . . . . . . 1.2 Proposed taxonomy of image retrieval approaches. . . . . . . . . . . . 1.3 The surrounding context is not always relevant with respect to image content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Semantic gap problem. . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Images and their semantic. These examples show that image semantics is not always straightforward and subtle. . . . . . . . . . . . . . . . . 1.6 From image data to structured knowledge Models. . . . . . . . . . . . 1.7 Using structured knowledge models for image annotation. . . . . . . . 2.1 The scope of our work within the proposed taxonomy of image retrieval approaches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Example of automatic image annotation. . . . . . . . . . . . . . . . . 2.3 Images and their semantic. . . . . . . . . . . . . . . . . . . . . . . . . 2.4 An image may have different interpretations depending on the observer's background knowledge. . . . . . . . . . . . . . . . . . . . . . 2.5 Levels of abstraction for multimedia content, and respectively image content. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Concept-detectors based approaches, also called sliding-window object detection methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 Scheme of the Viola and Jones face detector. . . . . . . . . . . . . . . 2.8 Basic scheme for image annotation as a classification problem. . . . . 2.9 Explicit knowledge structures used for image annotation. . . . . . . . 2.10 Architecture of a knowledge representation system based on Description Logics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.11 Comparison of the different categories of approaches for image annotation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 A concept hierarchy built by extracting the relevant subgraph of Word-Net that links the 20 concepts of the VOC'2010 dataset. . . . . . . . 3.2 Method of

4. 1

 1 From image data to structured knowledge models: Architecture of our approach for building multimedia ontologies dedicated to image annotation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Illustration of the used roles for defining concept relationships in our ontology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Spatial primitives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Directional relationships are computed according to an angle α with the x-axis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 The semantic hierarchy built on the Pascal VOC'2010 dataset. . . . . 4.6 The built multimedia ontology on Pascal VOC dataset. . . . . . . . . 5.1 Hierarchical classification using DDAG and BHDT . . . . . . . . . . 5.2 Existing approaches for multi-class image classification. . . . . . . . . 5.3 The built semantic hierarchy on VOC'2010 dataset. . . . . . . . . . . 5.4 Proposed method for training hierarchical classifiers: One-Versus-Opposite-Nodes (OVON). . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Decision function for the Bottom-Up Score Fusion method. . . . . . . 5.6 Decision function for the Top-Down Classifiers Voting method. . . . . 5.7 Comparison of the One-Versus-Opposite-Nodes (OVON) and the One-Versus-All (OVA) hierarchical classifiers on VOC'2010 dataset. . . . . 5.8 Comparison of our methods for hierarchical image classification with a

xv 6 .

 6 1 A global overview of our approach. . . . . . . . . . . . . . . . . . . . 6.2 Multi-stage reasoning framework for image annotation. . . . . . . . . 6.3 Illustrative examples of the proposed framework for image annotation. 6.4 Comparison of our framework for image annotation with: a flat classification method, a hierarchical classification one, and the baseline method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Comparison of our framework for image annotation to previous work on Pascal VOC'2010 dataset. Our approach outperforms on all classes comparing to the other ones. . . . . . . . . . . . . . . . . . . . . . . . 6.6 An example of a badly annotated image in the VOC'2010 dataset. . . xvi List of Tables

4. 1

 1 Syntax and semantics of the Fuzzy Description Logic f -SROIQ(D) used for designing our multimedia ontology. . . . . . . . . . . . . . . 4.2 Roles and functional roles used for defining concept relationships in our ontology (RBox ). Light-orange rows correspond to contextual relationships and light-blue rows correspond to spatial relationships. . . 5.1 Complexity of existing approaches for image classification. . . . . . . 5.2 Complexity of our methods compared to the DDAG and BHDT approaches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Comparison of our method for image annotation with the one of [Zhou et al., 2010] on Pascal VOC'2009 dataset. . . . . . . . . . . . . xvii Chapter 1

  Figure 1.1: Workflow of image retrieval systems.

Figure 1

 1 Figure 1.3: The surrounding context is not always relevant with respect to image content. Used query: French Flag, used search engine: Bing Images. As illustrated in this example, the search engine returns for instance the image of the Spanish flag because its surrounding context contains "French Flag Wallpaper Page 12 • • • ".

  Figure 1.4: Semantic gap problem. Images (a) and (b) have similar color histograms but different meanings. Images (a) and (d) have different color histograms but the same meaning.

  Antarctic penguins on ice floes. (f) Fractal Broccoli.(g) The crab nebula.

Figure 1 . 5 :

 15 Figure 1.5: Images and their semantic. These examples show that image semantics is not always straightforward and subtle. The picture (a) depicts Love (a man kissing a woman under a red umbrella), (b) depicts a family picture: a father carrying his daughter on his shoulders who is holding a soft toy in her hands, pictures (c) and (d) depict Jaguars, (e) depicts a global warming scene, (f) depicts a fractal broccoli, and (g) depicts the Crab Nebula which is the result of a supernova that occurred in 1054 AD. This image was taken by the Hubble telescope, and in the center is a pulsar, or a neutron star.

Figure 1 . 6 :

 16 Figure 1.6: From image data to structured knowledge Models. Workflow of our approaches for building structured knowledge models dedicated to image annotation.

Figure 2 . 2 :

 22 Figure 2.2: Example of automatic image annotation.

  (a) A spectacular jump of a killer whale in an aquatic parc. (b) A spectacular jump of a killer whale to eat penguins. (c) Global warming threatens the survival of polar animals (Antarctic penguins on ice floes). (d) Global warming threatens the survival of polar bears. (e) Our survival is threatened by global warming. (f) Cars mercedes-benz race tracks. (g) Corvette racing wins at american Le Mans -inside track.

Figure 2 . 3 :

 23 Figure 2.3: Images and their semantic. Images (a) and (b) depict a spectacular jump of a killer whale; images (c), (d) and (e) depict the global warming; and images (f) and (g) depict cars on a racetrack.

  (a) Hannibal counting the signet rings of Roman nobles killed during the battle, statue by Sébastien Slodtz, 1704, Louvre. (b) The Swordfish was a two-place torpedo bomber used during the Second World War.

Figure 2 . 4 :

 24 Figure 2.4: An image may have different interpretations depending on the observer's background knowledge.

Figure 2 Figure 2 . 5 :

 225 Figure 2.5: Levels of abstraction for multimedia content, and respectively image content.

  Pedestrian detection using Histograms of Oriented Gradients (HOG)[START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF]

Figure 2 . 6 :

 26 Figure 2.6: Concept-detectors based approaches, also called sliding-window object detection methods.

Figure 2

 2 Figure 2.7: Scheme of the Viola and Jones face detector[START_REF] Viola | Rapid object detection using a boosted cascade of simple features[END_REF].

Figure 2 . 8 :

 28 Figure 2.8: Basic scheme for image annotation as a classification problem.

Figure 2 .

 2 Figure 2.9: Explicit knowledge structures used for image annotation.

Figure 2 .

 2 Figure 2.10: Architecture of a knowledge representation system based on Description Logics.

Figure 2 .

 2 Figure 2.11: Comparison of the different categories of approaches for image annotation.

Figure 3

 3 Figure 3.1: A concept hierarchy built by extracting the relevant subgraph of WordNet that links the 20 concepts of the VOC'2010 dataset. Double octagon nodes are the initial concepts and the diamond one is the root of the hierarchy.

Figure 3

 3 Figure 3.3: An example illustrating that conceptual measures are not always relevant with respect to the image domain. According to the WordNet density measure [Deng et al., 2010], dolphins (a) are more similar to humans (b), than to sharks (c).

Figure 3 . 4 :

 34 Figure 3.4: Overview of the SVRC measure which is based on visual, conceptual and contextual similarities.

  the learned weight of the training sample x v k , and b is a learned threshold parameter. Note that the training samples x v k with weight α k > 0 are the support vectors.

Figure 3 . 7 :

 37 Figure 3.7: Rules in TRUST-Me allowing to infer the relatedness relationships between the different concepts. Preconditions (in red) and actions (in black).

Algorithm 1 :

 1 Semantic Hierarchy BuildingData: Images and their annotation Result: A semantic hierarchy a set of trained classifiers for each concept node in the hierarchy begin-∀ c i ∈ C, ⊤(concepts) ← c i foreach (c i , c j ∈ ⊤(concepts)) do -Compute the visual similarity ϕ(c i , c j ) -Compute the conceptual similarity π(c i , c j ) -Compute the contextual similarity γ(c i , c j ) -Compute the SVRC φ(c i , c j ) end while |⊤(concepts)| > 1) do ⊥(concepts) ← ⊤(concepts) ⊤(concepts) ← ∅-Connect one level of the semantic hierarchy using TRUST-ME :⊤(concepts) ← TRUST-ME(⊥(concepts)) foreach (c i , c j ∈ ⊤(concepts)) do -Compute φ(c i , c j )by averaging the distances of hyponyms of c i to hyponyms of c j end end end * ⊥: stands for lower level in the hierarchy * ⊤: stands for higher level in the hierarchy

Figure 3 Figure 3 Figure 3

 333 Figure3.9: A binary semantic hierarchy built on Pascal VOC'2010 dataset using our measure SVRC and TRUST-ME. Double octagon nodes are the original concepts and the diamond one is the root of the produced hierarchy.

Figure 3 .

 3 Figure 3.12: Heat maps illustrating the semantic affinity matrix of the visual similarity (a), conceptual similarity (b), contextual similarity (c) and SVRC similarity (d), between VOC'2010 concepts.

Figure 4 .

 4 Figure 4.1: From image data to structured knowledge models: Architecture of our approach for building multimedia ontologies dedicated to image annotation.

  , b) Irreflexive role irr(S) ∀a ∈ ∆ I , S I (a, a) = 0 Asymmetric role asy(S) ∀a, b ∈ ∆ I , if S I (a, b) > 0 then S I (b, a) = 0 Table 4.1: Syntax and semantics of the Fuzzy Description Logic f -SROIQ(D) used for designing our multimedia ontology. a, b ∈

Figure 4 .Figure 4 . 4 :

 444 Figure 4.3: Spatial primitives are computed according to the angle between the segment joining two points 'a' and 'b' and the x-axis of the image. 'a' and 'b' are the centroids of two given objects (here "Cow" and "Person") in a given image.

Example 6 (

 6 Spatial Relationships). a : Bottle ≥ 1 b : Dining_T able ≥ 1 (a : b) : hasAppearedAbove ≥ 0.76 (a : b) : hasAppearedBelow ≥ 0.02 (a : b) : hasAppearedAlignedW ith ≥ 0.62 (a : b) : hasAppearedCloseT o ≥ 0.97 • • •

Chapter 4 .Figure 4 . 5 :

 445 Figure 4.5: The built semantic hierarchy on the Pascal VOC'2010 dataset. Double octagon nodes are original concepts (i.e. concepts in C), and the diamond one is the root of the produced hierarchy.

Figure 5 . 1 :

 51 Figure 5.1: Hierarchical classification using Decision Directed Acyclic Graphs (DDAG) and Binary Hierarchical Decision Trees (BHDT). 5.1(a) illustrates the decision DAG for finding the best class out of four classes {1,2,3,4} [Platt et al., 2000]. 5.1(b) illustrates a simple BHDT with four classes: {1,2,3,4} [Griffin and Perona, 2008].

Figure 5 . 2 :

 52 Figure 5.2: Existing approaches for multi-class image classification. The approaches in red boxes present a square complexity, those in yellow boxes present a linear complexity, and those in green boxes present a logarithmic complexity.

Figure 5 . 5 :

 55 Figure 5.5: Decision function for the Bottom-Up Score Fusion method. For instance, given x v i the visual representation of an imagei i , f c 5 (x v i ) = 1 3 * (G c 5 (x v i ) + G c ′ 7 (x v i ) + G c ′ 2 (x v i )), where G c 5 (x v i )is the decision function of the classifier associated with c 5 , and so on. If f c 5 (x v i ) > 0, then the image i i is annotated by c 5 , c ′ 7 , c ′ 2 .

  is the value of a kernel function for the training sample x v k and the test sample x v i , y k ∈ {1, -1} the class label of x v k , α k the learned weight of the training sample x v k , and b is a learned threshold parameter.

Figure 5 . 6 :

 56 Figure 5.6: Decision function for the Top-Down Classifiers Voting method. Green arrows correspond to explored paths, red arrows correspond to rejected paths according to the classifier decision and black arrows correspond to not explored paths/nodes.

Figure 5 . 7 :

 57 Figure 5.7: Comparison of the One-Versus-Opposite-Nodes (OVON) and the One-Versus-All (OVA) hierarchical classifiers on VOC'2010 dataset.

Figure 5 . 8 :

 58 Figure 5.8: Comparison of our methods for hierarchical image classification: Bottom-Up Score fusion (BUSF) and Top-Down Classifiers Voting (TDCV), with the Baseline, H-SVM, and the flat classification method.

  (a) AP for the 2 nd layer concepts.(b) AP for the 3 rd layer concepts.(c) Average Precision for the fourth layer concepts.

Figure 5 .

 5 Figure 5.9: Recall/Precision curves for the concepts of each level of the hierarchy.

Figure 6 . 2 :

 62 Figure 6.2: Proposed method: a knowledge-based multi-stage reasoning framework for image annotation.

Algorithm 3 :-

 3 Reasoning using spatial Information Input: A valid candidate annotation: ValidCA Result: Semantically consistent image annotation begin FindMax spatial arrangement of E and C s.t Spatial arrangement of E and D is satisfiable -Reiterate the process with the remaining concepts in ValidCA end

Figure 6 .Figure 6 . 3 -

 663 Figure 6.3 -first example: "This picture depicts a person standing on the left of a sheep. They are close to each other." Figure 6.3 -Second example:"This picture depicts a cat sitting on a table in a living room. There is a table, a sofa and a television in the living room."

Figure 6 . 4 :

 64 Figure 6.4: Comparison of our framework for image annotation with: a flat classification method, a hierarchical classification one, and a baseline method. The comparison is performed on VOC'2010 dataset.

Figure 6 . 5 :

 65 Figure 6.5: Comparison of our framework for image annotation to previous work on Pascal VOC'2010 dataset. Our approach outperforms on all classes comparing to the other ones.

Figure 6 . 6 :

 66 Figure 6.6: An example of a badly annotated image in the VOC'2010 dataset. Ground-truth: Person. Annotation provided by our method: Bottle : 0.982, Chair : 0.281, Dining_table : 0.493, P erson : 1.00, T v_monitor : 0.333 .

Figure

  Figure 6.3, our method has detected the concept "Dining_table" which is absent from the ground-truth. However, the image depicts indeed a "coffee table" and therefore our prediction is semantically relevant, especially since the annotation vocabulary does not provide concepts such as "Table"and "Coffee_Table". In Figure6.6, we illustrate another image which is badly annotated in the dataset. Indeed, the groundtruth of this image contains only the concept "Person". However, the image depicts much more concepts: a bottle, chairs, tables, and screens. Our method has detected these concepts, but according to the ground-truth these detections counted as false positives.

  for the building of visual object class hierarchies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 An example illustrating that conceptual measures are not always rele-Overview of the SVRC measure which is based on visual, conceptual and contextual similarities. . . . . . . . . . . . . . . . . . . . . . . . . 3.5 From image data to structured knowledge models: Architecture of our approach for building semantic hierarchies dedicated to image annotation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Objective: for each concept c i ∈ C, find a centroid ϑ(c i ) that minimizes δ j (δ j is the intra-class variance) for all x v j ∈ S i . . . . . . . . . . . . . 3.7 Rules in TRUST-Me allowing to infer the relatedness relationships between the different concepts. Preconditions (in red) and actions (in black). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8 The semantic hierarchy built on Pascal VOC'2010 dataset using the proposed method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.9 A binary hierarchy built on Pascal VOC'2010 dataset. . . . . . . . . . 3.10 A hierarchy of concepts built on Pascal VOC'2010 dataset using TRUST-ME and the visual similarity. . . . . . . . . . . . . . . . . . . 3.11 A hierarchy of concepts built on Pascal VOC'2010 dataset using TRUST-ME and a similarity measure based on the shortest path in WordNet between the considered concepts. . . . . . . . . . . . . . . . 3.12 Heat maps illustrating the semantic affinity matrix of the visual similarity, conceptual similarity, contextual similarity and SVRC similarity.

vant with respect to the image domain. . . . . . . . . . . . . . . . . . xiv 3.4

  2.1 Comparison of image classification approaches. . . . . . . . . . . . . . 2.2 Comparison of the different families of approaches for image annotation. 2.3 Comparison of the different categories of approaches for image annotation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Comparative table of the different hierarchies used for image annotation. 3.2 Top correlated concepts according to image modalities. . . . . . . . .

Table 2 .

 2

1: Comparison of image classification approaches.

Table 2 .

 2 2: Comparison of the different families of approaches for image annotation.

		Semantic level (visual vs. semantics)	Reasoning capabilities (implicit vs. explicit Knowledge)	Scalability number)	(image	Scalability (concept number)
	Visual CBIR	-	-	++++		-
	Concept Detector	++	+	++++		+
	Supervised Learning	++	+	++		++
	Unsupervised Learning	+	+	+++		+++
	Semi-Supervised Lear.	+	+	+++		++
	Hierarchical Classif.	++	++	++++		+++
	Information Fusion	++	++	+++		+++
	LWO	+++	+++	+++		+++
	HWO	++++	+++	+		+
	Formal Ontologies	++++	++++	++++		++

Table 2 .

 2 3: Comparison of the different categories of approaches for image annotation.

  Rules for the Hierarchy Building . . . . . . . . . . . . . . 3.5 Experimental Results . . . . . . . . . . . . . . . . . . . . . 3.5.1 Visual Representation of Images . . . . . . . . . . . . . . . 3.5.2 Impact of Weighting on the Building of Hierarchies . . . . . 3.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . .

	Chapter 3
	Building Semantic Hierarchies
	Faithful to Image Semantics
	Part I
	Building Structured Knowledge
	Models Dedicated to Image
	Annotation

Contents 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Motivation for Using Semantic Hierarchies . . . . . . . . 3.2.1 Language-Based Hierarchies . . . . . . . . . . . . . . . . . . 3.2.2 Visual Hierarchies . . . . . . . . . . . . . . . . . . . . . . . 3.2.3 Semantic Hierarchies . . . . . . . . . . . . . . . . . . . . . . 3.2.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Proposed Measure: Semantico-Visual Relatedness of Concepts (SVRC ) . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Problem Formalization . . . . . . . . . . . . . . . . . . . . . 3.3.2 Visual Similarity Between Concepts . . . . . . . . . . . . . 3.3.3 Conceptual Similarity . . . . . . . . . . . . . . . . . . . . . 3.3.4 Contextual Similarity . . . . . . . . . . . . . . . . . . . . . 3.3.5 Computation of the Semantico-Visual Relatedness of Concepts 3.4

Table 3

 3

	Chapter 3. Building Semantic Hierarchies Faithful to Image Semantics

.1: Comparative table of the different hierarchies used for image annotation.

  Chapter 3. Building Semantic Hierarchies Faithful to Image Semantics

							Concept
					Abstraction			Organism
		Instrumentality		Whole		Conveyance		Person			Vertebrate
	Bottle	Tv_Monitor	Pot_Plant	Furniture	Car	Craft	Public_Transport	Wheeled_Vehicle	Bird	Carnivore	Bovid
		Sofa	Dining_Table	Chair	Boat	Aeroplane	Train	Bus	Bicycle	Motorbike	Cat	Dog	Sheep	Horse	Cow
	Figure 3.8: The semantic hierarchy built on Pascal VOC'2010 dataset using the
	proposed method. Double octagon nodes are the original concepts and the diamond
	one is the root of the produced hierarchy.

  .2, we illustrated some examples of the obtained results by the visual, conceptual, contextual and SVRC measures, and where we have

			Closest(Concept)
	Concept	Visual	Conceptual	Contextual	SVRC
	Aeroplane	Boat	Car	-	Boat
	Bicycle	Motorbike	Car	Motorbike	Motorbike
	Bird	Bottle	Train	-	Train
	Boat	Bus	Car	-	Bus
	Bottle	Bird	Cat	Dining_Table Dining_Table
	Bus	Boat	Motorbike	Car	Motorbike
	Car	Chair	Train	Bus	Bus
	Cat	Chair	Dog	Sofa	Dog
	Chair	Car	Train	Dining_Table Dining_Table
	Cow	Sheep	Sheep	-	Sheep
	Dining_table	Pot_Plant	Train	Chair	Pot_Plant
	Dog	Car	Cat	-	Cat
	Horse	Motorbike	Sofa	Person	Sofa
	Motorbike	Bicycle	Bus	Car	Bicycle
	Person	-	Car	Bottel	Car
	Pot_plant	Dining_Table	Cat	Chair	Dining_Table
	Sheep	Cow	Cow	-	Cow
	Sofa	Dining_Table	Horse	Chair	Dining_Table
	Train	Motorbike	Car	-	Motorbike
	Tv_monitor	Train	Bus	Chair	Sofa
	Table 3.2: Top correlated concepts according to image modalities.

  the value of a kernel function for the training sample x v k and the test sample x v i , y k ∈ {1, -1} is the class label of x v k , α k is the learned weight of the training sample x v k , and b is a learned threshold parameter. Radial Basis Function (RBF) kernel is again used for the training of our SVM:

Table 6 .

 6 1: Comparison of our method for image annotation with the one of[START_REF] Zhou | Image classification using super-vector coding of local image descriptors[END_REF] on Pascal VOC'2009 dataset.

	(AP)

More than

images were loaded on Facebook every second during 2011 [Pixable Blog, 2011]. 2 Flickr (http://www.flickr.com/) exceeded 5 billion hosted pictures in late 2010. 3 Picasa (https://picasaweb.google.com/) exceeded 7 billion hosted pictures in late 2010.

Iconography: is the branch of art history which studies the identification, description, and the interpretation of the content of images: the subjects depicted, the particular compositions and details used to do so, and other elements that are distinct from artistic style.[Wikipedia-Iconography, 2012]

http://www.gettyimages.com/ Compared to Flickr and Picasa, Getty Images provides well detailed and less subjective image annotations.

Surrounding context: We refer to surrounding context as the textual information surrounding a given image, such as user textual annotations, HTML tags, metadata or contextual text surrounding the image.

http://images.google.fr/

www.bing.com/images/

"Different persons, or the same person under different circumstances, may perceive the same visual content differently. This is called human perception subjectivity"[START_REF] Rui | Relevance feedback techniques in interactive content-based image retrieval[END_REF]

There is a general consensus in which the semantic gap would not be bridged for the near future, and it is safe to use 'narrow' (instead of 'bridge') the semantic gap.

http://www.wikipedia.org/

http://www.wikipedia.org/

Blobs are regions of an image that are somehow coherent.

A topic is a latent variable which represents a distribution over words. These approaches are known as 'topic models'.

two views of an item, for example: image and HTML text

In this definition, we refer to a visual class as an abstract node in the hierarchy without any associated label, whereas concept is a node in the hierarchy with a given associated label.

Late fusion: consists in merging the monomedia similarity profiles by means of aggregation functions (e.g mean average of both modalities)[START_REF] Escalante | Late fusion of heterogeneous methods for multimedia image retrieval[END_REF], Choi et al., 2010, Kulkarni et al., 2011].

This categorization is based on the level of expressiveness of the used knowledge model.

In this dissertation, we make difference between concept hierarchies and semantic hierarchies within the image retrieval context. We refer to the former as a hierarchy illustrating the subsumption relationship between concepts from the conceptual perspective, while the second illustrates the same relationship from the (image) semantics perspective and should therefore take account of the visual similarity of concepts. More details are given in Chapter 3.

A formal ontology is an ontology which is defined by axioms in a formal language.

The representation of each single real world object is split into many axioms about concepts and roles, leading to an overall design that is very difficult to apprehend[START_REF] Spaccapietra | On spatial ontologies[END_REF].

Examples of textual information used for building hierarchies are: tags, surrounding context, WordNet, Wikipedia, etc.

In the rest of this dissertation, we refer to the information extracted from these textual sources as conceptual information or also conceptual semantics.

Synonym Set: atomic component of WordNet, composed of a group of interchangeable words denoting a particular purpose or meaning. The meaning of the synsets is further clarified with short defining glosses. A concept may correspond to one or more synsets.

http://www.lscom.org/

Glosses: each synset contains a short defining glosses (definitions and/or example sentences).

A specific-context is the precise domain application in which data may be interpreted. The interpretation may be subjective to this domain.

A heat map is a graphical representation of data where the individual values contained in a matrix are represented as colors. Heat maps allow to visualize the proximity of the individuals based on a range of preset color.

In these examples, we consider a result as sufficiently relevant if it remains coherent with respect to the all these criteria: i) general context of appearing in images, ii) the visual similarity, and iii) the conceptual consistency.

http://www.loa.istc.cnr.it/DOLCE.html

Rules: statements in the form of an if-then sentence that describe the logical inferences that can be drawn from an assertion in a particular form.

Axioms: assertions (including rules) in a logical form that together comprise the overall theory that the ontology describes in its domain of application.

Mining the image annotations (provided in the metadata) to gather useful information about images context, namely the spatial and the contextual knowledge.

http://owlapi.sourceforge.net/index.html

n, m are natural numbers, such that n ≥ 0, m > 0. d is an unary fuzzy domain predicate.

⊲∈ {>, ≥}, ⊲⊳∈ {>, <, ≥, ≤}

In lexical semantics, opposites are words that lie in an inherently incompatible binary relationship. The notion of incompatibility here refers to the fact that one word in an opposite pair entails that it is not the other pair member [Wikipedia-Opposite_Semantics

, 2012].

[START_REF] Maillot | Ontology based complex object recognition[END_REF]] is a One-Class classification approach.

⊑: stands for the subsumption ("is-a") relationship.

A candidate annotation P consists of a set of candidate concepts {c j ∈ C ∪ C ′ , j = 1..n ii } and their confidence values {α j , j = 1..n ii }, predicted as describing the image content.

Acknowledgements

Both explanations and membership degrees are meaningful with respect to a given application. However, the product semantics allows having a more meaningful membership value for our target application than the one produced by Zadeh semantics. Suppose for example, that α 1 , α 2 , and α 3 are produced as a result of an image classification process, or an object detection one. Therefore, it would be more accurate to compute the membership degree of the image i to the class BeachImage as the product of the confidence values of these classifiers. This property is reachable by the use of product semantics.

Building our Multimedia Ontology

Main Concepts of our Ontology

Proposed concepts. For the building of our multimedia ontology and for achieving the reasoning tasks about image annotations, we propose to define the following main concepts:

• "Thing" represents the top concept (⊤) of the ontology,

• "Concept" is the generic concept in our ontology to represent a concept from the annotation vocabulary, i.e. any concept c j ∈ C ∪ C ′ used to describe the content of an image.

• "Image" is the generic concept to represent an image, i.e. each image i i of the database will be considered as an instance of the concept "Image" with a satisfiability degree of 1 ( i i : Image, 1 ).

• "Annotation" is a generic concept introduced to represent a given annotation, i.e. a set of concepts as a whole. We will come back on this notion later.

Definition of the RBox

As stated previously, our intent is to design an ontology of spatial and contextual information dedicated to reasoning about the consistency of image annotation. We therefore define in Table 4.2 the proposed roles, and their properties, that constitute the RBox of our multimedia ontology. These roles can be categorized as contextual relationships (the light-orange rows) and spatial relationships (the light-blue rows), and are detailed, respectively in Section 4. 5.4.1 and in Section 4.5.4.2. The choice of these specific roles is motivated by the expected (and conceived) reasoning scenarios in order to improve the problem of image annotation -cf. Section 4.7 and Chapter 6. For the clarity of presentation, the semantics of each of the proposed roles in Table 4.2 will be introduced in the section that corresponds to it. Problem Formalization.

In a formal way, given:

• DB, an image database consisting of a set of pairs image/textual annotation , i.e.

, where:

-

-L is the number of images in the database.

-C = c 1 , c 2 , • • • , c N is the annotation vocabulary used for annotating images in I,

-N is the size of the annotation vocabulary.

-A i is a textual annotation consisting of a set of concepts {c j ∈ C, j = 1..n i i } associated with a given image i i ∈ DB.

• SH, a semantic hierarchy consisting of a set of

Our objective is to train a set of (|C ∪ C ′ | -1) classifiers, where each classifier is associated with a concept c i ∈ {C ∪ C ′ }, and is trained using the following image samples:

-where label(i i ) is a function that returns the set of concepts associated with the image i i ∈ DB, ⊥= ∅.

Our approach is motivated by the above assumption. Indeed, we propose in the following a multi-stage reasoning framework based on the multimedia ontology proposed in Chapter 4. The proposed framework allows reasoning on the provided annotations by the image classification algorithm in order to achieve a semantically relevant image annotation. A global overview of the proposed approach is illustrated in Figure 6.1. Our Framework for image annotation Figure 6.1: A global overview of our approach, consisting of two frameworks. The first framework allows building multimedia ontologies dedicated to image annotation, while the second allows reasoning on the image annotation in order to achieve a semantically relevant annotation.

Hierarchical Image Classification

Specifically, we consider the following problem. Given a formal multimedia ontology designed as a fuzzy knowledge base KB = T ,R,A , where T is a fuzzy Terminological Box (TBox ), R is a regular fuzzy Role Box (RBox ), and A is a fuzzy Assertional Box (ABox ). This fuzzy knowledge base is assumed to contain the following explicit knowledge about the ontology concepts: i) subsumption relationships, ii) contextual relationships, and iii) spatial relationships (as for instance the proposed multimedia ontology in Chapter 4).

This multimedia ontology is then used within our framework for annotating previously unseen images. As illustrated in Figure 6.1, this is achieved by the following steps:

• A hierarchical classification is performed on the input image, and the confidence score for each concept c j ∈ C ∪ C ′ is recovered.

• These concepts and their confidence scores are thereafter transformed into fuzzy description logics assertions, and their consistency is checked using the subsumption relationships and our fuzzy DL reasoner. Inconsistent concepts are removed from the candidate annotation 1 of the input image. -1.040, Furniture: -0.135661, Instrumentality: -0.659, Organism: 0.636, Pub-lic_transport: -0.377, Seat: -0.243, Ungulate: 0.391, Vertebrate: 0.056, Wheeled_vehicle: -0.088, Whole: -0.106 Abstraction: 1.098, Bovid: -0.976, Carnivore: 0.875, Conveyance: 0.033, Craft: -0.671, Furniture: 1.229, Instrumentality: 0.785, Organism: 0.488, Public_transport: -0.108, Seat: 0.361, Ungulate: -0.682, Vertebrate: 0.508, Wheeled_vehicle: -0.294, Whole: 1.065 Abstraction: 1.072, Bovid: -0.368, Carnivore: -0.049, Conveyance: -1.077, Craft: -1.446, Furniture: 1.145, Instrumentality: 0.775, Organism: 0.647, Pub-lic_transport: -0.185, Seat: 0.513, Ungulate: -0.202, Vertebrate: -0.138, Wheeled_Vehicle: 0.020, Whole: 1.179

Reasoning on the annotations using the subsumption hierarchy: Consequently, with respect to the contextual information, the best explanation for the left image in Figure 6.3 is:

Please note that, since most images of the Pascal VOC dataset contain only one or two concepts [START_REF] Choi | Exploiting hierarchical context on a large database of object categories[END_REF], and thus the distribution of multi-labeled images is not uniform, we computed the Equation 4.8 for this dataset as:

where Λ = |Annotation i |.

Reasoning on Image Annotation using Spatial Information

Contextual knowledge can help the recognition of objects within a scene by providing predictions about objects that are most likely to appear in a specific setting, i.e. topological information, along with the locations that are most likely to contain objects in the scene, i.e. spatial information. Specifically, the spatial arrangement of objects provides important information for the recognition and interpretation tasks, and allows to solve ambiguity between objects having a similar appearance. In Chapter 4 -Section 4.7, we have proposed a usage scenario illustrating the usefulness of spatial information and reasoning over this kind of knowledge in order to improve image annotation.

As part of this work, we proposed an approach based on image classification for annotating images. Consequently, we do not dispose of the spatial position of make it suitable for the image domain [Bannour and Hudelot, 2013a].

Significance:

• The building of our ontology is fully automatic and allows therefore reducing the scalability problem of (formal) ontologies building.

• The specification of our ontology and the represented knowledge are not extracted from an existing commonsense ontology, but gathered by the mining of image databases. Consequently, the built ontology is representative of image semantics and is domain-independent.

• Our multimedia ontology is designed as a knowledge base and allows terminological and assertional reasoning. Specifically, it allows performing fuzzy reasoning in order to reduce the uncertainty about image annotations.

Improving Image Annotation using Semantic Hierarchies

Image annotation has been considered in the last decade as a multi-class classification problem, i.e. finding the belonging of a considered image to a given set of concepts (or classes). Recently, many approaches have proposed to use semantic hierarchies in order to improve image annotation. These hierarchies was used as a framework for hierarchical image classification, and thus to improve classifiers accuracy and to reduce the complexity of managing large scale data. In this dissertation, we investigated the contribution of semantic hierarchies for hierarchical image classification. We proposed therefore a new method based on the hierarchy structure to train efficiently hierarchical classifiers. Our method, named One-Versus-Opposite-Nodes, allows decomposing the problem into several independent tasks and therefore it scales well with large databases. We also proposed two methods for computing a hierarchical decision function that served to annotate new image samples. The former was performed using a top-down classifiers voting, while the second was based on a bottom-up score fusion [Bannour and[START_REF] Bannour | Hierarchical image annotation using semantic hierarchies[END_REF]Hudelot, 2013b].

Significance:

⇒ Previous approaches for hierarchical image classification have proposed to build hierarchies by the recursive partitioning of the set of classes, or by the agglomerative clustering of the classes, resulting in binary trees like structures. In this dissertation, we showed that the hierarchy depth has a serious impact on the classification accuracy, and therefore we proposed to not restrict the image classification to binary structures.

⇒ The proposed methods draw profits from the semantic structure of the hierarchy to decompose the image classification problem into several independent and complementary sub-tasks, resulting in a lower complexity and a higher accuracy for the image classification results.

Appendix 1: Publications of the Author

International Journals with Peer-review:

(1) Building and Using Fuzzy Multimedia Ontologies for Semantic Image Annotation, MTAP 2013, Multimedia Tools and Applications, to appear, doi={10.1007/s11042-013-1491-z}.

( National Journals with Peer-review: International Conferences with Peer-review:

(