Marian-Andrei Rizoiu

Julien Velcin

Stéphane Lallich

Semantic-enriched Visual Vocabulary Construction in a Weakly Supervised Context

Keywords: bag-of-features representation, visual vocabulary construction, semantic-enriched representation, image numerical representation, semi-supervised learning

One of the prevalent learning tasks involving images is content-based image classification. This is a difficult task especially because the low-level features used to digitally describe images usually capture little information about the semantics of the images. In this paper, we tackle this difficulty by enriching the semantic content of the image representation by using external knowledge. The underlying hypothesis of our work is that creating a more semantically rich representation for images would yield higher machine learning performances, without the need to modify the learning algorithms themselves. The external semantic information is presented under the form of non-positional image labels, therefore positioning our work in a weakly supervised context. Two approaches are proposed: the first one leverages the labels into the visual vocabulary construction algorithm, the result being dedicated visual vocabularies. The second approach adds a filtering phase as a pre-processing of the vocabulary construction. Known positive and known negative sets are constructed and features that are unlikely to be associated with the objects denoted by the labels are filtered. We apply our proposition to the task of content-based image classification and we show that semantically enriching the image representation yields higher classification performances than the baseline representation.

Introduction

The large scale production of image data has been facilitated in modern days by the maturing of the image acquisition, storing, transmission and reproduction devices and techniques. The Web 2.0 allowed easy image sharing and recently even search capabilities (e.g., Instagram2 , Flickr3 ). Social Networks rely heavily on image sharing. Because of the sheer volumes of created images, automatic summarization, search and classification methods are required.

The difficulty when analyzing images comes from the fact that digital image numerical formats poorly embed the needed semantic information. For example, images acquired using a digital photo camera are most often stored in raster format, based on pixels. A pixel is an atomic image element, which has several characteristics, the most important being the size (as small as possible) and its color. Other information can be color coding, alpha channel etc.. Therefore, an image is stored numerically as a matrix of pixels. The difficulty raises from the fact that low-level features, such as position and color of individual pixels, do not capture too much information about the semantic content of the image (e.g., shapes, objects). This problem is also known as the semantic gap between the numerical representation of the image and its intended semantics. To address this issue, multiple representation paradigms have been proposed, some of which will be presented in Section 2. The one showing the most promising results is the "bag-of-features" representation, a representation inspired from the textual "bag-of-words" textual representation. Whatsoever, the results obtained by the state-of-the-art image representations still leave plenty of room for improvements. One of the privileged tracks to closing the semantic gap is to take into account additional information stored in other types of data (e.g., text, labels, ontologies of concepts) associated with the images. With today's Web, additional information of this type is often available, usually created by anonymous contributors. Our work presented in this paper is targeted towards improving a baseline, unsupervised, image description strategy by rendering it semi-supervised, in order to take into account user-generated additional information. The purpose is to capture more of the semantics of an image in its numerical description and to improve the performances of an image-related machine learning task.

An overview of our proposals The focus of the work is embedding semantic information into the construction of image numerical representation. The task of contentbased image classification is used only to assess the validity of our proposals. The content-based image classification literature provides examples (some of which are mentioned in Section 2) of systems which achieve good results. Our objective is not to compare with these approaches or show the superiority of our methods on well-known image benchmarks. Neither we do not propose a new image representation system. The objective of our work is to show how embedding semantics into an existing image representation can be beneficial for a learning task, in this case image classification. Starting from the baseline image representation construction present in Section 1.1, we propose two algorithms that make use of external information to enrich the semantics of the image representation. The external information is under the form of non-positional labels, which signal the presence in the image of an object (e.g., car, motorcycle) or give information about the context of the image (e.g., holiday, evening), but do not give any information about its position of the image (in the case of objects). Furthermore, the labels are available only for a part of the image collection, therefore positioning our work in a semi-supervised learning context. We use both the baseline representation and our semantically improved representation in an image classification task and we show that leveraging semantics consistently provides higher scores.

Our work is focused on the visual vocabulary construction (which is also referred in the literature as codebook or model). In the "bag-of-features" (BoF) representation, the visual words serve a similar role as the real textual words do in the "bag-of-words" representation. We propose two novel contributions that leverage external semantic information and that allow the visual vocabulary to capture more accurately the semantics behind a collection of images. The first proposal deals with introducing the provided additional information early in the creation of the visual vocabulary. A dedicated visual vocabulary is constructed starting from the visual features sampled from images labeled with a given label. Therefore, a dedicated vocabulary contains visual words adapted to describing the object denoted by the given label. In the end, the complete visual vocabulary is created by merging the dedicated vocabularies. In the second proposal, we add a filtering phase as a pre-processing of the visual vocabulary construction. This reduces the influence of irrelevant features on the visual vocabulary construction, thus enabling the latter to be more adapted to describe the semantics of the collection for images. For any given image, we construct a known positive set (images labeled with the same labels as the given image) and a known negative set (images that do not share any labels with the given image). If a visual feature, sampled from the target image, is more similar to features in the known negative set than to features in the known positive set, then there are high chances that it does not belong to the objects denoted by the labels of the given image and it can, therefore, be eliminated. As our experiments in Section 4.5 show, this approach increases the overall accuracy of the image-related learning task. The two approaches are combined into a visual vocabulary construction technique and shown to consistently provide better performances than the baseline technique presented in Section 1.1.

The layout of this article

The remainder of this paper is structured as follows: the rest of this section presents how to construct a baseline "bag-of-features" image description (in Section 1.1). In Section 2, we present a brief overview on constructing a numerical image representation, concentrating on some of the state-of-the-art papers that relate to visual vocabulary construction and knowledge injection into image representation. Section 3 explains our two proposals, followed, in Section 4, by the experiments that were performed. Some conclusions are drawn and future work perspectives are given in Section 5.

Baseline "bag-of-features" image numerical description

The "bag-of-features" [START_REF] Csurka | Visual categorization with bags of keypoints[END_REF][START_REF] Zhang | Local features and kernels for classification of texture and object categories: A comprehensive study[END_REF] (BoF) representation is an image representation inspired from the "bag-of-words" (BoW) textual representation. The BoW representation is an orderless document representation, in which each document is depicted by a vector of frequencies of words over a given dictionary. BoF models have proven to be effective for object classification [START_REF] Csurka | Visual categorization with bags of keypoints[END_REF][START_REF] Willamowski | Categorizing nine visual classes using local appearance descriptors[END_REF], unsupervised discovery of categories [START_REF] Fei-Fei | A bayesian hierarchical model for learning natural scene categories[END_REF][START_REF] Quelhas | Modeling scenes with local descriptors and latent aspects[END_REF][START_REF] Sivic | Discovering objects and their location in images[END_REF] and video retrieval [START_REF] Chavez | An interactive video content-based retrieval system[END_REF][START_REF] Sivic | Video google: A text retrieval approach to object matching in videos[END_REF]. For object recognition tasks, local features play the role of "visual words", being predictive of a certain "topic" or object class. For example, a wheal is highly predictive of a bike being present in the image. If the visual dictio-nary contains words that are sufficiently discriminative when taken individually, then it is possible to achieve a high degree of success for whole image classification. The identification of the object class contained in the image is possible without attempting to segment or localize that object, simply by looking which visual words are present, regardless of their spatial layout. Overall, there is an emerging consensus in recent literature that BoF methods are effective for image description [START_REF] Zhang | Local features and kernels for classification of texture and object categories: A comprehensive study[END_REF].

Image Sampling

Feature Description

Visual Vocabulary Construction Assign Features to Visual Words

Image Dataset "Bag-offeatures" representation Baseline construction Typically, constructing a BoF image representation is a four phase process, as shown in Figure 1. Starting from a collection P containing n images, the purpose is to translate the images into a numerical space, in which the learning algorithm is efficient. In phase 1, each image p i ∈ P is sampled and l i patches (features) 4 are extracted. Many sampling techniques have been proposed, the most popular being dense grid sampling [START_REF] Fei-Fei | A bayesian hierarchical model for learning natural scene categories[END_REF][START_REF] Vogel | Semantic modeling of natural scenes for content-based image retrieval[END_REF] and salient keypoint detector [START_REF] Csurka | Visual categorization with bags of keypoints[END_REF][START_REF] Fei-Fei | A bayesian hierarchical model for learning natural scene categories[END_REF][START_REF] Sivic | Discovering objects and their location in images[END_REF]. In phase 2, using a local descriptor, each feature is described using a h-dimensional5 vector. The SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] and the SURF [START_REF] Bay | Surf: Speeded up robust features[END_REF] descriptors are popular choices. Therefore, after this phase, each image p i is numerically described by V i ⊂ R h , the set of h-dimensional vectors describing features sampled from p i .

Based on these numeric features, in phase 3, a visual vocabulary is constructed using, for example, one of the techniques presented in Section 2.2. This is usually achieved by means of clustering of the described features, and the choice is usually the K-Means clustering algorithm, for its linear execution time required by the high number of features. The visual vocabulary is a collection of m visual words, which are described in the same numerical space as the features and which serve as the bases of the numerical space in which the images are translated. More precisely, the centroids created by the clustering algorithm serve as visual words. In clustering, centroids are the abstractions of a group of documents, therefore summarizing the common part of the documents. In the above example, all the visual features extracted from the region of an image depicting the wheal of a bike will be regrouped together into one or several clusters. The centroid of each cluster represents a visual word, which is associated with the wheal. Figure 2, we depict three examples of images portraying bikes. In each image, we highlight 3 features: two corresponding to visual words associated with "wheal" and one associated with a visual word associated with "exhaust pipe".

In phase 4, each sampled feature is assigned to a visual word. Similarly to the BoW numerical description for texts, each image is described as a distribution over the visual words, using one of the term weighting scheme (e.g., tf , tf xidf etc.). In the previous example, the distribution vector associated with each of the images in Figure 2 has a high count for the visual words associated with "wheal", "exhaust pipe", and "sadle". The resulting numerical description can then be used for classification, information retrieval or indexation tasks.

Context and related work

Over the past decades computer vision domain has seen a large interest from the research community. Its application are larger than image analysis and include augmented reality, robotic vision, gesture recognition etc. Whatsoever, in the context of Internet-originating images, one of the prevailing task is content-based image classification. Some of the initial image classification systems used color histograms [START_REF] Swain | Color indexing[END_REF] for image representation. Such a representation does not retain any information about the shapes of objects in images and obtains moderate results. Other systems [START_REF] Haralick | Computer classification of reservoir sandstones[END_REF][START_REF] Lazebnik | A sparse texture representation using affine-invariant regions[END_REF][START_REF] De Medeiros Martins | A new method for multi-texture segmentation using neural networks[END_REF][START_REF] Varma | Texture classification: Are filter banks necessary? In: Computer Vision and Pattern Recognition[END_REF] rely on texture detection. Texture is characterized by the repetition of basic elements or textons. For stochastic textures, it is the identity of the textons, not their spatial arrangement, that matters. The BoF orderless representation has imposed itself as the state-of-the-art in image representation, for classification and indexation purposes. The process of constructing the representation includes sampling the image (phase 1 in Figure 1), describing each features using an appearance-based descriptor (phase 2), constructing a visual vocabulary (phase 3) and describing images as histograms over the visual words (phase 4). The remainder of this section presents a brief overview (i) of the sampling strategies and numerical descriptors for image keypoints present in literature (in Section 2.1) and (ii) of the visual vocabulary construction techniques, concentrating on how external information can be used to improve the vocabularies representativity (in Section 2.2).

Sampling strategies and numerical description of image features

Image sampling methods Image sampling for the BoF representation is the process of deciding which regions of a given image should be numerically described. In Figure 1, it corresponds to phase 1 of the construction of a BoF numerical representation. The output of feature detection is a set of patches, identified by their locations in the image and their corresponding scales and orientations. Multiple sampling methods exist [START_REF] O'hara | Introduction to the bag of features paradigm for image classification and retrieval[END_REF], including Interest Point Operators, Visual Saliency and random or dense grid sampling.

Interest Point Operators [START_REF] Kadir | Saliency, scale and image description[END_REF][START_REF] Lowe | Object recognition from local scale-invariant features[END_REF] search to find patches that are stable under minor affine and photometric transformations. Interest point operators detect locally discriminating features, such as corners, blob-like regions, or curves. A filter is used to detect these features, measuring the responses in a three dimensional space. Extreme values for the responses are considered as interest points. The popular choice is the Harris-Affine detector [START_REF] Mikolajczyk | Scale & affine invariant interest point detectors[END_REF], which uses a scale space representation with oriented elliptical regions. Visual Saliency [START_REF] Frintrop | Computational visual attention systems and their cognitive foundations: A survey[END_REF] feature detectors are based on biomimetic computational models of the human visual attention system. Less used by the BoF literature, these methods are concerned with finding locations in images that are visually salient. In this case, fitness is often measured by how well the computational methods predict human eye fixations recorded by an eye tracker. There are research [START_REF] Sivic | Video google: A text retrieval approach to object matching in videos[END_REF] that argue that interest point-based patch sampling, while useful for image alignment, is not adapted for image classification tasks. Examples are city images, for which the interest point detector does not consider relevant most of the concrete and asphalt surroundings, but which are good indicators of the images' semantics. Some approaches sample patches by using random sampling [START_REF] Maree | Random subwindows for robust image classification[END_REF]. [START_REF] Nowak | Sampling strategies for bag-of-features image classification[END_REF] compare a random sampler with two interest point detectors: Laplacian of Gaussian [START_REF] Lindeberg | Detecting salient blob-like image structures and their scales with a scalespace primal sketch: a method for focus-of-attention[END_REF] and Harris-Laplace [START_REF] Lazebnik | Affine-invariant local descriptors and neighborhood statistics for texture recognition[END_REF]. They show that, when using enough samples, random sampling exceeds the performance of interest point operators. Spatial Pyramid Matching, proposed in [START_REF] Lazebnik | Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories[END_REF], introduces spacial information in the orderless BoF representation by creating a pyramid representation, where each level divides the image in increasingly small regions. Feature histogram is calculated for each of these regions. The distance between two images using this spatial pyramid representation is a weighted histogram intersection function, where weights are largest for the smallest regions.

Feature descriptors With the image sampled and a set of patches extracted, the next questions is how to numerically represent the neighborhood of pixels near a localized region. In Figure 1, this corresponds to phase 2 of the construction of a BoF numerical representation. Initial feature descriptors simply used the pixel intensity values, scaled for the size of the region. The normalized pixel values have been shown [START_REF] Fei-Fei | A bayesian hierarchical model for learning natural scene categories[END_REF] to be outperformed by more sophisticated feature descriptors, such as the SIFT descriptor.

The SIFT (Scale Invariant Feature Transform) [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] descriptor is today's most widely used descriptor. The responses to 8 gradient orientations at each of 16 cells of a 4x4 grid generate the 128 components of the description vector. Alternative have been proposed, such as the SURF (Speeded Up Robust Features) [START_REF] Bay | Surf: Speeded up robust features[END_REF] descriptor. The SURF algorithm contains both feature detection and description. It is designed to speed up the process of creating features similar to those produced by a SIFT descriptor on Hessian-Laplace interest points by using efficient approximations.

Unsupervised visual vocabulary construction

The visual vocabulary is a mid-level transition key between the low-level features and a high-level representation. It is a prototypic representation of features that are discriminative in a classification context. The visual vocabulary is used to reduce dimensionality and to create a fixed length numerical representation for all images 6 . Most BoF approaches use clustering to created the visual vocabulary, usually the K-Means [START_REF] Jiang | Towards optimal bag-of-features for object categorization and semantic video retrieval[END_REF][START_REF] Lazebnik | Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories[END_REF][START_REF] Sivic | Video google: A text retrieval approach to object matching in videos[END_REF] algorithm. K-Means is used for the fact that it produces centroids, which are prototypes of similar features in the same cluster. Its linear execution time is a plus considering the high volume of individuals to be processed [START_REF] Pham | Factorial correspondence analysis for image retrieval[END_REF]. Some authors [START_REF] Jurie | Creating efficient codebooks for visual recognition[END_REF] argument that in K-Means, centroids are attracted by dense regions and under-represent less denser, but equally informative regions. Therefore, methods were proposed for allocating centers more uniformly, inspired by mean shift [START_REF] Comaniciu | Mean shift: A robust approach toward feature space analysis[END_REF] and on-line facility location [START_REF] Meyerson | Online facility location[END_REF]. Other visual vocabulary construction techniques do not rely on K-Means. For example, [START_REF] Moosmann | Fast discriminative visual codebooks using randomized clustering forests[END_REF] use an Extremely Randomized Clustering Forest, an ensemble of randomly created clustering trees. This technique provides good resistance to background clutter, but the main advantage over K-Means is the faster training time.

One of the most important parameters in the construction of the visual vocabulary is its dimension, which has a powerful impact on both performance and computational complexity [START_REF] Csurka | Visual categorization with bags of keypoints[END_REF][START_REF] Jurie | Creating efficient codebooks for visual recognition[END_REF]. It has been shown [START_REF] Jiang | Towards optimal bag-of-features for object categorization and semantic video retrieval[END_REF][START_REF] López-Sastre | Towards a more discriminative and semantic visual vocabulary[END_REF][START_REF] Nowak | Sampling strategies for bag-of-features image classification[END_REF]] that a large vocabulary may lead to overfitting for construction techniques based on interest points detection. As our experiments show (in Section 4.6), even a random vocabulary (in a random vocabulary, a number of features are randomly chosen to serve as visual words) can lead to overfitting if its dimension is high enough.

Leveraging additional information

The BoF representation yields surprising results for image classification and indexing. This is because there is an intrinsic relation between the "quantity" of semantic information captured by the description space and the performances of machine learning algorithms (e.g., in a classification task, the separability of individuals in the description space is crucial). Therefore, one direction to further improve results is to construct new representations that capture even more semantics from the raw image data. Another direction, the one that we privilege in our work, is to use external information to further enrich the semantic content of the constructed representation. In the case of Internet-originating images, precious information is given either by the textual context of images (e.g., titles, descriptions etc.), or by labels attached to the images (e.g., on social networks websites, users have the option to label the presence of their friends in images). Of course, the literature presents approaches that leverage other resources to semantically enrich enrich the image representation (e.g., [START_REF] Athanasiadis | Using a multimedia ontology infrastructure for semantic annotation of multimedia content[END_REF] propose a system that links low-level visual descriptors to high-level, domain-specific concepts in an ontology). In the following paragraphs, we detail some of the methods present in the literature that address the use of additional information under the form of text or labels in order to improve image classification results and we position our work relative to these approaches.

Leveraging the image's textual context In [START_REF] Morsillo | Semi-supervised learning of visual classifiers from web images and text[END_REF], the text that comes alongside the images is used to improve the visual query accuracy. A BoF representation for images is created as shown in Section 1.1, with the exception that color information is also added to the keypoint description. An 11-dimension vector coding the color information of the sampled patches is added to the 128-dimension vector generated by the SIFT. The text that surrounds the images in the web pages is used to extract topics, using LDA [START_REF] Blei | Latent dirichlet allocation[END_REF]. The inferred topics are, afterwards, used to describe the textual information (therefore functioning as a dimension reduction technique). The textual and the image data are used together to estimate the parameters of a probabilistic graphical model, which is trained using a small quantity of labeled data. Another approach that uses the text accompanying images originating from the Internet is presented in [START_REF] Wang | Building text features for object image classification[END_REF].

An auxiliary collection of Internet-originating images, with text attached, is used to create a textual description of a target image. Images are described using three types of features: the SIFT features, the GIST features [START_REF] Oliva | Modeling the shape of the scene: A holistic representation of the spatial envelope[END_REF] and local patch color information. For each test image, the K most similar images (in terms of visual features) are identified in the auxiliary collection. The text associated with these near neighbor images is summarized to build the text feature. The label of each image is considered as a unit (i.e., a whole phrase is considered as an item) and the text feature is constructed as a normalized histogram over labels. A text classifier and a visual classifier are trained and the outputs of the two classifiers are merged for a more accurate description of the photo. [START_REF] Mooney | Watch, listen & learn: Co-training on captioned images and videos[END_REF] use co-training [START_REF] Blum | Combining labeled and unlabeled data with co-training[END_REF] to construct a classifier starting from textual and visual data. Text is described using a BoW representation, whereas images are described using region-based features. Each image is divided into a number of regions of fixed dimension (4-by-6 pixels), which are described using texture and color features. Co-training is a semi-supervised classification technique, which first learns a separate classifier for textual data and image data, using any labeled examples. The most confident predictions of each classifier on the unlabeled data are then used to iteratively construct additional labeled training data, and the classifiers are re-trained.

Leveraging external semantic knowledge Other solutions rely on external expert knowledge in order to guide the visual vocabulary construction. This knowledge is most often expressed under the form of class/category annotations or labels (e.g. signaling the presence of an object inside an image), or semantic resources, such as Word-Net [START_REF] Miller | Wordnet: a lexical database for english[END_REF]. An iterative boosting-like approach is used in [START_REF] Zhang | Learning non-redundant codebooks for classifying complex objects[END_REF]. Each iteration of boosting begins by learning a visual vocabulary according to the weights assigned by the previous boosting iteration. The resulting visual vocabulary is then applied to encode the training examples, a new classifier is learned and new weights are computed. The visual vocabulary is learned by clustering using K-Means a "learning" subset of image features. Features from images with high weights have more chances of being part of the learning subset. To classify a new example, the AdaBoost [START_REF] Freund | A decision-theoretic generalization of on-line learning and an application to boosting[END_REF] weighted voting scheme is used.

[45] construct both a generic vocabulary and a specific one for each class. The generic vocabulary describes the content of all the considered classes of images, while the specific vocabularies are obtained through the adaptation of the universal vocabulary using class-specific data. Any given image can, afterwards, be described using the generic vocabulary or one of the class-specific vocabularies. A semi-supervised technique [START_REF] Ji | Towards semantic embedding in visual vocabulary[END_REF], based on Hidden Random Markov Fields, uses local features as Observed Fields and Semantic labels as Hidden Fields and employs WordNet to make correlations. Some works [START_REF] Fulkerson | Localizing objects with smart dictionaries[END_REF][START_REF] Hsu | Visual cue cluster construction via information bottleneck principle and kernel density estimation[END_REF][START_REF] Lazebnik | Supervised learning of quantizer codebooks by information loss minimization[END_REF][START_REF] Winn | Object categorization by learned universal visual dictionary[END_REF] use mutual information between features and class labels in order to learn class-specific vocabularies, by merging or splitting initial visual words quantized by K-Means. Another work [START_REF] Liu | Learning semantic visual vocabularies using diffusion distance[END_REF] presents an algorithm used for learning a generic visual vocabulary, while trying to preserve and use the semantic information in the form of a point-wise mutual information vector. It uses the diffusion distance to measure intrinsic geometric relations between features. Other approaches [START_REF] Marszałek | Spatial weighting for bag-of-features[END_REF] make use of label positioning in the images to distinguish between foreground and background features. They use weights for features, higher for the ones corresponding to objects and lower for the background.

Our positioning In the methods presented earlier, we identify several approaches towards improving the results of classification algorithms: (a) improving image repre-sentation semantics by combining multiple types of visual features (e.g., SIFT, color, texture etc., no external information is leveraged), (b) modifying the classification algorithm to take into account the text/label information (usually by training separate classifiers for (i) text and image or (ii) based on each label), (c) training and using multiple vocabularies to describe an image and (d) making use of positional labels to filter features unlikely to be relevant. Positional labels are labels in which the position of the objects in images are known, in addition to their presence. This kind of labeling is usually more costly to perform than non-positional labeling.

Our proposals deal with leveraging external information to enrich the semantics of the image representation. The additional information is taken into account at the level of the representation construction. We do not modify the learning algorithm, therefore our proposals are compatible with existing classification algorithm. Our proposals can be classified under the previously defined point (c), since we construct multiple dedicated visual vocabularies. To the best or our knowledge, the feature filtering proposal, detailed in Section 3.2, is the first algorithm aimed at filtering features irrelevant for an object, without making use of positional labels. This greatly reduces the effort of manually labeling and allows the usage of broader sources of data available on the Web.

Including semantic knowledge The semantic knowledge is presented under the form of a collection T of k labels, T = {t i |i = 1, 2...k}. Each label is considered to denote an object in the image (e.g., a car, a person, a tree), but no positional markers are available. We make the assumption that the objects denoted by labels do not overlap in the images and their appearance in the dataset is not correlated (e.g., if a car appears, it does not necessarily mean that there is a person next to it). While these are strong assumptions, we will discuss ways of relaxing them in Section 5. Furthermore, we consider the labeling to be complete (i.e., if an image does not have a given label, than the object does not appear in the image). In Section 3.2, we discuss in further detail the effects of incomplete labeling, after presenting our proposals.

Only a fraction of the image dataset is labeled and we use both labeled and unlabeled images to construct the semantic-aware representation, therefore positioning our work in the domain of semi-supervised learning. We denote by P the input collection, having n images. n 1 images are labeled, thus forming the labeled set (P 1 ), while the remaining images have no labels. The a priori label information is presented in the form of a boolean matrix Y ∈ {0, 1} n1×k , having n 1 lines and k columns so that y i,j = 1 if image p i ∈ P 1 is labeled using t j ; 0 otherwise.

Dedicated visual vocabulary generation

The idea behind the BoF representation is that the visual words are predictive for certain objects (as seen in Section 1.1). The quality of the visual words (and their predictive power) would be enhanced if they are constructed starting only from the features extracted from the respective objects. This would eliminate the background originating features and features belonging to other objects. In a weakly supervised context, the object boundaries are unknown, but selecting only the images that contain a certain object increases the relevant/noise feature ratio. Consequently, the resulted visual words are more accurate descriptions of the objects denoted by the labels. We propose to construct a dedicated visual vocabulary for each label t i ∈ T , starting only from features extracted from the images labeled with t i . The proposed method is presented in Algorithm 1. We make no assumptions about the number of visual words needed to describe each object and, therefore, visual words are distributed equally among objects. We construct k dedicated vocabularies, each one containing m/k visual words. Other division techniques can be imagined and make part of the perspectives of our work. Each dedicated vocabulary is created in the standard BoF approach, shown in Section 1.1. For a given label t i , we create C i , the collection of all the features extracted from images labeled with t i . Formally:

C i = n1 j=1 yj,i=1
V j where V j is the set of numerically described features sampled from image p j . The function choose features at random is used to initialize the dedicated vocabulary M i with m/k features randomly picked from C i . The function ameliorate using K-Means evolves the visual vocabulary M i by clustering the features in C i around the visual words, using the K-Means algorithm. The Euclidean distance is used to measure the similarity between the numeric descriptions of two features, since this is the distance Algorithm 1 Dedicated vocabulary generation algorithm.

Input: C = {V i | i = 1, 2..n 1 } -set of
C i = n1 j=1 V j | y j,i = 1
// set of features in images labeled with t i // construct dedicated visual vocabulary

M i M i ← choose features at random (m i , C i ) M i ← ameliorate using K-Means (M i , C i ) // merge the dedicated visual vocabularies M ← ∅ for i = 1 to k do M ← concatenate vocabularies(M , M i )
employed in the original work [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] that proposed the SIFT descriptor. Subsequently, it has been used by most of the literature to measure the similarity between features. The set of resulted visual words represent more accurately the object denoted by the label t i . At the end of the algorithm, the concatenate vocabularies function merges the dedicated vocabularies M i , i = 1, 2..k into the general visual vocabulary M . This ensures that the generated visual vocabulary contains visual words which describe all the objects labeled with labels in T .

Temporal complexity Algorithm 1 has a linear execution time, if we consider that matrix operations are indivisible and executed in O(1), which is the case in modern vectorial mathematical environments. Since we are executing K-Means k times, the temporal complexity will be no iter × k × O(m/k × n ti ), where n ti is the number of images labeled with t i and no iter is the number of performed iterations (usually limited, thus ignored in practice). That leads to a theoretical complexity of O(m × n), equal to that of K-Means.

Filtering irrelevant features

We propose a filtering mechanism in order to further increase the relevant/noise features ratio in the dedicated vocabulary construction technique presented in the previous Section 3.1: we detect and filter the features that are unlikely to be related to the object denoted by a given label. Given an image p i ∈ P 1 , we construct two auxiliary image collections: the known positive set, which contains only images that are labeled identically as p i , and the known negative set, which contains images that do not share any tags with p i (given the complete labeling assumption). In practice, we limit the sizes of the known positive set and the known negative set to a maximum number of images, given by a parameter maxF iles. We define KP pi as the set of features sampled from images in the positive set and KN pi as the set of features sampled from the negative

KP pi = {f + ∈ V j | ∀ t l ∈ T for which y i,l = 1 =⇒ y j,l = 1} KN pi = {f -∈ V j | ∀ t l ∈ T for which y i,l = 1 =⇒ y j,l = 0}
Consider a feature sampled from p i (f ∈ V i ), which is more similar to the features in the negative collection (f -∈ KN pi ) rather than the ones in the positive collection (f + ∈ KP pi ). Such a feature has a higher chance of belonging to the background of p i rather than to the objects in the image. It can, therefore, be filtered. To measure the similarity of two features, the euclidean distance is usually used:

||f 1 -f 2 || = Σ h i=1 (f 1,i -f 2,i ) 2 .
Formally, for a feature f sampled from an image p i :

f ∈ V i is filtered ⇔ f + ∈ KP pi so that ||f -f + || ≤ δ with δ = α × min f ∈KNp i ||f -f -|| (1)
where δ is the filtering threshold and α ∈ R + is a parameter, which allows the fine tuning of the filtering threshold. The filtering threshold δ is defined as the distance from the feature f to the closest feature in the known negative set, scaled by tuning parameter α. The influence of parameter α on the effectiveness of the filtering is studied in Section 4.7. A feature f is considered similar to a feature f + ∈ KP pi if and only if ||f -f + || is lower than the filtering threshold. Therefore, the feature f is removed when it has no similar feature in the known positive set. Let's take the example of image collection depicted in Figure 3. The images in Figures 3a and3b are labeled "motorbike", whereas the image in Figure 3c is labeled "city". The target image in Figure 3a has buildings in the background, and any feature sampled from that region of the image would be irrelevant for the object motorbike. Figure 3b serves as known positive set, while Figure 3c serves as known negative set. We take the example of two features: f 1 sampled from the wheal of the motorbike (shown in green) and f 2 sampled from the buildings in the background (shown in red), of the target image. For f 1 , at least one similar feature exists in the positive set. For f 2 , no similar features exist in the known positive set, its most similar feature being in the known negative set (shown in red in Figure 3c). f 2 is, therefore, eliminated as it is considered not relevant for the object motorbike.

Algorithm 2 presents the proposed filtering algorithm. The algorithm has two parameters maxF iles, which controls the maximum size of the KP pi and KN pi sets, and α, which controls how strict is the filtering. For each labeled image p i , the func-Algorithm 2 Filtering irrelevant features.

Input: C = {V i | i = 1, 2.
.n 1 } -set of features sampled from labeled images Input: Y ∈ {0, 1} n1×k -image/label association matrix Parameter: α -parameter controlling the filtering threshold Parameter: maxF iles -controls the size of the known positive and known negative sets Output: V f i , i = 1, 2..n 1 -sets of the filtered features in each labeled image // for each labeled image

for i = 1 to n 1 do V f i ← ∅ T i ← {t j | y i,j = 1} // the labels of image p i KP pi ← create KP(i, T i , Y , C, maxF iles) // KnownP ositive set KN pi ← create KN(i, T i , Y , C, maxF iles) // KnownN egative set // process each feature in current image p i for each f ∈ V i do δ ← α× min distance(f , KN pi ) count ← count similar(f , KP pi , δ) if count > 0 then V f i ← V f i {f }
tions create KP and create KN are used to construct the feature sets KP pi and, respectively, KN pi . The count similar function is used to count how many features in KP pi have the similarity distance lower than the filtering threshold. If there exists at least one such feature in the KP pi set, then f is added to V f i , the filtered feature set of p i .

Temporal complexity In Algorithm 2, for comprehension reasons, operations are presented for each feature f sampled from the image p i . In reality, in vectorial mathematical environments (e.g. Octave), matrix operations are unitary and they can be considered to be executed in O(1). Thus, the algorithm has a linear execution time O(n 1 × maxF iles).

Incomplete labeling

In the proposed approaches, as well as in the experiments presented in Section 4, we make the assumption of complete labeling: if an object occurs in an image, then it is sure that the image has its corresponding label attached. In the case of incomplete labeling, an object might appear in an image p, but the associate label t is not set for the image p. For the dedicated vocabulary construction, incomplete labeling has a limited impact, especially if the dataset is large enough. It only means that the image p is left out when constructing the vocabulary for label t. For the filtering proposal, missing labels mean that the image p has a chance of being selected for the known negative set for an image labeled with t. This translates into a very high filtering threshold. Still, this should not pose problems if the known positive set also contains images depicting the given object. A given feature needs to have only one similar feature in the known positive set to be considered representative for the object. Furthermore, considering that our algorithms are devised to work in a semisupervised context, a limited number of completely labeled images is required. This reduces considerably the manual labeling effort.

Experiments and results

As already pointed out in Section 1, the focus of our work is enriching the semantics of the numerical representation of images. Therefore, the purpose of the experiments presented in this section is to compare the semantically-enriched representations created by our proposals to a standard baseline representation, created as described in Section 1.1. Whatsoever, directly comparing the discriminative power of two representations is not possible, unless in the context of an image-related machine learning task in this case a content-based image classification. In a nutshell, starting from a collection of images, we construct multiple numerical representations (corresponding to the techniques to be compared) and we train identical classifiers based on each of these representations. In the end, we attribute the differences of classifier performance as a direct consequence of the representation construction technique. More precisely, given the fact that we perform the semantic injection at the level of the visual vocabulary construction, the experimental protocol streamlined in Figure 4 and further detailed in Section 4.1, is designed to quantify the differences of performance due only to the visual vocabulary construction. The evaluation is a five phase process, out of which four phases (1, 2, 3 and 5) are identical for all techniques. Phases 1 to 4 correspond to the BoF representation construction (see Figure 1), while the last phase corresponds to the learning algorithm.

We summarize here after each of these phases, which are further detailed in the next sections:

• phase 1: image sampling, identical for all compared approaches;

• phase 2: feature numerical description of patches, identical for all compared approaches;

• phase 3: visual vocabulary construction, using the baseline approaches and our semantically-enriching approaches;

• phase 4: feature assignment to visual words, identical for all compared approaches;

• phase 5: learning algorithm, each resulted representation is used with two classifiers (a clustering-based and an SVM), identical for all compared approaches.

Experimental protocol

Starting from a given image dataset, we construct, for each image, four BoF representations corresponding to the four evaluated visual vocabulary construction techniques (in phase 3). The image sampling (phase 1), the feature description (phase 2) and the image description (phase 4) are performed each time using the same algorithms and with the same parameters. In the end, the performances of each obtained BoF representation are measured and compared in the context of a content-based image classification task (detailed in Section 4.2). The visual vocabulary construction phase is the only phase to vary between the different constructed representations. Therefore, we consider the classifier performance differences a direct consequence of the vocabulary construction.

The invariant phases 1, 2 and 4 In phase 1, images are sampled using a Hessian-Affine region detector and patches are described, in phase 2, using the SIFT descriptor [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF]. We use the default parameters for these algorithms and we keep them unchanged during the experiments. The visual vocabulary is constructed in phase 3 using the construction technique to be evaluated. In phase 4, the final numerical representation is created, for each image, by associating features to visual words, using the tf term weighting scheme. To reduce the hazard component that appears in all the considered techniques, each construction is repeated 3 times and average results are presented.

Compared vocabulary construction techniques (phase 3) Four visual vocabulary construction techniques are evaluated: two classical techniques random, random+km and our proposals model and filt+model. random constructs a random vocabulary (features are randomly chosen to serve as visual words). For random+km, we take the random features selected previously and we ameliorate them by using the ameliorate using K-Means function presented in Section 3.1. random+km is the baseline construction technique presented in Section 1.1. model is our proposal for dedicated vocabulary construction presented in Algorithm 1. In filt+model we applied the filtering technique presented in Algorithm 2 as a pre-processing phase before the dedicated vocabulary construction.

The learning task: content-based image classification

Each of the image representations obtained as shown in the previous sections, are used in a content-based image classification task. Two classifiers, an SVM and a clusteringbased classifier, are trained and evaluated on each representation, as described in the following paragraphs. The SVM classifier is chosen since it is the most widely used classifier in the object-based image classification literature, showing some of the best results and being a de facto standard. The K-Means-based classifier is used in order to asses our proposals with a second, weaker classifier. The choice of these classifiers is not unique and any other classifier can be used. In the following sections, we evaluate our proposals by measuring the performance gain for any given classifier.

The SVM classifier [START_REF] Cortes | Support-vector networks[END_REF] The SVM classifier evaluation respects the experimental setup recommended by the authors of the Caltech1017 dataset. We used the SVM implementation present in the LibSVM [START_REF] Chang | Libsvm: a library for support vector machines[END_REF] library, using a linear kernel and default parameter values. One of the challenges when evaluating in Data Mining is the disequilibrium between the class cardinality (usually it is the minority class that is of interest). This disequilibrium can cause errors in estimating the generalization error of the constructed model. Usually, the disequilibrium is the result of a certain reality in the population from which the sample was extracted (e.g. the population of sick individuals is a minority compared to the healthy population). But in the case of image datasets like Caltech101, the disequilibrium is only the result of the choice of its creator and represents no reality that needs to be taken into account. We choose to equilibrate the classes before training the classifier, by randomly selecting 30 examples for each label to be part of the learning set. 15 images in the learning corpus are randomly selected to be part of the labeled set P 1 . We test on all remaining individuals, which means that the generalization error on majority classes will be better estimated. Evaluation indicators are calculated for each class and we report only the non-weighted averages.

The process is repeated 10 times: we create 10 learning sets and the corresponding 10 testing sets. We report the average performances over the 10 executions. The results are expressed using the True Positive Rate, because this measure is usually used in the literature when reporting results on Caltech101 and RandCaltech101.

A clustering-based classifier The clustering-based evaluation task is inspired from the unsupervised information retrieval field and it is based on clustering. A learning set of the image collection is clustered into a number of clusters and each cluster is assigned a label, using a majority vote. Each image in the test corpus is assigned to its nearest centroid and it is given the predicted label of the cluster. Predicted labels are compared to the real labels and classical information retrieval measures (i.e., precision, recall, F score ) are calculated.

The evaluation of the clustering-based classifier is performed using a stratified holdout strategy. The images are divided into a learning corpus (67% of images in each category) and a test corpus (33% of the images in each category). 50% of images in the learning corpus are randomly selected to be part of the labeled set P 1 . For the rest, the labels are hidden. Images in the learning set are then clustered into nc clusters using K-Means. nc varies between 50 and 1000 (step 50) for Caltech101 and RandCaltech101 and between 3 and 90 (step 3) for Caltech101-3 (Caltech101-3 contains only 3 classes, see Section 4.3). To eliminate the effect of disequilibrium between class sizes, we calculate and report the non-weighted averages over tags of these indicators. To measure the classification accuracy, we use the F score (the harmonic average of precision and recall), a classical Information Retrieval measure. For each combination (vocabulary dimension, nc, vocabulary algorithm), the clustering and prevision phase is repeated 25 times, to eliminate the influence of the random initialization of the K-Means in the clustering-based classifier.

Datasets

Experiments were performed on the Caltech101 [START_REF] Fei-Fei | Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories[END_REF] and RandCaltech101 [START_REF] Kinnunen | Making visual object categorization more challenging: Randomized caltech-101 data set[END_REF] datasets. Caltech101 contains 9144 images, most of them in medium resolution (300×300 pixels). It is a heterogeneous dataset, having 101 object categories and one reserve. Each category class is considered to be a label. Spatial positioning of objects is not used, therefore positioning ourselves in a weakly supervised context. Some authors argue that Caltech101 is not diverse enough and that backgrounds often provide more information than the objects themselves. RandCaltech101 is obtained from Caltech101 by randomly modifying the backgrounds and the posture (position, orientation) of objects. It has been shown [START_REF] Kinnunen | Making visual object categorization more challenging: Randomized caltech-101 data set[END_REF] that classification is more challenging on RandCaltech101 than on Caltech101.

Because Caltech101 is an unbalanced dataset, with category sizes ranging from 31 to 800 images, we have taken 3 out of the biggest categories (airplanes, Motorbikes and Faces easy) and created another corpus, denoted Caltech101-3. It contains 2033 images. The advantage of the new corpus is that it provides many examples for each category and it is balanced category-wise. This allows us to study how our propositions behave on both balanced and unbalanced datasets. In a classification tasks, some classes are naturally easier to recognize than others. This happens when the numerical description is better adapted to translate them into a separable numerical space. On Caltech101, the best classification scores are almost invariably obtained by the same categories, independent of the choice of visual construction algorithms or parameters.

Qualitative evaluation

Figure 5 shows some examples of images belonging to "easy classes", categories that obtain good classification scores (on the upper row), and examples of "difficult classes", categories that obtain low scores (on the bottom row). The objects belonging to the "easy classes" either appear in the same posture in all examples or they have a specific color pattern that makes them easily recognisable. Most of the examples of airplanes and garfield appear with the same shape, size and orientation. Other categories like yin yang, soccer ball or dalmatian have a specific white-black alternation pattern, which makes them easily recognizable even in the real world. By contrast, the objects depicted in picture of "difficult classes", like seahorse or butterfly appear in different colors, multiple postures and sometimes hidden in the background.

We perform the same analysis on RandCaltech101. Table 1 presents a comparative view of "easy classes" and "difficult classes" constructed for Caltech101 and RandCaltech101, with the non-identical categories (between the two datasets) printed in boldface. We observe the high degree of overlapping of the constructed wrench wrench sets: most of the "easy classes" in Caltech101 also appear as "easily" recognizable for RandCaltech101. Similarly, difficult classes on Caltech101 remain difficult on RandCaltech101. In Table 1, the only category that changes difficulty is metronome, which is an "easy class" in Caltech101 and a "difficult class" in RandCaltech101. This proves that the background randomization performed in order to create RandCaltech101, while it makes the dataset more challenging to classify as a whole, does not change the relative difficulty between categories. Categories that obtain good classification scores for Caltech101 also obtain good scores for RandCaltech101.

Quantitative evaluation

In this section, we show how the performances of the two classifiers vary, depending on the visual vocabulary construction technique and the size of the visual vocabulary.

We show that the semantically-enriched representation clearly outperform the baseline approach, mostly by increasing the score of "difficult" categories, and we discuss the overfitting. For all the experiments presented in this subsection, the parameter α (introduced in Equation 1) of the filtering heuristic filt+model is set at one (α = 1) and its influence is studied later, in Section 4.7.

Aggregating the number of clusters in the clustering-based classifier When using the clustering-based classification algorithm, for a fixed visual vocabulary size, varying the number of clusters nc leads to an F score variation as shown in Figure 6. For all visual vocabulary techniques, the F score has a steep amelioration for lower values of nc and stabilizes once nc reaches a value which is approximately two-three times bigger than the number of categories. Starting from this point F score augments slowly and reaches its theoretical maximum when nc equals the number of individuals in the testing set. Due to the fact that once stabilized, the score can be considered relatively constant, we compute the mean F score over all the values for nc. We obtain, for each visual vocabulary dimension, an aggregated F score . show the variation of the T rueP ositiveRate, using the SVM classifier. We make vary the vocabulary dimension between 100 and 5300 for Caltech101 and RandCaltech101 and between 10 and 5500 for the Caltech101-3, using a variable step. For the three datasets, the horizontal axis is logarithmic. When observing the graphics for every tuple (dataset, classifier, vocabulary construction technique), we observe the pattern of a dome-like shape, corresponding to the three phases: underfitting, maximum performance and overfitting. We analyze more in detail the overfit- Table 2 gives the average gain of performance relative to random for the generic random+km and our semantic-aware proposals model and filt+model. For the clusteringbased classifier, we show the average relative F score gain, while for the SVM we show the average relative T rueP ositiveRate gain. The best scores for each dataset are shown in bold. In five out of six cases, the best scores are obtained by filt+model. model also performs better than the generic random+km in four out of the six cases. This shows that a semantically-enriched representation outperforms the generic method random+km in a classification task. The maximum gain of performance is achieved on RandCaltech101, where, by eliminating the background noise, our filtering algorithm considerably improves the classification performances. When used with the SVM classifier on Caltech101 and RandCaltech101, the model technique obtains average scores lower than random+km. This is because model exits the underfitting later than the other techniques, thus lowering its average score (as shown in Figures 7b and9b).

Obtained graphics

The ROC curves Similar conclusions regarding the overfitting and the relative performances of the different visual vocabulary construction techniques can be drawn by plotting the evolution using ROC [10] curves. Figure 10 shows the ROC curves obtained using the clustering-based classifier on Caltech101(Figure 10a) and on RandCaltech101 (Figure 10b). The visual vocabulary size varied between 100 and 5300. The sign * on the graphic indicates the smallest size. The plots are zoomed to the relevant part. Overfitting is clearly visible on the ROC curves. All the curves start by climbing towards the ideal point (0, 1) (first and second region on the graphics in Figures 7a and9a). After reaching a maximum, the ROC curves start descending towards the "worst" point (1, 0), showing the overfitting region. The curve corresponding to filt+model clearly dominates all the other, confirming the conclusions drawn from studying Table 2: the proposed approaches and especially their combination in filt+model, achieve higher classification results.

Scores for "easy" and "difficult" categories In Section 4.4, we have shown that in both Caltech101 and RandCaltech101 some classes are easier to learn than others. Regardless of the visual vocabulary construction technique, "easy classes" obtain higher classification scores. Nonetheless, the construction particularities of each technique influence the accuracy for difficult categories. In random, features are randomly picked to serve as visual words. Score differences between easy and difficult categories are pronounced and the overall accuracy is low. The K-Means iterations in random+km fit the visual vocabulary to "easy" classes. Few categories achieve good scores, accentuating the gap between easy and difficult categories. model and filt+model techniques achieve for "difficult" categories, better scores than random and random+km. The visual vocabulary is representative for all categories and difficult categories like pyramid, minaret or stapler obtain higher scores than those obtained with a baseline representation.

Overfitting

Evaluating using the clustering-based classifier In the clustering-based classifier, for each pair (dataset, vocabulary construction technique), the F score graphic shows a dome-like shape with three regions. In the first one, corresponding to low vocabulary dimensions, the visual vocabulary is under-fitted, there are not enough visual words to describe the objects [START_REF] Jiang | Towards optimal bag-of-features for object categorization and semantic video retrieval[END_REF]. Consequently, in the assign phase (phase 4 in "bag-offeatures" construction schema in Figure 1), features are assigned to the same visual word even if they are not similar to each other. The second region represents the interval in which the vocabulary obtains the best results. In the third region (corresponding to large sizes of the visual vocabulary), performance degrades gradually. This is due to the fact that, in the assign phase, relevant features are grouped densely, while noise is evenly distributed. Some of the visual words regroup relevant features, while other regroup only the noise. As the visual vocabulary dimension augments, more and more visual words will regroup only noise. This generates a numerical space of high dimensionality, which is separable only on a few dimension. This leads to degrading the overall separability of the numerical space and the classification performances.

Evaluating using the SVM classifier The same conclusions apply for the SVM classifier. Being a strong classifier, in Figures 7b (Caltech101) and 9b (RandCaltech101) the dome-shape is less visible for the SVM. The overfitting appears for higher visual vocabulary sizes than in the clustering-based classifier. For example, in Figure 9a, for random+km, clustering-based classifier starts to overfit at a vocabulary size of 300. When using the SVM, in Figure 9b, overfitting starts only at 1300. The model technique does not appear to enter overfitting in Figure 9b. But this is likely to happen for dimensions higher than 5300 (the maximum considered), because model is the last technique to enter overfitting for the clustering-based classifier (as shown in Figure 9a). The overfitting region is even more visible for Caltech101-3 (Figure 8). The visual vocabulary sizes are considerably higher than for the other datasets, relative to the number of classes. In Figure 8a performances of all visual vocabulary techniques descend sharply for higher values of vocabulary size. The evaluation using the SVM classifier, in Figure 8b, also clearly shows the dome-like shape. In Equation 1, we have defined δ, the filtering threshold, which is used to decide if a feature has any similar features in the known positive set. The parameter α is used to fine-tune this threshold. If α is set too low, only the features that are very close (in terms of Euclidean distance) are considered to be similar. Consequently, the filtering is very strict, lowering the number of false positives, with the risk of an inflation of false negatives. On the other hand, setting α too high allows distant features to be considered as similar, causing a high number of false positives. In the previous experiments, we have set the parameter α = 1. In this section, we study the influence of this parameter on the performances obtained by the filt+model construction technique.

Figure 11 shows the evolution of the filt+model visual vocabulary construction technique as a function of the vocabulary size, when using α ∈ {0.8, 1, 1.25, 1.5}. The horizontal axis is logarithmic. A value for α = 0.8 is too strict and the high number of false negatives decreases the classification performances. Augmenting α = 1 improves performances, both when using the clustering-based classifier (Figure 11a) and when using the SVM classifier (Figure 11b).

If α is set too high, performances decrease again. Too many features are considered similar and less features get filtered. Performances approach those obtained when no filtering is applied. α = 1.25 and α = 1.5 show similar performances, since both levels are already too high for filtering to be effective. For α ≥ 1.25, filt+model is equivalent to the model visual vocabulary construction technique. In Figure 11a, filt+model with α ∈ {1.25, 1.5} obtains, for high visual vocabulary sizes (m > 2000), better results than filt+model with α ∈ {0.8, 1}. This behavior is similar with that already seen in Figure 9a, when model enters overfitting later than filt+model, and obtains better results for high vocabulary sizes.

These initial experiments make us believe that α is dataset independent (a value of 1 provided best results on all three datasets), but further experiments on other datasets are required for a firm conclusion. Furthermore, a heuristic for automatically determining its value is part of our future plans.

Conclusion and future work Conclusion

In the work presented in this article, we have focused on constructing a semantically-enriched representation for images, by leveraging additional information under the form of non-positional labels. We argue that enriching the semantics of the image representation boosts the performances of learning algorithms and we apply our proposed method to the learning task of content-based image classification.

We use the additional information in the phase of visual vocabulary construction, when building a "bag-of-features" image representation. We have proposed two novel approaches for incorporating this semantic knowledge into the visual vocabulary creation. The first approach creates dedicated vocabularies for each label, while the second uses a pre-processing phase for filtering visual features unlikely to be associated with a given object. We have shown that the semantically-enriched image representations built using our proposals obtain higher scores than a baseline BoF representation, in the context of a task of content-based image classification. This shows that incorporating semantic knowledge in the vocabulary construction results in more descriptive visual words, especially on datasets where the background noise is significant. Even when overfitted, our proposals continue to outperform the generic approach.

Future work Our visual vocabulary construction techniques, proposed in Section 3, are not limited to the task of object-based image classification. They can be used with any image-related machine learning task that involves constructing a BoF representation for images. Whatsoever, scaling our approaches to other applications involves relaxing some of the assumptions. For example, we assumed that labels which denote objects appear independently in the image collection. We are working on relaxing this strict condition and on passing from a learning task of object categorization to one of scene classification. This raises the difficulty of object co-occurrence. For example, a picnic scene is defined by the simultaneous presence of "people", "trees", "grass" and "food". In terms of labels, this translates into label co-occurrence. Our approaches can be scaled to image classification by addressing the label co-occurrence issue. We are currently working on using the unsupervised feature8 construction algorithm proposed in [START_REF] Rizoiu | Unsupervised feature construction for improving data representation and semantics[END_REF] to reconstruct the image labels and to reduce, even eliminate, their co-occurrence. The new labels are constructed as conjunctions of existing labels and their negations, and would actually no longer be used to label objects, but scenes. For example, if the labels "motorcycle" and "rider" appear often together, a new label "motorcycle ∧ rider" will be created to mark the scene identified by the presence of the two objects.

Figure 1 :

 1 Figure 1: Construction flow of a "bag-of-features" numerical representation for images

Figure 2 :

 2 Figure 2: Example of feature corresponding to the visual words associated with "wheal" (in red) and "exhaust pipe" (in green)

  features sampled from labeled images Input: Y ∈ {0, 1} n1×k -image/label association matrix Input: m -the dimension of the visual vocabulary M Output: the visual vocabulary M having m visual words // for each label for i = 1 to k do m i ← m/k // size of the dedicated vocabulary

Figure 3 :

 3 Figure 3: (a) An image labeled "motorbike", (b) an image from the known positive set and (c) an image from the known negative set

5 Figure 4 :

 54 Figure 4: Schema for evaluating multiple visual vocabulary construction techniques.

Figure 5 :

 5 Figure 5: Example of images from "easy" classes (top row) and "difficult" classes (bottom row)

Figure 6 :

 6 Figure 6: A typical F score evolution for the clustering-based classifier for m = 1000 on Caltech101 (a) and on RandCaltech101 (b)

Figures 7 ,

 7 8 and 9 present the score evolution as a function of the visual vocabulary size on, respectively, the datasets Caltech101, Caltech101-3 and RandCaltech101. More precisely, Figures7a, 8a and 9ashow the evolution of the aggregated F score , for the clustering-based classifier, and Figures7b, 8b and 9b

Figure 10 :

 10 Figure 10: ROC curves: clustering-based classifier on Caltech101 (a) and RandCaltech101 (b)

4. 7 Figure 11 :

 711 Figure 11: RandCaltech101: influence of parameter α on filt+model construction technique in the clustering-based classifier (a) and the SVM classifier (b)

Table 1 :

 1 "Easy" classes and "difficult" classes in Caltech101 and RandCaltech101

	"Easy" classes	"Difficult" classes
	Caltech101	RandCaltech101	Caltech101	RandCaltech101
	airplanes	accordion	beaver	bass
	car side	airplanes	buddha	binocular
	dalmatian	car side	butterfly	brontosaurus
	dollar bill	dalmatian	ceiling fan	buddha
	Faces easy	dollar bill	cougar body	butterfly
	garfield	Faces easy	crab	crab
	grand piano	garfield	crayfish	crayfish
	Leopards	laptop	cup	crocodile
	metronome	Motorbikes	dragonfly	cup
	Motorbikes	panda	ewer	dragonfly
	panda	snoopy	ferry	ewer
	scissors	soccer ball	flamingo	flamingo
	snoopy	stop sign	flamingo head	flamingo head
	soccer ball	watch	ibis	gerenuk
	stop sign	windsor chair	kangaroo	helicopter
	tick	yin yang	lamp	ibis
	watch		lobster	kangaroo
	windsor chair		mandolin	lamp
	yin yang		mayfly	lobster
			minaret	mandolin
			pigeon	mayfly
			platypus	metronome
			pyramid	minaret
			rhino	okapi
			saxophone	pigeon
			schooner	platypus
			sea horse	saxophone
			stapler	sea horse
			strawberry	stapler
			wild cat

Table 2 :

 2 Results interpretation When comparing the relative performances of the different techniques presented in Figures7, 8 and 9, we observe that our semantic-aware proposals (i.e., model and filt+model) generally obtain better results than the generic (random+km) and random ones. The three regions of evolution are wider (they enter overfitting later) for model and filt+model than for random and random+km. On the other hand, they also exit the under-fitting later. The generic random+km obtains better results than model and filt+model, for lower dimensions of visual vocabulary, on Caltech101 and RandCaltech101. After exiting the under-fitting region, model and filt+model constantly obtain better scores than random+km, even when Average gain of performance relative to random.

		FScore for different vocabulary creation techniques		True Positive Rate for different vocabularies
		0.17
		0.16
		0.15				0.37
						0.36
	FScore value	0.11 0.12 0.13 0.14			True Positive Rate	0.3 0.31 0.32 0.33 0.34 0.35
						0.29
		0.1				0.28
		0.08 0.09		model filt+model random+km random		0.27 0.25 0.26		model filt+model random+km random
		100	200 300	500 700 1000	2300 3300 5300	100	200 300	500 700 1000	2300 3300 5300
			Visual vocabulary size (log)			Visual vocabulary size (log)
				(a)				(b)
	Figure 7: Caltech101: Aggregated F score with clustering-based classifier (a) and
	T rueP osiviteRate for SVM (b) as functions of the vocabulary size
		FScore for different vocabulary creation techniques		True Positive Rate for different vocabularies
		0.95				0.98
		0.93				0.96
		0.91				0.94
		0.89
	FScore value	0.75 0.77 0.79 0.81 0.83 0.85 0.87			True Positive Rate	0.84 0.86 0.88 0.9 0.92
		0.73				0.82
		0.67 0.71 0.69		model filt+model random+km random		0.8 0.78		model filt+model random+km random
		10	30 50 70 110 160 260 500 1000 2000 35005500	10	30 50 70 110 160 260 500 1000 2000 35005500
			Visual vocabulary size (log)			Visual vocabulary size (log)
				(a)				(b)
	Figure 8: Caltech101-3: Aggregated F score with clustering-based classifier (a)
	and T rueP osiviteRate for SVM (b) as functions of the vocabulary size
	ting behavior for each vocabulary construction technique in Section 4.6. Furthermore,
	the somehow low results obtained by the clustering-based classifier can be explained by
	the fact that the clustering-based classifier is a weak classifier (i.e., a classifier which
	perform only slightly better than a random classifier), whereas the SVM is a strong
	classifier.

http://instagram.com/

http://www.flickr.com/

l i is dependent on the content on the image (number of objects, shape etc.) and the extraction algorithm used. It can vary from a couple of hundreds of features up to several tens of thousands.

e.g. for the SIFT descriptor h = 128.

The number of extracted features can greatly vary depending on the image and the method used for sampling.

Improving the BoF representation using semantic knowledgeIn this section, we present our two novel methods that leverage external semantic information, under the form of non-positional object labels, into the visual vocabulary construction. This kind of information is often freely available on the Web, being constantly produced by anonymous users. There are some distinctive differences between using public domain knowledge and using domain experts: (i) the public domain knowledge is not created especially for our application, but rather we make use it to improve the quality of the constructed image representations and (ii) unlike using domain experts, no evaluation feedback link can be made back to the creators of additional knowledge, as they are often anonymous. For these reasons, we privilege a semi-supervised approach and we use the additional information to guide the algorithm in the solutions space.Our work is positioned in a weakly supervised context, similar to the one defined in[START_REF] Zhang | Local features and kernels for classification of texture and object categories: A comprehensive study[END_REF]. Each label signals the presence of a given object in an image, but not its position or boundaries. Our approaches use the semantic information to increase the relevancy of the visual vocabulary. In our first approach, for each label, we construct a dedicated visual vocabulary, based only on the images with a certain label. Such approaches have been

shown[START_REF] Jianjia | Combined category visual vocabulary: A new approach to visual vocabulary construction[END_REF][START_REF] Perronnin | Adapted vocabularies for generic visual categorization[END_REF] to improve accuracy over a general purpose vocabulary, since specialized vocabularies contain visual words that more appropriately describe the objects appearing in the image collection. In our second approach, we further improve accuracy by proposing a novel pre-processing phase, which filters out features that are unlikely to belong to the respective object. Our filtering proposal follows the framework of the object recognition algorithm proposed in[START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] and uses a positive and a negative example set, constructed based on the labels. The filtering pre-processing is combined with the dedicated visual vocabulary construction, and we show in Section 4 that this approach consistently achieves higher accuracy then both a dedicated vocabulary (with no filtering) and a general purpose vocabulary.

http://www.vision.caltech.edu/Image_Datasets/Caltech101/

Note that, in this context, the word feature is used in the sense of attribute. It should not be confused with the definition of feature in image processing literature, where it has the sense of visual feature.