Paula Louro

Vitor Silva

Manuela A Vieira

A Karmali

Use of a-SiC:H Semiconductor-Based Transducer for Glucose Sensing through FRET Analysis

Keywords: Glucose sensing, FRET, nanosensors, fluorescence proteins, a-SiC:H photodiodes, multilayered heterostructures, optical sensors

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

There is a great scientific and medical interest to develop suitable technology for continuous in vivo glucose monitoring in patients with diabetes mellitus. There are several problems with existing devices based on electrochemistry which have encouraged alternative approaches to glucose sensing in recent years. An approach based on fluorescence intensity and lifetime, has special advantages, including sensitivity and the potential for non-invasive measurement when near infrared light is used. Such techniques include measuring changes in fluorescence resonance energy transfer (FRET) between a fluorescent donor and an acceptor within a protein which undergoes glucose-induced changes in conformation [1]. First fluorescence excitation is performed at the donor wavelength. As the protein changes its conformation the donor and acceptor fluorophores, attached to the protein, change their distance. When the distance between fluorophores is reduced the donor transfers energy to the acceptor through nonradiative dipole-dipole coupling. The energy transfer causes on one hand a decrease in intensity at the donor emission wavelength and on the other an increase in intensity at the acceptor emission wavelength. Such variation in the emission spectrum can be used to detect and measure the concentration of glucose in the sample.

In this paper we use a-SiC:H optical devices to detect the cyan and yellow emission signals of fluorescent proteins. The transducer consists of two heterostructures based on a-SiC:H/a-Si:H optimized for the detection of the fluorescence emissions at wavelengths 470 nm (cyan) and 588 nm (yellow) [2] 2 Contribution to Internet of Things Internet of Things refers to the concept of expanding the communication among things. This will bring the ability of objects behaviour to change due to what they access through the Internet. In a FRET based sensor with a wireless system, a warning or emergency message could be generated and sent in a multicast network. This same message would be received by the nearest hospital and by a patient transportation system and also by the patient's doctor cell phone. Another scenario would be a feedback response based on the transmitted values which would tell the patient what to do.

Devices based on new materials for glucose sensing are an important contribution to the future development of the Internet of Things.

Materials and Methods

The following recombinant plasmids were purchased from Addgene (USA): 13561 pRSET FLIPglu-2uDelta13 and 13660 pRSET FLIPglu-3.2mDelta13. Luria Bertani culture medium, iminodiacetic acid, epichlrohydrin, butanodioldiglycidyl ether and ampicillin were obtained from Sigma Chemical Company (USA) and Sepharose 6B, Sepharose CL 4B, and Sepharose CL 6B were purchased from GE HealthCare (Sweden).

All cell-free extracts of recombinant E. coli strains were assayed for protein by using the Coomassie blue dye binding method.

The recombinant plasmids were used to transform E. coli DH5 strain which was grown overnight in Luria Bertani (LB) culture medium containing 100 g/ ml of ampicillin at 37ºC and 180 rpm as described previously [1]. The cultures were harvested by centrifugation at 10.000 rpm for 1 min. at 4ºC and the cells were ressuspended in 2 volumes of 20 mM phosphate buffer pH 7.0 containing 1 mM benzamidine and sonicated five times for 30 s at 160 W with an interval of 5 min. The cell-suspension was centrifuged at 20.000 rpm for 1 h to remove cell debris and the supernatant was frozen at -20ºC for further studies. The cell-free extract was used as the source of glucose binding protein which was purified by immobilized metal affinity chromatography (IMAC) on immobilized copper chelates as described previously [3].

Fluorescence measurements were carried out in a Jasco FP-8300 spectrofluorometer equipped with the standard 10 mm cuvette holder.

Wavelength scans were performed on all glucose binding protein variants to determine excitation and emission peaks. Spectra were collected from 450-700 nm using the following configuration: bandwidth (excitation/emission) -5 nm; excitation wavelength -433 nm, data pitch -0.2 nm, scanning speed -125 nm/min. Unless otherwise stated, the fluorescent intensity per unit protein was determined with the samples diluted in 20 mM sodium phosphate buffer pH. 7.0 with excitation and emission set at 433 nm and 485 and 528 nm, respectively. Glucose solutions were prepared in 20 mM phosphate buffer pH 7.0 containing 1 mM benzamidine which were added to the cell-free extract containing the glucose binding protein variants to a final concentration of 100 mM as follows: pRSET FLIPglu-2uDelta13 and pRSET FLIPglu-3.2mDelta13. Representative scans of the prepared glucose nanosensors are shown in Figure 1. In both spectra it is clear the presence of fluorescent emissions with peaks at the cyan and yellow regions. The observed spectrum changes along the reaction time are due to the induced conformation changes of the fluorescent proteins that stabilize after a few minutes. The correlation of these changes with time is probably related to the use of cellular extracts in our samples. The saturation curves are related to the glucose sensor concentration of the sample. Results show also that when the reaction is complete and saturation occurs the emission spectrum of both sensors exhibit similar trends. However, the cyan peak is more evident in glucose nanosensor pRSET FLIPglu-3.2mDelta13.

Device Configuration

The device described herein works from 400 to 700 nm which makes it suitable for FRET applications using fluorophore operating in the visible spectrum. The device is a multilayered heterostructure based on a-Si:H and a-SiC:H. The configuration of the device includes two stacked p-i-n structures between two electrical and transparent contacts (Fig. 2). Both front (pin1) and back (pin2) structures act as optical filters confining, respectively, the short and the long optical carriers, while the intermediate wavelengths are absorbed across both [4,5]. The device was operated within the visible range using as optical signals to simulate the emitted fluorescent signals of the recombinant plasmids from E. coli strains (yellow and cyan) the modulated supplied by a cyan and a yellow LED with wavelengths of 470 nm and 588 nm, respectively. Steady state optical bias of different wavelength from the front and back sides was also used for light soaking the device.

Spectral Sensitivity

Figure 3 displays the spectral photocurrent, measured along the visible spectrum, under reverse bias without and with optical light bias focusing the device from back and front sides. Results of Fig. 3a) show that the use of steady state light bias from the front side, i.e., from the a-SiC:H structure, enhances the photocurrent in the range of wavelengths longer than 500 nm when using background light of short wavelength (400 nm, 470 nm and 524 nm). For optical bias of longer wavelengths (588 nm and 626 nm) the photocurrent signal is decreased. On the remaining part of the spectrum an opposite behavior is observed as the amplification effect occurs for longer light bias wavelengths and the reduction of the signal for shorter wavelengths. Thus, the use of a 400 nm light source as optical bias amplifies both cyan and yellow fluorescent. On the other hand, the optical biasing from the back side (a-Si:H) is not so much dependent on the background light wavelength (Fig. 3b). Generally, in the range of wavelengths shorter than 480 nm it is observed an increase of the signal for every background, and a decrease in the complementary range. The enhancement of the photocurrent signal using different optical bias for the background light result is an effective method for tuning the device sensitivity and detecting the emission fluorescent wavelengths.

Transfer Function of Each Fluorescent Optical Signal

The photocurrent signal obtained at reverse electric bias (-8 V) with (dark lines) and without background light (violet at 400 nm from the front side and cyan at 470 nm from the back side) is displayed in Fig. 4 for simulated fluorescent optical signals (cyan and yellow emissions, respectively, at 470 nm and 588 nm). As already indicated by the spectral photocurrent measurements of Fig. 4 the same wavelength is focusing on the device back side. The use of violet light from the front side has a negligible effect on this signal. For the yellow emission at 588 nm the effect of the background light is opposite. The violet steady state light from the front side causes a large amplification of the photocurrent (amplification factor about 4 times) and at the same time the cyan light from the back side reduces the signal by half. This amplification mechanism is useful for the determination of each fluorescent signal contribution to the whole output signal. With cyan background light from the back side the cyan fluorescent signal is enhanced and the yellow one reduced. On the other hand violet background from the front side will increase the photocurrent component due to the yellow signal.

Transfer Function of a FRET Optical Signal

Several time dependent wavelength combinations of both fluorescent pulsed input optical signals (at 470 nm and 588 nm) were used to generate a multiplexed signal in the device. The output photocurrents, with and without optical background light are displayed in Figure 5. The reference level was assumed to be the signal when all the input optical signal channels are OFF. At the top of the figure, the individual optical signals are displayed to guide the eyes in relation to the different ON-OFF states. In both chromatic sequences it is clearly observed that the shape of the photocurrent signal measured under dark (black line) exhibits four threshold levels, each assigned to the correspondent optical conditions of the input signals. The ON-ON state corresponds to the maximum intensity of light bias, while the ON-OFF and OFF-ON to a lower intensity and the OFF-OFF to the dark conditions. 0,0 However, when the device is optically biased the output photocurrent changes enhancing the presence of each input fluorescent optical signal. In both sequences of Fig. 5 the shape of the photocurrent, measured by soaking the device with violet light by the front side, follows the shape of the yellow fluorescent signal). Whenever the yellow optical signal is ON, the measured photocurrent is higher than its correspondent without optical bias. On the other hand, the use of cyan optical bias by the device back side results in an output signal that follows the cyan fluorescent emission, which allows the recognition of the presence of this fluorescent signal. Thus, the proper optical bias of the device through the choice of light wavelength and the device soaking light side results in the correct tuning of each fluorescence signal. These features allow immediate decoding of both yellow and cyan emitted signals.

Electrical Model

Based on the experimental results and device configuration an optoelectronic model was developed [6]. The device was modeled by a two single-tuned stages circuit with two variable capacitors and interconnected phototransistors through a resistor (Fig. 6a). Two optical gate connections ascribed to the different light penetration depths across the front and back phototransistors were considered to allow independent yellow and cyan optical signals transmission. a) 0,0 0,1 0,2 0,3 0,4 0,5 0,6 0,0 The operation is based upon the following principle: the flow of current through the resistor connecting the two front and back transistor bases is proportional to the difference in the voltages across both capacitors (charge storage buckets). So, it uses a changing capacitance to control the power delivered to the load acting as a state variable filter circuit. In Figure 6b the simulated current without and under red backgrounds is displayed (symbols) using the same test signal of Fig. 5a. The input channels are also displayed (lines). To simulate the red background, current sources intensities (input channels) were multiplied by the on/off ratio between the input channels with and without red optical bias. Good agreement between experimental and simulated data was observed. The four expected levels, under reversed bias, and their reduction under red irradiation, are clearly seen. Under red background the expected optical amplification of the cyan channel and the quenching of the yellow one were observed due to the effect of the active multiple-feedback filter when the back diode is light triggered.

2,0 i(OFF) R 1 =1KΩ R 2 =5kΩ i(ON) α R =0.4 α B =1.75

Conclusions

We present a novel device semiconductor based on an a-SiC:H p-i-n/p-i-n heterostructures for the detection of optical signals near the cyan and yellow regions of the visible spectrum, which can be used for the detection of the emission signals used in the FRET technique. We used a commercial spectrofluorometer to measure the fluorescent signal emitted by recombinant plasmids from E. coli strain, which showed emission peaks at the cyan and yellow bands of the visible spectrum accordingly to the conformation changes suffered by the fluorescent proteins. Simultaneously different modulated optical signals were used to simulate the emission signals of the glucose nanosensors. The output emission spectrum was analyzed by reading out the photocurrent generated by the device under reverse bias and using different wavelengths of background light to soak the device from the front and back side. The influence of the fluorescent emission optical signals on the output photocurrent was also analyzed. Future work comprises the use of lower power intensities for the simulated emission signals in order to reach the same range of the ones produced during the FRET phenomenon. In a further stage tests with the

I4 Q2 I1 1 I2 Q1 C2 R2 R1 (V) C1 I3
emission fluorescence signals of the prepared recombinant plasmids from E. coli strain with the semiconductor device and the quantification of the glucose concentration from the changes in the measured spectra.

Fig. 1 .

 1 Fig. 1. Emission spectra for glucose nanosensors scanned during 12 minutes for: a) 13660 pRSET FLIPglu-3.2mDelta13 and b) 13561 pRSET FLIPglu-2uDelta13.

Fig. 2 .

 2 Fig. 2. Simplified schematic diagram of the structure of the device.

Fig. 3 .

 3 Photocurrent (nA)

Fig. 4 .

 4 Photocurrent (µA)

Fig. 5 .

 5 Output photocurrent signals without (Ф L =0) and with optical bias (violet light at 400 nm from the front side, and cyan light at 400 nm from the back side) of two different optical signals waveforms (shown at the top of the figure).

Fig. 6 .

 6 Photocurrent (µA)

Acknowledgements. This work was supported by PTDC/EEA-ELC/111854/2009, PTDC/EEA-ELC/115577/2009 and PTDC/EEA-ELC/120539/2010.


Maxime Bucher

Stéphane Herbin

Frédéric Jurie

Improving Semantic Embedding Consistency by Metric Learning for Zero-Shot Classification

Keywords: zero-shot learning, attributes, semantic embedding

This paper addresses the task of zero-shot image classification. The key contribution of the proposed approach is to control the semantic embedding of images -one of the main ingredients of zero-shot learning -by formulating it as a metric learning problem. The optimized empirical criterion associates two types of sub-task constraints: metric discriminating capacity and accurate attribute prediction. This results in a novel expression of zero-shot learning not requiring the notion of class in the training phase: only pairs of image/attributes, augmented with a consistency indicator, are given as ground truth. At test time, the learned model can predict the consistency of a test image with a given set of attributes, allowing flexible ways to produce recognition inferences. Despite its simplicity, the proposed approach gives state-of-the-art results on four challenging datasets used for zero-shot recognition evaluation.

Introduction

This paper addresses the question of zero-shot learning (ZSL) image classification, i.e., the classification of images belonging to classes not represented by the training examples. This problem has attracted much interest in the last decade because of its clear practical impact: in many applications, having access to annotated data for the categories considered is often difficult, and requires new ways to increase the interpretation capacity of automated recognition systems.The efficiency of ZSL relies on the existence of an intermediate representation level, effortlessly understandable by human designers and sufficiently formal to be the support of algorithmic inferences. Most of the studies have so far considered this representation in the form of semantic attributes mainly because it provides an easy way to describe compact yet discriminative descriptions of new classes.

It has also been observed [START_REF] Mahajan | A joint learning framework for attribute models and object descriptions[END_REF][START_REF] Romera-Paredes | An embarrassingly simple approach to zero-shot learning[END_REF] that attribute representations as provided by humans may not be the ideal embedding space because it can lack the informational quality necessary to conduct reliable inferences: the structure of the attribute manifold for a given data distribution may be rather complex, redundant, noisy and unevenly organized. Attribute descriptions, although semantically meaningful and useful to introduce a new category, are not necessarily isomorphic to image data or to image processing outputs.

To compensate for the shortcomings induced by attribute representations, the recent trends in ZSL studies are aiming at better controlling the classification inference as well as the attribute prediction in the learning criteria. Indeed, if attribute classifiers are learned independently of the final classification task, as in the Direct Attribute Prediction model [START_REF] Lampert | Learning to detect unseen object classes by between-class attribute transfer[END_REF], they might be optimal at predicting attributes but not necessarily at predicting novel classes.

In the work proposed in this paper, we instead suggest that better controlling the structure of the embedding attribute space is at least as important than constraining the classification inference step. The fundamental idea is to empirically disentangle the attribute distribution by learning a metric able to both select and transform the original data distribution according to informational criteria. This metric is obtained by optimizing an objective function based on pairs of attributes/images without assuming that the training images are assigned to categories; only the semantic annotations are used during training. More specifically, we empirically validated the idea that optimizing jointly the attribute embedding and the classification metric, in a multi-objective framework, is what makes the performance better, even with a simple linear embedding and distance to mean attribute classification.

The approach is experimentally validated on 4 recent datasets for zero-shot recognition, i.e. the 'aPascal&aYahoo', 'Animals with Attributes', 'CUB-200-2011 and 'SUN attribute' datasets for which excellent results are obtained, despite the simplicity of the approach.

The rest of the paper is organized as follows: Section 2 presents the related works, Section 3 describes the proposed approach while the experimental validation is given in Section 4.

Related work

Visual features and semantic attributes

Image representation -i.e. the set of mechanisms allowing to transform image raw pixel intensities into representations suitable for recognition tasks -plays an important role in image classification. State-of-the art image representations were, a couple of years ago, mainly based on the pooling of hard/soft quantized local descriptors (e.g. SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF]) through the bag-of-words [START_REF] Csurka | Visual categorization with bags of keypoints[END_REF] of Fisher vectors [START_REF] Sánchez | Image classification with the Fisher vector: Theory and practice[END_REF] models. However, the work of Krizhevsky et. al. [START_REF] Krizhevsky | ImageNet Classification with Deep Convolutional Neural Networks[END_REF] has opened a new area and most of the state-of-the-art image descriptors nowadays rely on Deep Convolutional Neural Networks (CNN). We follow this trend in our experiments and use the so-called 'VGG-VeryDeep-19' (4096-dim) descriptors of [START_REF] Simonyan | Very Deep Convolutional Networks for Large-Scale Image Recognition[END_REF].

Two recent papers have exhibited existing links between CNN features and semantic attributes. Ozeki et al. [START_REF] Ozeki | Understanding Convolutional Neural Networks in Terms of Category-Level Attributes[END_REF] showed that some CNN units can predict some semantic attributes of the 'Animals with Attributes' dataset fairly accurately. One interesting conclusion of their paper is that the visual semantic attributes can be predicted much more accurately than the non-visual ones by the nodes of the CNN. More recently, [START_REF] Escorcia | On the Relationship between Visual Attributes and Convolutional Networks[END_REF] showed the existence of Attribute Centric Nodes (ACNs) within CNNs trained to recognize objects, collectively encoding information pertinent to visual attributes, unevenly and sparsely distributed across all the layers of the network.

Despite these recent findings could certainly make the performance of our method better, we don't use them in our experiments and stick to the use of standard CNN features, with the intention of making our results directly comparable to the recent zero-shot learning papers (e.g. [START_REF] Zhang | Zero-Shot Learning via Semantic Similarity Embedding[END_REF]).

Describing images by semantic and non-semantic attributes

Zero-shot learning methods rely on the use of intermediate representations, usually given as attributes. This term can, however, encompass different concepts. For Lampert et al. [START_REF] Lampert | Attribute-Based Classification for Zero-Shot Visual Object Categorization[END_REF] it denotes the presence/absence of a given object property, assuming that attributes are nameable properties (color or presence or absence of a certain part, etc.). The advantage of so-defined attributes is that they can be used easily to define new classes expressed by a shared semantic vocabulary.

However, finding a discriminative and meaningful set of attributes can sometimes be difficult. [START_REF] Parikh | Interactively building a discriminative vocabulary of nameable attributes[END_REF][START_REF] Duan | Discovering localized attributes for fine-grained recognition[END_REF] addressed this issue by proposing an interactive approach that discovers local attributes both discriminative and semantically meaningful, employing a recommender system that selects attributes through human interactions. An alternative for identifying attribute vocabulary without human labeling is to mine existing textual description of images sampled from the Internet, such as proposed by [START_REF] Berg | Automatic attribute discovery and characterization from noisy web data[END_REF]. In the same line of thought, [START_REF] Ba | Predicting deep zero-shot convolutional neural networks using textual descriptions[END_REF] presented a model for classifying unseen categories from their (already existing) textual description. [START_REF] Elhoseiny | Write a Classifier: Zero-Shot Learning Using Purely Textual Descriptions[END_REF] proposed an approach for zero-shot learning where the description of unseen categories comes in the form of typical text such as an encyclopedia entries, without the need to explicitly define attributes.

Another drawback of human generated attributes is that they can be redundant or not adapted to image classification. These issues have been addressed by automatically designing discriminative category-level attributes and using them for tasks of cross-category knowledge transfer, such as in the work of Yu et al. [START_REF] Yu | Designing category-level attributes for discriminative visual recognition[END_REF]. Finally, the attributes can also be structured into hierarchies [START_REF] Verma | Learning hierarchical similarity metrics[END_REF][START_REF] Rohrbach | Evaluating knowledge transfer and zero-shot learning in a large-scale setting[END_REF][START_REF] Akata | Evaluation of Output Embeddings for Fine-Grained Image Classification[END_REF] or obtained by text mining or textual descriptions [START_REF] Akata | Evaluation of Output Embeddings for Fine-Grained Image Classification[END_REF][START_REF] Ba | Predicting deep zero-shot convolutional neural networks using textual descriptions[END_REF][START_REF] Elhoseiny | Write a Classifier: Zero-Shot Learning Using Purely Textual Descriptions[END_REF].

Beside these papers which all consider attributes as meaningful for humans, some authors denoted by attributes any latent space providing an intermediate representation between image and hidden descriptions that can be used to transfer information to images from unseen classes. This is typically the case of [START_REF] Wang | A Discriminative Latent Model of Object Classes and Attributes[END_REF] which jointly learn the attribute classifiers and the attribute vectors, with the intention of obtaining a better attribute-level representation, converting undetectable and redundant attributes into discriminative ones while retaining the useful semantic attributes. [START_REF] Farhadi | Describing objects by their attributes[END_REF] has also introduced the concept of discriminative attributes, taking the form of random comparisons. The space of class names can also constitute an interesting embedding, such as in the works of [START_REF] Mensink | COSTA: Co-Occurrence Statistics for Zero-Shot Classification[END_REF][START_REF] Norouzi | Zero-Shot Learning by Convex Combination of Semantic Embeddings[END_REF][START_REF] Zhang | Zero-Shot Learning via Semantic Similarity Embedding[END_REF][START_REF] Fu | Zero-shot object recognition by semantic manifold distance[END_REF] which represent images as mixtures of known classes distributions.

Finally, it worth pointing out that the aforementioned techniques are restricting attributes to categorical labels and dont allow the representation of more general semantic relationships. To counter this limitation, [START_REF] Parikh | Relative attributes[END_REF] proposed to model relative attributes by learning a ranking function.

Zero shot learning from semantic and attribute embedding

As defined by [START_REF] Palatucci | Zero-shot learning with semantic output codes[END_REF], the problem of zero-shot learning can be seen as the problem of learning a classifier f : x → y that can predict novel values of y not available in the training set. Most of existing methods rely on the computation of a similarity or consistency function linking image descriptors and the semantic description of the classes. These links are given by learning two embeddings -the first from the image representation to semantic space and the second from the class space to the semantic space -and defining a way to describe the constraints between the class space and the image space, the two being strongly interdependent.

DAP (Direct Attribute Prediction) and Indirect attribute prediction (IAP), first proposed by [START_REF] Lampert | Learning to detect unseen object classes by between-class attribute transfer[END_REF], use the between layer of attributes as variables decoupling the images from the layer of labels. In DAP, independent attribute predictors are used to build the embedding of the image, the similarity between two semantic representations (one predicted from the image representation, one given by the class) being given as the probability of the class attribute knowing the image. In IAP, attributes form a connecting layer between two layers of labels, one for classes that are known at training time and one for classes that are not known. In this case, attributes are predicted from (known) class predictions. Lampert et al. [START_REF] Lampert | Learning to detect unseen object classes by between-class attribute transfer[END_REF] concluded that DAP gives much better performance than IAP.

However, as mentioned in the introduction, DAP has two problems: first, it does not model any correlation between attributes, each being predicted independently. Second, the mapping between classes and the attribute space does not weight the relative importance of the attributes nor the correlations between them. Inspired by [START_REF] Weston | WSABIE: scaling up to large vocabulary image annotation[END_REF], which learns a linear embedding between image features and annotations, [START_REF] Akata | Label-Embedding for Image Classification[END_REF] and [START_REF] Romera-Paredes | An embarrassingly simple approach to zero-shot learning[END_REF] tried to overcome this limitation. The work of Akata et al. [START_REF] Akata | Label-Embedding for Image Classification[END_REF] introduced a function measuring the consistency between an image and a label embedding, the parameters of this function is learned to ensure that, given an image, the correct classes rank higher than the incorrect ones. This consistency function has the form of a bilinear relation W associating the image embedding θ(x) and the label representation φ(y) as S(x, y; W ) = θ(x) t W φ(y). Romera et al. [START_REF] Romera-Paredes | An embarrassingly simple approach to zero-shot learning[END_REF] proposed a simple closed form solution for W , assuming a specific form of regularization is chosen. In comparison to our work none of the two papers ( [START_REF] Akata | Label-Embedding for Image Classification[END_REF], [START_REF] Romera-Paredes | An embarrassingly simple approach to zero-shot learning[END_REF]) use a metric learning framework to control the statistical structure of the attribute embedding space.

The coefficients of the consistency constraint W can be also predicted from a semantic textual description of the image. As an example, the goal of [START_REF] Elhoseiny | Write a Classifier: Zero-Shot Learning Using Purely Textual Descriptions[END_REF] is to predict a classifier for a new category based only on the learned classes and a textual description of this category. They solve this problem as a regression function, learnt from the textual feature domain to the visual classifier domain. [START_REF] Ba | Predicting deep zero-shot convolutional neural networks using textual descriptions[END_REF] builds on these ideas, extending them by using a more expressive regression function based on a deep neural network. They take advantage of the architecture of CNNs and learn features at different layers, rather than just learning an embedding space for both modalities. The proposed model provides means to automatically generate a list of pseudo-attributes for each visual category consisting of words from Wikipedia articles.

In contrast with the aforementioned methods, Hamm et al. [START_REF] Hamm | Probabilistic Zero-shot Classification with Semantic Rankings[END_REF] introduced the idea of ordinal similarity between classes (eg. d('cat','dog') < d('cat', 'automobile')), claiming that not only this type of similarity may be sufficient for distinguishing cat and truck, but also that it seems a more natural representation since the ordinal similarity is invariant under scaling and monotonic transformation of numerical values. It is also worth mentioning the work of Jayaraman et al. [START_REF] Jayaraman | Zero-shot recognition with unreliable attributes[END_REF] which proposed to leverage the statistics about each attribute error tendencies within a random forest approach, allowing to train zero-shot models that explicitly account for the unreliability of attribute predictions.

Wu et al. [START_REF] Wu | Zero-Shot Event Detection Using Multi-modal Fusion of Weakly Supervised Concepts[END_REF] exploit natural language processing technologies to generate event descriptions. They measure their similarity to images by projecting them into a common high-dimensional space using text expansion. The similarity is expressed as the concatenation of L 2 distances of the different modalities considered. Strictly speaking, there is no metric learning involved but a concatenation of L 2 distances. Finally, Frome et al. [START_REF] Frome | DeViSE: A Deep Visual-Semantic Embedding Model[END_REF] aim at leveraging semantic knowledge learned in the text domain, and transfer it to a model trained for visual object recognition by learning a metric aligning the two modalities. However, in contrast to our work, [START_REF] Frome | DeViSE: A Deep Visual-Semantic Embedding Model[END_REF][START_REF] Wu | Zero-Shot Event Detection Using Multi-modal Fusion of Weakly Supervised Concepts[END_REF] do not explicitly control the quality of the embedding.

Zero-shot learning as transductive and semi-supervised learning

All the previously mentioned approaches consider that the embedding and the consistency function have to be learned from a set of training data of known classes, used in a second time to infer predictions about the images of new classes not available during training. However, a different problem can be addressed when images from the unknown classes are already available at training time and can hence be used to produce a better embedding. In this case, the problem can be cast as a transductive learning problem i.e. the inference of the correct labels for the given unlabeled data only, or a semi-supervised learning problem i.e. the inference of the best embedding using both labeled and unlabeled data.

Wang and Forsyth [START_REF] Wang | Joint learning of visual attributes, object classes and visual saliency[END_REF] proposed MIL framework for jointly learning attributes and object classifiers from weakly annotated data. Wang and Mori [START_REF] Wang | A Discriminative Latent Model of Object Classes and Attributes[END_REF] treated attributes of an object as latent variables and captured the correlations among attributes using an undirected graphical model, allowing to infer object class labels using the information of both the test image and its (latent) attributes. In [START_REF] Mahajan | A joint learning framework for attribute models and object descriptions[END_REF], the class information is incorporated into the attribute classifier to get an attribute-level representation that generalizes well to unseen examples of known classes as well as those of the unseen classes, assuming unlabeled images are available for learning. [START_REF] Socher | Zero-Shot Learning Through Cross-Modal Transfer[END_REF] considered the introduction of unseen classes as a novelty detection problem in a multi-class classification problem. If the image is of a known category, a standard classifier can be used. Otherwise, images are assigned to a class based on the likelihood of being an unseen category.

Fu et al. [START_REF] Fu | Transductive multi-view embedding for zero-shot recognition and annotation[END_REF] rectified the projection domain shift between auxiliary and target datasets by introducing a multi-view semantic space alignment process to correlate different semantic views and the low-level feature view, by projecting them onto a latent embedding space learnt using multi-view Canonical Correlation Analysis. More recently Li et al. [START_REF] Li | Semi-Supervised Zero-Shot Classification With Label Representation Learning[END_REF] learned the embedding from the input data in a semi-supervised large-margin learning framework, jointly considering multi-class classification over observed and unseen classes. Finally, [START_REF] Kodirov | Unsupervised Domain Adaptation for Zero-Shot Learning[END_REF] formulated a regularized sparse coding framework which used the target domain class label projections in the semantic space to regularize the learnt target domain projection, with the aim of overcoming the projection domain shift problem.

Zero-shot learning as a metric learning problem

Two contributions, [START_REF] Mensink | Metric learning for large scale image classification: Generalizing to new classes at near-zero cost[END_REF] and [START_REF] Kuznetsova | Exploiting view-specific appearance similarities across classes for zero-shot pose prediction: A metric learning approach[END_REF], exploit metric learning to Zero shot class description. Mensink et al. [START_REF] Mensink | Metric learning for large scale image classification: Generalizing to new classes at near-zero cost[END_REF] learn a metric adapted to measure the similarity of images, in the context of k-nearest neighbor image classification, and apply it in fact to One Shot Learning to show it can generalize well to new classes. They don't use any attribute embedding space nor consider ZSL in their work. Kuznetsova et al. [START_REF] Kuznetsova | Exploiting view-specific appearance similarities across classes for zero-shot pose prediction: A metric learning approach[END_REF] learn a metric to infer pose and object class from a single image. They use the expression zero-shot to actually denote a (new) transfer learning problem when data are unevenly sampled in the joint pose and class space, and not a Zero-Shot Classification problem where new classes are only known from attribute descriptions.

As far as we know, zero-shot learning has never been addressed explicitly as a metric learning problem in the attribute embedding space, which is one of the key contributions of this paper.

Method

Embedding consistency score

Most of the inference problems can be cast into an optimal framework of the form:

Y * = arg min Y∈Y S(X, Y)
where X ∈ X is a given sample from some modality, e.g. an image or some features extracted from it, Y * is the most consistent association from another modality Y, e.g. a vector of attribute indicators or a textual description, and S is a measure able to quantify the joint consistency of two observations from the two modalities. In this formulation, the smaller the score, the more consistent the samples. One can think of this score as a negative likelihood. When trying to design such a consistency score, one of the difficult aspects is to relate meaningfully the two modalities. One usual approach consists in embedding them into a common representational space A1 where their heterogeneous nature can be compared. This space can be abstract, i.e. its structure can be obtained from some optimization process, or semantically interpretable e.g. a fixed list of attributes or properties each indexed by a tag referring to some shared knowledge or ontology, leading to a p-dimensional vector space. Let ÂX (X) and ÂY (Y) be the two embeddings for each modality X and Y , taking values in X and Y and producing outputs in A.

In this work, it is proposed to define the consistency score as a metric on the common embedding space A. More precisely, we use the Mahalanobis like description of a metric parametrized by a linear mapping W A :

d A (A 1 , A 2 ) = (A 1 -A 2 ) T W A 2 ,
assuming that the embedding space is a vector space, and define the consistency score as:

S(X, Y) = d A ( ÂX (X), ÂY (Y)) = ( ÂX (X) -ÂY (Y)) T W A 2 .
The Mahalanobis mapping W A can be interpreted itself as a linear embedding in an abstract m-dimensional vector space where the natural metric is the Euclidean distance, and acts as a multivariate whitening filter. It is expected that this property will improve empirically the reliability of the consistency score (1) by choosing the appropriate linear mapping.

We are now left with two questions: how to define the embedding? How build the Mahalanobis mapping? We see in the following that these two questions can be solved jointly by optimizing a unique criterion.

Embedding in the attribute space

The main problem addressed in this work is to be able to discriminate a series of new hypotheses that can only be specified using a single modality, the Y one with our notations. In many Zero-Shot Learning studies, this modality is often expressed as the existence or presence of several attributes or properties from a fixed given set. The simplest embedding space one can think of is precisely this attribute space, implying that the Y modality embedding is the identity: ÂY (Y) = Y with A = Y. In this case, the consistency score simplifies as:

S(X, Y) = ( ÂX (X) -Y) T W A 2 (1)
The next step is to embed the X modality into Y directly. We suggest using a simple linear embedding with matrix W X and bias b X , assuming that X is in a d-dimensional vector space. This can be expressed as:

ÂX (X) = max(0, X T W X + b X ). ( 2
)
We use a reLu-type output normalization to keep the significance of the attribute space as property detectors, negative numbers being difficult to interpret in this context.

In the simple formulation proposed here, we do not question the way new hypotheses are specified in the target modality, nor use any external source of information (e.g. word vectors) to map the attributes into a more semantically organized space such as in [START_REF] Socher | Zero-Shot Learning Through Cross-Modal Transfer[END_REF]. We leave the problem of correcting the original attribute description to the construction of the metric in the common embedding space.

Metric learning

The design problem is now reduced to the estimation of three mathematical objects: the linear embedding to the attribute space W X of dimensions d × p, a bias b X of dimension p, and the Mahalanobis linear mapping W A of dimensions p × m, m being a free parameter to choose.

The proposed approach consists in building empirically those objects from a set of examples by appling metric learning techniques. The training set is supposed to contain pairs of data (X i , Y i ) sampling the joint distribution of the two modalities: X i is a vector representing an image or some features extracted from it, while Y i denotes an attribute-based description. Notice that we do not introduce any class information in this formulation: the link between class and attribute representations is assumed to be specified by the use case considered.

The rationale behind the use of metric learning is to transform the original representational space so that the resulting metric takes into account the statistical structure of the data using pairwise constraints. One usual way to do so is to express the problem as a binary classification on pairs of samples, where the role of the metric is to separate similar and dissimilar samples by thresholding (see [START_REF] Bellet | A Survey on Metric Learning for Feature Vectors and Structured Data[END_REF] for a survey on M.L.). It is easy to build pairs of similar and dissimilar examples from the annotated examples by sampling randomly (uniformly or according to some law) the two modalities X and Y and assigning an indicator Z ∈ {-1, 1} stating whether Y i is a good attribute description of X i (Z i = 1) or not (Z i = -1). Metric learning approaches try to catch a data-dependent way to encode similarity. In general, the data manifold has a smaller intrinsic dimension than the feature space, and is not isotropically distributed.

We are now given a dataset of triplets {(X i , Y i , Z i )} N i=1 , the Z indicator stating that the two modalities are similar, i.e. consistent, or not2 . The next step is to describe an empirical criterion that will be able to learn W X , b X and W A . The idea is to decompose the problem in three objectives: metric learning, good embedding and regularization.

The metric learning part follows a now standard hinge loss approach [START_REF] Shalev-Shwartz | Online and batch learning of pseudometrics[END_REF] taking the following form for each sample:

l H (X i , Y i , Z i , τ ) = max 0, 1 -Z i (τ -S(X i , Y i ) 2 ) . (3)
The extra parameter τ is free and can also be learned from data. Its role is to define the threshold separating similar from dissimilar examples, and should depend on the data distribution. The embedding criterion is a simple quadratic loss, but only applied to similar data:

l A (X i , Y i , Z) = max(0, Z i ). Y i -ÂX (X i ) 2 2 . ( 4
)
Its role is to ensure that the attribute prediction is of good quality, so that the difference Y -ÂX (X) reflects dissimilarity due to modality inconsistencies rather than bad representational issues.

The size of the learning problem (d × p + p + p × m) can be large and requires regularization to prevent over fitting. We use a quadratic penalization:

R(W A , W X , b X ) = W X 2 F + b X 2 2 + W A 2 F (5)
where . F is the Frobenius norm.

The overall optimization criterion can now be written as the sum of the previously defined terms:

L(W A , W X , b X , τ ) = i l H (X i , Y i , Z i , τ ) + λ i l A (X i , Y i , Z i ) +µR(W A , W X , b X ) (6)
where λ and µ are hyper-parameters that are chosen using cross-validation. Note that the criterion (6) can also be interpreted as a multi-objective learning approach since it mixes two optimal but dependent issues: attribute embedding and metric on the embedding space.

To solve the optimization problem, we do not follow the approach proposed in [START_REF] Shalev-Shwartz | Online and batch learning of pseudometrics[END_REF] since we also learn the attribute embedding part W X jointly with the metric embedding W A . We use instead a global stochastic gradient descent (see section 4 for details).

Application to image recognition and retrieval

The consistency score ( 1) is a versatile tool that can be used for several image interpretation problems. Section 4 will evaluate the potential of our approach on three of them.

Zero-shot learning

The problem can be defined as finding the most consistent attribute description given the image to classify, and a set of exclusive attribute class descriptors

{Y * k } C k=1
where k is the index of a class:

k * = arg min k∈{1...C} S(X, Y * k ) (7)
In this formulation, classifying is made equivalent to identifying between the C classes the best attribute description. A variant of this scheme can exploit a voting process to identify the best attribute among a set of k candidates, inspired from a k-nearest neighbor approach.

Few-shot learning Learning a metric in the embedding space can conveniently be used to specialize the consistency score to new data when they are available. We study a simple fine tuning approach using stochastic gradient descent on criterion (6) applied to novel triplets (X, Y, Z) from unseen classes only, starting with the model learned with seen classes. This makes "few-shot learning"possible. The decision framework is identical to the ZSL one.

Zero-shot retrieval The score (1) can also used to retrieve the data from a given database that have at least a consistent level λ with a given query defined in the Y (or A) modality:

Retrieve(A, λ) = {X ∈ X / S(X, A) < λ}
The performance is usually characterized by precision-recall curves.

Experiments

This section presents the experimental validation of the proposed method. The section first introduces the 4 datasets evaluated as well as the details of the experimental settings. The method is empirically evaluated on three different tasks as described in section 3: Zero-Shot-Learning (ZSL), Few-Shot Learning (FSL) and Zero-Shot Retrieval (ZSR). The ZSL experiments aim at evaluating the capability of the proposed model to predict unseen classes. This section also evaluates the contribution of the different components of the model to the performance, and makes comparisons with state-of-the-art results. In the FSL experiments, we show how the ZSL model can serve as good prior to learning a classifier when only a few samples of the unknown classes are available. Finally, we evaluate our model on a ZSR task, illustrating the capability of the algorithm to retrieve images using attribute-based queries.

Datasets and Experimental Settings

The experimental valuation is done on 4 public datasets widely used in the community, allowing to compare our results with those recently proposed in the literature: the aPascal&aYahoo (aP&Y) [START_REF] Farhadi | Describing objects by their attributes[END_REF], Animals with Attributes (AwA) [START_REF] Lampert | Learning to detect unseen object classes by between-class attribute transfer[END_REF], CUB-200-2011 (CUB) [START_REF] Wah | The Caltech-UCSD Birds-200-2011 Dataset[END_REF] and SUN attribute (SUN) [START_REF] Patterson | The SUN Attribute Database: Beyond Categories for Deeper Scene Understanding[END_REF] datasets (see Table 1 for few statistics on their content). Theses datasets exhibit a large number of categories (indoor and outdoor scenes, objects, person, animals, etc.) and attributes (shapes, materials, color, parts, etc.)

These datasets have been introduced for training and evaluating ZSL methods and contain images annotated with semantic attributes. More specifically, each image of the aP&Y, CUB and SUN datasets has its own attribute description, meaning that two images of the same class can have different attributes. This is not the case for AwA where all the images of a given class share the same attributes. As a consequence, in the ZSL experiments on aP&Y, CUB and SUN, the attribute representation of unknown classes, required for class prediction, is taken as their mean attribute frequencies.

In order to make comparisons with previous works possible, we use the same training/testing splits as [START_REF] Farhadi | Describing objects by their attributes[END_REF] (aP&Y), [START_REF] Lampert | Learning to detect unseen object classes by between-class attribute transfer[END_REF] (AwA), [START_REF] Akata | Evaluation of Output Embeddings for Fine-Grained Image Classification[END_REF] CUB and [START_REF] Jayaraman | Zero-shot recognition with unreliable attributes[END_REF] (SUN).

Regarding the representation of images, we used both the VGG-VeryDeep-19 [START_REF] Simonyan | Very Deep Convolutional Networks for Large-Scale Image Recognition[END_REF] and AlexNet [START_REF] Krizhevsky | ImageNet Classification with Deep Convolutional Neural Networks[END_REF] CNN models, both pre-trained on imageNet -without fine tuning to the attribute datasets -and use the penultimate fully connected layer (e.g., FC7 4096-d layer for VGG-VeryDeep-19) for representing the images. Very deep CNN models act as generic feature extractors and have been demonstrated to work well for object recognition. They have been also used in many recent ZSL experiments and we use exactly the same descriptors as [START_REF] Akata | Evaluation of Output Embeddings for Fine-Grained Image Classification[END_REF][START_REF] Zhang | Zero-Shot Learning via Semantic Similarity Embedding[END_REF].

One of the key characteristics of our model is that it requires a set of image/attributes pairs for training. Positive (resp. negative) pairs are obtained by taking the training images associated with their own provided attribute vector (resp. by randomly assigning attributes not present in the image) and are assigned to the class label '1 (resp. '-1). In order to bound the size of the training set we generate only 2 pairs per training image, one positive and one negative.

Our model has three hyper-parameters: the weight λ, the dimensionality of the space in which the distance is computed (m) and the regularization parameters µ. These hyper-parameters are estimated through a grid search validation procedure by randomly keeping 20% of the training classes for cross-validating the hyper-parameters, and choosing the parameters giving best accuracy for these so-obtained validation classes. The parameter are searched in the following ranges: m ∈ [20%, 120%] of the initial attribute dimension, λ ∈ [0.05, 1.0] and µ ∈ [0.01, 10.0. τ is a parameter learned during training.

Once the hyper-parameters are tuned, we take the whole training set to learn the final model and evaluate it on the test set (unseen classes in case of ZSL).

The optimization of W A and W X is done with stochastic gradient descent, the parameters being initialized randomly with normal distribution. The size of the mini-batch is of 100. As the objective function is non-convex, different initializations can give different parameters. We addressed this issue by doing 5 estimations of the parameters starting from 5 different initializations and selecting the best one on a validation set (we keep a part of the train set for doing this and fine-tune the parameters on the whole train set when the best initialization is known).We use the optimizer provided in the TensorFlow framework [START_REF] Abadi | TensorFlow: Large-scale machine learning on heterogeneous systems[END_REF]. Using the GPU mode with a Nvidia 750 GTX GPU, learning a model (W A Table 2: Zero-shot classification accuracy (mean ± std). We report results both with VGG-verydeep-19 [START_REF] Simonyan | Very Deep Convolutional Networks for Large-Scale Image Recognition[END_REF] and AlexNet [START_REF] Krizhevsky | ImageNet Classification with Deep Convolutional Neural Networks[END_REF] features for fair comparisons, whenever it's possible (AwA images are not public anymore preventing the computation of their AlexNet representations). and W X ) takes 5-10 minutes for a given set of hyper-parameters. Computing image/attribute consistency takes around 4ms per pair.

Zero-Shot Learning Experiments

The experiments follow the standard ZSL protocol: during training, a set of images from known classes is available for learning the model parameters. At test time, images from unseen classes are processed and the goal is to find the class described by an attribute representation most consistent with the images. Table 2 gives the performance of our approach on the 4 datasets consideredexpressed as multi-class accuracy -and makes comparisons with state-of-the-art approaches. The performances of previous methods are taken from [START_REF] Zhang | Zero-Shot Learning via Semantic Similarity Embedding[END_REF][START_REF] Akata | Evaluation of Output Embeddings for Fine-Grained Image Classification[END_REF][START_REF] Zhang | Zero-shot learning via joint latent similarity embedding[END_REF]. Performance is reported with 2 different features i.e. VGG-VeryDeep-19 [START_REF] Simonyan | Very Deep Convolutional Networks for Large-Scale Image Recognition[END_REF] and AlexNet [START_REF] Krizhevsky | ImageNet Classification with Deep Convolutional Neural Networks[END_REF] for fair comparisons. As images of AwA are not public anymore it is only possible to use the features available for download. On the four datasets our model achieves above state-of-the-art performance (note : [START_REF] Zhang | Zero-shot learning via joint latent similarity embedding[END_REF] was published after our submission), with a noticeable improvement of more than 8% on aP&Y.

As explained in the previous section, our model is based on a multi-objective function trying to maximize metric discriminating capacity as well as attribute prediction. It is interesting to observe how the performance degrades when one of the two terms is missing. In Table 2, the 'Ours w/o ML' setting makes use of the Euclidean distance i.e. W A = I. The 'Ours w/o constraint' setting is when the attribute prediction term (Eq. 4) is missing in the criterion. This term gives a 4% improvement, on average.

Figure 1a shows the accuracy as a function of the embedding dimension. This projection maps the original data in a space in which the Euclidean distance is good for the task considered. It can be seen as a way to exploit and select the correlation structure between attributes. We experimented that the best performance is generally obtained when the dimension of this space less than 40% smaller than the size of the initial attribute space.

Few-Shot Learning

Few-shot learning corresponds to the situation where 1 (or more) annotated example(s) from unseen classes are available at test time. In this case our model is first trained using only the seen classes (same as with ZSL), and we introduced the examples from unseen class data one by one before fine-tuning the model parameters by doing a few more learning iterations using these new data only.

Figure 1b shows the accuracy evolution, given as a function of the number of additional images from the unseen classes. Please note that for the SUN dataset we have used a maximum of 10 additional examples as unseen classes contain only 20 images. We observed that knowing even a very few number of annotated examples significantly improves the performance. It is a very encouraging behavior for large-scale applications where annotations for a large number of categories are hard and expensive to get.

Zero-Shot Retrieval

The task of Zero-Shot image Retrieval consists in searching an image database with attribute-based queries. For doing this, we first train our model as for standard ZSL. We then take the attribute descriptions of unseen classes as queries, and rank the images from the unseen classes based on the similarity with the query. Table 3 reports the mean average precision on the 4 datasets. Our model outperforms the state-of-the-art SEE method [START_REF] Zhang | Zero-Shot Learning via Semantic Similarity Embedding[END_REF] by more than 10% on average. Figure 2 shows the average precision for each class of the 4 datasets. In the aP&aYa dataset, the 'donkey', 'centaur' and 'zebra' classes have a very low average precision. This can be explained by the strong visual similarity between these classes which only differ by a few attributes.

Conclusion

This paper has presented a novel approach for zero-shot classification exploiting multi-objective metric learning techniques. The proposed formulation has the nice property of not requiring any ground truth at the category level for learning a consistency score between the image and the semantic modalities, but only requiring weak consistency information. The resulting score can be used with versatility on various image interpretation tasks, and shows close or above stateof-the-art performance on four standard benchmarks. The formal simplicity of the approach allows several avenues for future improvement. A first one would be to provide a better embedding on the semantic side of the consistency score ÂY (Y). A second one would be to explore more complex functions than the linear mappings tested in this work, and introduce, for instance, deep network architectures.

Fig. 1 :

 1 Fig. 1: (a) ZSL accuracy as a function of the dimensionality of the metric space; best results are obtained when the dimension of the metric embedding is less than 40% of the image space dimension. It also shows the improvement due to the attribute prediction term in the objective function. (b) Few-shot learning: Classification accuracy (%) as a function of the amount of training examples per unseen classes.

Fig. 2 :

 2 Fig. 2: Precision Recall curve for each unseen class by dataset. For CUB dataset we randomly choose 10 classes (best viewed on a computer screen).

Table 1 :

 1 Dataset statistics

	Dataset	#Training classes #Test classes #Instances #Attributes
	aPascal & aYahoo [23]	20	12	15,339	64
	Animals with Attributes [3]	40	10	30,475	85
	CUB 200-2011 [44]	150	50	11,788	312
	SUN Attributes [45]	707	10	14,340	102

Table 3 :

 3 Zero-Shot Retrieval task: Mean Average Precision (%) on the 4 datasets aP&Y AwA CUB SUN Av. Zhang et al. [11] 15.43 46.25 4.69 58.94 31.33 Ours (VGG features) 36.92 68.1 25.33 52.68 45.76

We use the letters A and A in our notations since we will focus on the space of attribute descriptions as the embedding space.

To make notations simpler, we do not rename or re-index from the original dataset the pairs of data for the similar and dissimilar cases.