Eric Lichtfouse
email: eric.lichtfouse@dijon.inra.fr

Patrick Wehrung

Pierre Albrecht

Plant wax n-alkanes trapped in soil humin by non-covalent bonds

Keywords: soil organic matter, humin, plant wax n-alkane, maize, pesticide, 13C, GC-IRMS, pollutant

Soil organic matter is a very complex mixture of compounds derived from the decay of numerous living organisms. Earlier reports suggest that soil organic molecules may occur unaltered in a "bound" form, e.g. by encapsulation in the humic substances matrix or by binding involving weak forces such as H bonds and Van der Waals forces. This issue is relevant to the release of pesticides and other toxic compounds in waters during several years, which suggests a particular mechanim of storage of organic compounds in soil organic matter. However, so far, the mechanim of weak binding is poorly known due to possible analytical bias in the isolation of "weakly-bound" molecules, and to the lack of analytical approaches that can distinguish a "free" compound from its "weakly-bound" counterpart. Here, we analysed the 13 C isotope composition of plant-derived soil n-alkanes in a soil sample from an experimental field cropped 23 years with maize to label soil organic carbon. Indeed, cultivating maize, a C 4 plant with 13 C-enriched carbon, on a soil previously cropped with C 3 plants, introduces 13 C-enriched carbon in the soil. Thus maize-derived soil carbon can be distinguished from previously-cropped plant soil carbon by 13 C analysis. We used a recently developped analytical technique, gas chromatography -isotope ratio monitoring mass spectrometry (GC-IRMS), that allows to measure the 13 C isotope ratio of individual substances occurring in complex mixtures. We analysed three pools of C 27 -C 33 n-alkanes in the same soil sample: 1) "free" n-alkanes that were extracted with CHCl 3 -MeOH, 2) "bound" n-alkanes that were extracted from humin, the macromolecular part of soil organic matter, and 3) "bound" n-alkanes that were released by pyrolysis of the pre-extracted humin. Our results show that free n-alkanes are 13 C-enriched by +5.8-7.0‰ versus their bound n-alkanes counterparts. This finding have several implications. First, this clear isotope difference of the the same substance occurring either in free or bound form proves that there is no analytical bias such as a lack of exhautive extraction. Second, the higher 13 C content of free n-alkanes evidences their higher turnover versus bound n-alkanes. Here a first-order kinetic law yields an age difference of 7 years between free and bound n-alkanes. Third, since n-alkanes are apolar compounds our results demonstrate the existence of a mechanism of weak binding involving either encapsulation or weak forces. This mechanism could explain why chemicals such as pesticides are well preserved in soil then released in waters several years after the end of their use in cropping systems. It is also relevant to the storage of organic-N compounds as possible plant nutrient precursors.

Soil organic matter is a complex mixture of organic substances inherited from the decay of plants and micro-organisms (1). Its main part, humic substances, is made of a high network (2) which is able to trap small components such as pesticides and phthalates [START_REF] Bollag | [END_REF]. However, the binding pathways of molecules with humic substances are poorly known [START_REF] Bollag | [END_REF]. Several molecular studies using selective chemical degradation with labelled reagents have revealed the occurrence of strong covalent bonds between xenobiotics and humic substances (4). Nonetheless, the occurrence of weak interactions at the molecular level is so far poorly known due to the lack of suitable analytical tools. Recent work involving 13 C-NMR characterisation of 13 C-labelled organic substances bound with humic substances strongly suggests the existence of weak bonds (5). Here, we bring indirect molecular and isotopic evidence for the occurrence of non-covalent bonds within the soil matrix. We isotopically distinguish two pools of the same molecular substances, plant wax n-alkanes, within the soil matrix: a first pool amenable by organic extraction of the soil, and a second, bound pool amenable by extraction of the soil humin. This sequestration of apolar substances within humin by non-covalent bonds represents a novel way of preservation of soil organic matter. Furthermore, weak binding of apolar molecules has implications in agricultural and environmental fields.

Soil organic substances can be naturally labelled by cultivation of Zea mays, a 13 C-enriched plant, on soils previously cropped with 13 C-depleted plants such as wheat (6). As a result, soil organic carbon becomes progressively enriched in 13 C due to the addition of maize carbon. At any time of maize cultivation, it is thus possible to distinguish by isotope balance the maize and wheat contributions within each soil organic component, allowing long-term turnovers to be calculated. Here, we hypothesised that if molecules are somehow trapped within the soil matrix, they should have a lower turnover than 'free' molecules, due for instance to slower bioavailability and degradation.

The free and bound fractions were isolated from a soil cropped 23 years with maize. The free fraction enriched in wax n-alkanes was obtained by exhaustive extraction of the soil with CHCl 3 -MeOH followed by silica gel chromatography steps (7). Humin was then isolated from the extracted soil by NaOH extraction to remove humic and fulvic acids, followed by HF-HCl treatment to remove the main minerals (8). CHCl 3 -MeOH extraction of humin followed by silica gel chromatography gave the bound fraction enriched in n-alkanes.

Long chain n-alkanes show similar distributions for the free and bound soil fractions (Figure 1). This typical distribution of odd carbon-numbered n-alkanes implies that both fractions derived from plant sources (9), namely maize and previously grown wheat, rather than from other living organisms. The high predominance of odd C 27 -C 35 n-alkanes is indeed a typical feature of plant cuticular waxes (10). Moreover, since n-alkanes are non-functionalized, apolar substances, a similar distribution in the free and bound fractions means clearly that plants waxes have been trapped without chemical changes, such as degradation and covalent binding. Therefore, the occurrence of these unaltered components in the humin extract shows that plant waxes can be trapped into the soil matrix by weak bonds. Nonetheless, based on these molecular observations, it could also be argued that bound n-alkanes result from a low efficiency of soil extraction, unless an independent information shows that bound and free soil n-alkanes have different properties. A such information is given by isotope analysis at the molecular level. Indeed, δ 13 C values of bound n-alkanes are significantly depleted in 13 C relatively to free n-alkanes (Table 1, Figure 2). Given that each soil n-alkane is a mixture of 13 C-enriched n-alkane from maize and 13 C-depleted n-alkane from previously grown wheat, this finding implies that bound n-alkanes have less maize carbon than their free counterparts. Calculation of the maize contribution within individual compounds by isotope balance (5) indeed reveals that free n-alkanes contain 13 ± 3 % more maize derived n-alkanes than bound n-alkanes (Table 1). Bound n-alkanes are therefore composed of molecules that have on the average a lower turnover and a lower age than free n-alkanes. This age difference can be estimated by calculation of the cultivation time t at which free n-alkanes have had the same maize carbon content as the bound n-alkanes from the 23-years maize cultivated soil, using the following first order kinetic law established for the C 31 n-alkane (6):

Ln[(100 -M)/100] = -0.031 t where M is the percentage of maize n-alkane within the C 31 soil n-alkane. The calculation gives a value of 16 years, meaning that if the bound C 31 n-alkane of the 23-years maize cultivated soil would derive from a once free C 31 n-alkane, then the trapping has occurred 7 years earlier on the average. 13 C values relative to the Pee Dee Belemnite standard : δ 13 C = [( 13 C/ 12 Csample -13 C/ 12 Cstd)/( 13 C/ 12 Cstd)]x10 3 . δ 0 refers to soils from the Boigneville experimental field (6) cultivated only with wheat. δ m refer to leaf maize cuticular waxes (13). 'Free' and 'bound' refer to n-alkanes occurring respectively in the soil extract and humin extract of the soil from the field cultivated 23 years with maize. M is the percentage of maize derived n-alkane in each soil n-alkane (6) : M = 100.(δ soilδ 0 )/δ mδ 0 ), where δ soil is either δ free or δbound. ΔM = M free -M bound . Noteworthy, the good accuracy of such calculations is confirmed by a recent study showing the absence of carbon isotope fractionation of plant n-alkanes in a 23-year field decomposition experiment (14). Detailed experimental procedures are described elsewhere (6,7,8). The Boigneville soil is a Hapludalf developed on loess containing 22% clay, 1% organic carbon, and 0.1% organic nitrogen. Typically, 0-30 cm cores, corresponding to the ploughed horizon, were sampled at random position in the fields, mixed, dried at 20°C, sieved to 2 mm to remove coarse debris, then finely ground; 100g were thoroughly extracted with CHCl 3 -MeOH 3/1 v/v. Isolation of free n-alkanes : the extract (10.5 mg) was fractionated into acid and neutral fractions by passage through silica gel impregnated with KOH followed by HCO 2 H acidification. The neutral fraction was fractionated into polar, alcohol, ketone, and hydrocarbon-ester fractions by silica-gel thin layer chromatography using CH 2 Cl 2 as developer and 1,2:3,4-dibenzanthracene, friedelin, and cholesterol as reference compounds. Hydrocarbon-ester fractions were fractionated into alkane-alkene (0.4 mg, including the free n-alkanes), aromatic, and ester fractions by silica-gel thin layer chromatography using n-hexane as developer and 1-phenyldodecane, 2-methylphenanthrene, and 1,2:3,4dibenzanthracene as reference compounds. Isolation of bound n-alkanes : the CHCl 3 -MeOH extracted soil was treated with NaOH 0.1 M to remove fulvic and humic acids, then with HF/HCl to give 1.34 g of concentrated humin. The humin extract (28,1 mg) and the alkanealkene fraction (8.5 mg, including the bound n-alkanes) were separated from the humin as described above. Isotopic analyses of individual n-alkanes from alkane-alkene fractions (3 to 4 replicates, deviation ≤ 0.3‰) were carried out under a continuous helium flow using a Hewlett Packard gas chromatograph coupled to a CuO furnace and a Finigan Mat 252 isotope ratio mass spectrometer. Accurate isotope values were measured by comparison with CO 2 , 5α-androstane and n-tetracontane standards. Furthermore, we have also studied another bound alkane fraction released by 600°C pyrolysis of the extracted humin. Again, we observed the odd-carbon numbered predominance of C 27 -C 33 nalkanes, a typical feature indicating the input of plant waxes (10). These pyrolysate n-alkanes have also been analysed for 13 C compositions. The data show that δ 13 C values of pyrolysate n-alkanes and of humin extract n-alkanes are very close (Table 2). A such finding clearly confirm the occurrence of an older pool of wax n-alkane trapped in humin. Moreover, it means that a part of apolar organic molecules sequestred into humin can not be released by regular solvent extraction. Plant wax n-alkanes may be enclosed within the soil matrix by different processes involving van der Waals forces. First, wax n-alkanes could be adsorbed on soil humic substances by hydrophobic-hydrophobic interactions (3), as suggested by the occurrence of n-alkane in humic acids (11). Indeed, in view of the high polarity of the soil aqueous solution, apolar substances are prone to absorption on the hydrophobic surfaces of the organic matter (3). This hypothesis is in agreement with the fact that the amount of pesticide bound residues is increasing with organic matter content (3). Alternatively, plant wax n-alkanes could be encapsulated as inclusion compounds in organo-mineral lattices during humification processes. Here, there is also no binding between the n-alkane guest and the soil host, except van der Waals forces. Whatever mode of sequestration, our findings show unambiguously the existence of a pool of apolar molecules trapped by non-covalent binding.

Several implications may be foreseen from the occurrence of weakly bound apolar substances in soils. Firstly, it gives evidence for a novel and alternative process of formation of soil organic matter which involves preservation by non covalent binding. Indeed, it contrasts sharply with other possible processes such as polycondensation of small molecules (3) and selective preservation of resistant biopolymers (8). Moreover, our findings are in agreement with recent 13 C-NMR studies showing non-covalent interactions between fulvic acids and acenaphthenone (5). Secondly, it has been shown that lipids act as cements of soil mineral aggregates (12), thus preventing soil erosion by intense cropping. If that is the case, then keeping a high pool of trapped lipids, later released by ploughing, may favour sustainable agricultural practices. Thirdly, a broad range of xenobiotics including pesticides (3), polychlorobiphenyls (3) and aromatic hydrocarbons (7) are highly hydrophobic. It is therefore expected that such compounds would be trapped within the soil matrix in the similar way as n-alkane by non covalent binding. Trapping by non covalent bonds may thus lead to the temporary storage of unaltered xenobiotics within the soil matrix.

Figure 1 .

 1 Figure 1. Distribution of n-alkanes from the soil extract (free) and from the humin extract (bound). Relative concentrations were measured by capillary gas chromatography coupled to a flame ionisation detector.

Figure 2 .

 2 Figure 2. δ 13 C values of n-alkanes from the soil extract (free) and from the humin extract (bound).

Table 1 .

 1 δ 13 C values of n-alkanes and maize C content

	C nb.	δ 0	δ m	δ free	δ bound	M free M bound ΔM
		(‰)	(‰)	(‰)	(‰)	(%)	(%)	(%)
	__________________________________________________________
	27	-33.0 -19.1 -24.0	-26.0	64.7	50.4	14.3
	29	-35.7 -18.4 -28.3	-30.0	42.8	32.9	9.9
	31	-35.7 -20.6 -28.1	-29.7	50.3	39.7	10.6
	33	-33.7 -22.2 -25.6	-27.6	70.4	53.0	17.4
	__________________________________________________________

Isotopic compositions are expressed in per mil as δ

Table 2 .

 2 δ13 C values of bound n-alkanes from CHCl 3 -MeOH humin extract, and from the 600°C pyrolysate of the extracted humin.

		Humin extract	Humin pyrolysate
	CARBON	δ bound	δ bound
	NUMBER	(‰)	(‰)
	_____________________________________________
	27	-26.0	-26.5
	29	-30.0	-29.5
	31	-29.7	-29.7
	33	-27.6	-27.8
	_____________________________________________

* author for correspondance. <lichtfouse@ensaia.u-nancy.fr>


Cédric Maigrot
email: cedric.maigrot@irisa.fr

Vincent Claveau
email: vincent.claveau@irisa.fr

Ewa Kijak
email: ewa.kijak@irisa.fr

Fusion-based multimodal detection of hoaxes in social networks

Keywords: Hoax detection, Knowledge fusion, Text analysis, Image analysis, Source credibility

Social networks make it possible to share information rapidly and massively. Yet, one of their major drawback comes from the absence of verification of the piece of information, especially with viral messages. This is the issue addressed by the participants to the Verification Multimedia Use task of Mediaeval 2016. They used several approaches and clues from different modalities (text, image, social information). In this paper, we explore the interest of combining and merging these approaches in order to evaluate the predictive power of each modality and to make the most of their potential complementarity.

I. INTRODUCTION

Social networks are becoming increasingly important in our professional and personal lives, thanks to their ability to keep us informed through the news shared by our contacts. It has become common that important news is first broadcast on social networks before being processed by traditional media. This speed of the propagation of information, combined with the number of people receiving it, defines the virality of information. But this virality has a drawback: users rarely check the veracity of the information they share. It is therefore common to see false and/or manipulated information circulating (including hoaxes, rumors, urban legends, or fake news). In addition, even information identified as a hoax can be difficult to stop when it is already shared a large number of times.

The purpose of this project is to automatically detect the veracity of viral information. The ultimate goal is to create for example a system that will warn the user before he shares false information. Starting from the observation that this viral information is often composed of multimedia elements (text accompanied by images or videos), we propose to develop systems exploiting different modalities. We present first approaches exploiting only the textual content, the contents of the images or the sources cited in the messages, and then different strategies of combination of these mono-modal approaches. These different approaches are evaluated and discussed on the shared data of the Verifying Multimedia Use (VMU) task from the challenge MediaEval2016 that was specifically on this issue. From the methods of all the teams involved in this task, we explore different fusion strategies to analyze the contribution of different approaches and the predictive ability of a collaborative system. After a review of the state of the art in the next section, we present in Section III the task VMU and its datasets. We then present in Section IV the approaches we have implemented, as well as the systems proposed by the other teams participating in the VMU task. Sec. V presents the experimental protocol and the results obtained by the different approaches. Different fusion strategies are tested and discussed in Sec. VI. Finally, Sec. VII summarizes the main observations and discusses possible avenues for the future.

II. STATE OF THE ART

The analysis of the information veracity is a research challenge that is studied from several angles. Here, we are only interested in viral information circulating in social networks (SN). It should be noted that research about fact checking is not discussed here; even if it shares a common goal of verification, the differences in the nature of information (source, mode of dissemination) and purpose (help for journalism) imply different methods from those used for hoax.

Several families of features have been exploited to detect hoax in social networks. We can notably cite:

• the textual indices; the message itself obviously brings potentially exploitable information. • multimedia cues, in the case of messages containing images or videos; these media contents can sometimes be analysed to detect some deliberate modifications (tampering). • the diffusion in the SN: what is the source of the information, what is the path of the message. Analysis of the sources of messages and relations between members has been the subject of several studies. [START_REF] Golbeck | Inferring binary trust relationships in web-based social networks[END_REF] offer a measure of trust between users of SN, which characterizes the trust of a relationship between two users. This measure of trust can thus serve as an index to judge the reliability of the information transmitted. Several approaches have also been proposed to determine the credibility of a source. [START_REF] Gupta | Evaluating event credibility on twitter[END_REF] adapt the PageRank algorithm [START_REF] Page | The pagerank citation ranking: bringing order to the web[END_REF] on a graph representing the relationships between tweets, the authors of these tweets and the events associated with these tweets. These approaches, however, require extensive knowledge of the network that makes them difficult to apply in practice for large-scale commercial SN.

The analysis of the mode of dissemination of the messages as well has been the subject of several works, the purpose of which is to distinguish rumors from conventional messages by observing how they propagate in the SN. These analyzes are based on diffusion models but require access to a large part of the network to follow these messages, which is rarely possible with the mainstream SN we are aiming for.

Analysis of multimedia clues is at the heart of the European project InVid1 which focuses on the automatic detection of fake videos in SN. For this purpose, [START_REF] Mezaris | Online multi-task learning for semantic concept detection in video[END_REF] work on images from videos and analyze them with forensics techniques to detect modifications in the images. The problem of image authenticity has been addressed in the Image forensics community in several ways [START_REF] Farid | Photo Forensics[END_REF], [START_REF] Warif | Copy-move forgery detection: Survey, challenges and future directions[END_REF]. However most of the approaches poorly perform on images circulating on SN [START_REF] Zampoglou | Large-scale evaluation of splicing localization algorithms for web images[END_REF]. In this context, other approaches exploits external information to determine the integrity of an image. In this case, similar or identical images are searched in database (or Web) in order to determine if the image has been modified or diverted. The problem of finding similar images is an active domain whose latest work is based on deep convolutional neural networks to describe and compare images [START_REF] Gordo | Deep image retrieval: Learning global representations for image search[END_REF].

Other works exploit several of these indices together. This is the case of [START_REF] Middleton | Reveal project-trust and credibility analysis[END_REF], which, within the framework of the European Reveal Project2 , aims to develop tools and services for information verification in social networks, from a journalistic and professional perspective. Different media such as image [START_REF] Zampoglou | Detecting image splicing in the wild (web)[END_REF], video [START_REF] Middleton | Extracting attributed verification and debunking reports from social media: mediaeval-2015 trust and credibility analysis of image and video[END_REF], and text are analyzed. However, their purpose is not to develop automatic tools but to help journalists. These works also extensively use external resources [START_REF] Gottron | Focused exploration of geospatial context on linked open data[END_REF].

The European project PHEME3 [START_REF] Derczynski | Analysis of named entity recognition and linking for tweets[END_REF] is interested in detecting rumors in SN or online media. In particular, they study the responses and reactions to tweets to decide of their truth. This project is not intended, as we do in this article, to classify the tweet based on its unique content, but instead, aims to crowdsourced verification, that is to say relies on (human) analyzes produced by the users of the SN.

III. Verifying Multimedia Use TASK FROM MediaEval2016

The Verifying Multimedia Use (VMU) task of the 2016 MediaEval evaluation campaign proposed to classify messages from Twitter according to their veracity (classes true or false, with the possibility of using an unknown class if the system is not able to take a decision). Allowing the system not to decide can result in high precision for the true and false classes [START_REF] Boididou | Verifying multimedia use at mediaeval 2016[END_REF].

In the provided evaluation data, all messages are labeled either true or false, and all are accompanied by one or more images, or a video (see Fig. 1). However, several messages may share the same image. It is important to note all messages sharing the same video or a image will always have the same class (creating some bias resulting from the dataset building method). While some images are only used by a single message, others are shared by more than 200 messages. In addition, messages are grouped by event. The sizes of the events are not balanced. For instance, the biggest event in this collection is Paris Attack with 580 messages sharing 25 different images or videos, while the smaller ones are the events Soldier Stealing and Ukrainian Nazi with one single message and one single image. Tab. I shows the distribution of data between learning and test sets, as well as the number of images and videos per set. It should be noted that in Sec. V, the results presented are those obtained on the test set, and that the fusion techniques presented in Sec. VI are used on the predictions of participants' systems on this test set (i.e. the submitted runs).

Several features have been proposed by the organizers, falling into three categories: textual, user or image. The proposed textual features, noted T , are shallow ones: number of words, length of the text, occurrence of the symbols ? and !, presence of the symbols ? and ! as well as happy or unhappy emoticons, pronouns at the 1st, 2nd or 3rd person, number of capital letters, number of positive and negative words, number of mentions Twitter, hashtags, urls and retweets.

The set of feature associated with the user, noted U, consists of the following information: number of friends, number of subscribers (followers), ratio of the number of friends on the number of subscribers, if the account contains an url, if the account is verified and the number of messages posted.

The set of features associated with the images, noted FOR, comes from the domain of forensics: indexes of double JPEG compression [START_REF] Bianchi | Image forgery localization via block-grained analysis of jpeg artifacts[END_REF], Block Artifact Grid [START_REF] Li | Passive detection of doctored jpeg image via block artifact grid extraction[END_REF], Photo Response Non-Uniformity [START_REF] Goljan | Defending against fingerprintcopy attack in sensor-based camera identification[END_REF] and Benford-Fourier coefficients [START_REF] Pasquini | A benford-fourier jpeg compression detector[END_REF].

IV. SYSTEMS PARTICIPATING TO MEDIAEVAL VMU TASK

Four teams participated in the task for a total of 14 bids. Teams are subsequently denoted by LK (our team), MMLAB, MCG-ICT and VMU (task organizers). In this section, we present the approaches we have developed, then briefly the approaches proposed by the other participating teams.

In our approaches, all messages sharing the same image are associated with the same true, false, or unknown class. It is thus sufficient to determine the class of each image and assign the predicted class to all messages associated with this image, according to the following rule: a message is predicted as true if all the associated images are classified true; it is predicted false otherwise. We propose three approaches: the first is based on the textual content of the message; the second on sources; the third on the images. None of our approaches uses the descriptors T , U or FOR presented in Sec. III, but other participants do. We present their approaches at the end of this section. A comparative study between all approaches will be proposed later in the article.

A. Text-based run (LK-T)

This approach exploits the textual content of messages and does not rely on additional external knowledge. As previously explained, a tweet is classified from the associated image. An image is itself described by the union of the textual contents of the messages that use this image. The idea behind this approach is to capture similar comments between a test set message and those of the training set (e.g. "it's photoshopped") or more stylistic aspects (e.g. presence of emoticons, popular expressions. . . ).

Let I q be the description of an unknown image (i.e. the union of the textual contents of all the messages that use this image), and {I di } be the set of descriptions of the images in the training set. The class of I q is determined by a k-nearest neighbors classification (majority vote). The calculation of similarity between textual descriptions is therefore at the heart of this approach, as the k images in {I di } whose descriptions are the most similar to I q should be identified. The similarity used is Okapi-BM25 [START_REF] Robertson | Okapi at TREC-7: Automatic Ad Hoc, Filtering, VLC and Interactive[END_REF].

B. Source-based run (LK-S)

This approach, similar to [START_REF] Middleton | Extracting attributed verification and debunking reports from social media: mediaeval-2015 trust and credibility analysis of image and video[END_REF], is based on external knowledge (static). As for the previous approach, the prediction is done at the image level, and the image is represented by the union of the textual contents (translated in English if necessary) of the messages in which it appears. The prediction is made by detecting a source of confidence in the description of the image. Two types of sources are sought: 1) a known information organization; 2) an explicit quote of the source of the image. For the first type of source, we determine a list of news agencies and newspapers in the world (mainly French and English) based on established lists 4 , and information television networks (French and English) 5 . For the second type, we manually define several extraction patterns, like photographed by + Name, captured by + Name, . . . Finally, an image is classified as unknown by default, except if a source of confidence is found in his description.

C. Image-based runs (LK-I and LK-I2)

In this approach, only the content of the images is used to make a prediction. Tweets containing videos are not handled by this approach and get the unknown class. It relies on a similar-image search in a database of reference images, listed as false or true. A given query image (which we are looking for the class) receives the class of the most similar image of the database (if it exists), otherwise query image and associated messages are labeled unknown.

The image database was built by collecting images from five websites specialized in debunking false information: www.hoaxbuster.com, hoax-busters.org, urbanlegends.about.com, snopes.com and www.hoax-slayer.com. The database contains about 500 original images (i.e. true) and 7,500 tampered images (i.e. false).

The descriptors we generate from images are calculated using a deep convolutional neural network (CNN) [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]. The images are first resized to the standard size of 544 × 544 and passed in the CNN layers [START_REF] Tolias | Particular object retrieval with integral max-pooling of cnn activations[END_REF] to get a 4, 096 dimension description vector. Once the image descriptors are obtained, a cosine similarity is calculated between the query images and the images of the database. The search system therefore returns a list of images ordered by similarity. To consider that two images are sufficiently similar, their cosine must exceed 0.9 (determined empirically on the training set).

In the LK-I approach, if no image of the database is found to be similar, the query image receives the unknown class. Due to the small size of the reference base, this case is common. An alternative version of this approach, noted LK-I2, assigns to these uncertain images the maximum prior probability class that is false.

D. Presentation of other teams' runs

For each of the other participating teams, we describe below the data and method used to predict the class of messages.

1) VMU runs: Five methods were tested by the task organizers. These methods are based on two systems, of which they are variants [START_REF] Boididou | The VMU participation @ verifying multimedia use 2016[END_REF].

VMU-F1 and VMU-F2 rely on a first system which is a meta-classifier in which two sets of descriptors are used separately by two classifiers, trained on the training set. Each classifier then predicts true or false for each message, which makes it possible to obtain two predictions per message. Messages from the test set that have received different predictions are further processed by a third classifier (Random Forest) trained on the union of the training set and messages of the test set having received predictions in agreement on both first classifiers. VMU-F1 uses the T and U descriptors for the first two classifiers, while VMU-F2 uses the union of T and FOR for one of the classifiers, and U for the other.

VMU-S1 and VMU-S2 are based on VMU-F1, to which is added a second system which exploits two lists of known sources: the first is a list of sources of trust while the second groups sources of non-confidence. When a sourcebased prediction is not possible, the first system is used to provide a prediction. Finally, VMU-B is a reference obtained by the application of a classifier on the concatenation of T , U and FOR descriptors.

2) MMLAB runs: The proposed approach is based on two random forest classifiers [START_REF] Phan | A hybrid approach for multimedia use verification[END_REF]. The first classifier, called MML-T, takes as input the concatenation of the T and U descriptors proposed by the task organizers.

The second classifier, denoted MML-I, uses the multimedia contents (images and videos) associated with the messages. It takes as input the concatenation of forensics descriptors (the set FOR) and textual descriptors obtained using an external knowledge base. For each event, using the TF-IDF metric on the texts, a list of the most relevant terms related to the event is established from the most relevant web sites returned by an online text search engine. For each image, an inverted search engine (Google image search) is then used and frequency measurements of (i) previously identified relevant terms, (ii) positive and negative polarity terms (derived from the lexicon used in sentiment analysis) are applied to the texts of the most relevant sites found. In the case of a video coming from Youtube, these frequency measurements are applied to the comments of the video. The other videos are not analyzed.

Finally MML-F is the fusion (linear combination) of the scores of each class provided by MML-T and MML-I with respective coefficients of 0.2 and 0.8 in order to favor the second module while ensuring a prediction in the case of prediction failure of the second module (e.g. video not coming from Youtube).

3) MCG-ICT runs: The first approach proposed by [START_REF] Cao | Mcg-ict at mediaeval 2016: Verifying tweets from both text and visual content[END_REF] is based on the text content of messages. The T and U descriptors are used, and a new descriptor is added to this set. The computation of this new descriptor is based on the separation of an event into themes, a theme being defined as the set of messages sharing the same image or video. Each theme is described by the average of the T and U descriptors of its messages, and by statistics such as the number of messages in the theme, the number of distinct messages (i.e. hashtags, to discriminate retweets), the ratios of distinct messages, of messages containing a URL or mention, and of messages containing multiple URLs, mentions, hashtags or question marks. From these characteristics, a theme-level classifier is constructed, and indicates the probability of a message to be true or false. This probability is the new descriptor added to each message. The classifier at the message level, built on the enriched textual descriptors, is called MCG-T.

A second module evaluates the credibility of the visual content. For images, authors use the FOR descriptors (without specifying the classifier used). Videos are treated differently.

Referring to [START_REF] Silverman | Verification Handbook: An Ultimate Guideline on Digital Age Sourcing for Emergency Coverage[END_REF], the authors define four characteristics to describe videos: a measure of the sharpness of the image, the contrast ratio, defined as the ratio of the size of a video over its duration, the duration of the video and the presence of logos. These four characteristics are combined by a binary decision tree. The predictions corresponding to this approach are MCG-I.

Finally, MCG-F combines these two previous predictions.

V. RESULTS OF THE SUBMITTED RUNS A. Experimental setup

The data used to evaluate these systems are those from the test set of the VMU task presented in Sec. III. The evaluation measure used in the task was the F-score on the false class. However, this measure is not discriminating between the predictions true and unknown, and is based on the majority class false, which represents a bias: a system predicting false for every message would get 71.14% as F-score. We thus use instead the micro-F-score and the accuracy which are global measurements on all the classes to be predicted.

In addition, as an image that can be used by several messages, the evaluation is done by cross validation on the events, in order to guarantee that all the messages using the same image are in the same fold so that the assessment is not biased. To implement this cross-validation, all events are randomly subdivided into n packets. The evaluation therefore accounts for the performance that can be expected when processing a new event generating a set of messages that can be true or false. The results of the methods described in Sec. IV, re-evaluated according to the protocol described above (scores and evaluation by sets of messages sharing the same image), are presented in Tab. II.

Between the two evaluation modes (per message, or group of messages sharing the same multimedia content), there are great differences for some methods. In fact, approaches that assign contradictory classes to different messages sharing the same multimedia content are penalized in our second evaluation framework (drop in recall). In contrast, our LK-I2 approach benefits from its default strategy for media content classified as unknown by LK-I. The results of each approach are discussed in the following subsections.

B. Comparison of the different modalities

In addition to the quantitative results provided above, we examine the approaches according to the type of descriptors that they exploit (text, source or image modality) and their possible complementarity for the fusion experiments presented in the following section. We exclude from this study predictions that already involve fusion between modalities. Thus only predictions LK-T, LK-I and LK-S will be kept among our predictions, MML-T, MML-I, MCG-T and MCG-I for the teams MMLAB and MCG-ICT. The predictions of the VMU team are all based on fusion (see Sec. IV-D). However, we retain VMU-S1 which is mainly based on the sources and which achieves the best performances. These eight predictions, hereafter denoted elementary, will be used in the following.

1) Text-based runs: Three predictions can be associated with a textual approach: LK-T, MML-T and MCG-T. The prediction LK-T tends to classify all messages as false, which can be explained by the strong imbalance of classes in the training set (three times more false than true messages). Thus, 636 real messages are classified as false. Conversely, the MML-T and MCG-T predictions tend to wrongly classify false messages as true (i.e. respectively 557 and 457 false messages on 1,230 are predicted true).

We can also note a difference between these three predictions depending on the descriptors used. While the predictions MML-T and MCG-T are based on shallow descriptors (essentially the set of descriptors T ), LK-T uses content descriptors (i.e. precise patterns). These predictions are thus possibly adapted to a fusion in order to merge their capacities.

2) Source-based runs: Two predictions are identified as using sources: LK-S and VMU-S1. While both approaches are based on a list of trusted sources, VMU-S1 also considers a source of non-trust. It can be noted that since the two lists of trusted sources are not identical, they can be complementary. A second difference is the choice of the assigned class in case of absence of source. While VMU-S1 chooses the false class, which is the majority class of the training set, LK-S makes the choice of the unknown class which inevitably gives a misclassified message (since 'no message actually has this class) but which allows a high accuracy of messages classified as true or false (respectively 100.00% and 92.97%) at the expense of the recall (respectively 41.22% and 87.47%).

3) Multimedia-based runss: Multimedia approaches are the most diverse. There are three predictions in which images or videos are used: LK-I, MML-I and MCG-I. Thus, even if the multimedia approaches have the weakest results individually, they may be complementarity.

LK-I predicts the true or false class only for a few messages (170 out of 2, 228), but obtains a high accuracy (97.30% on the class false). Messages for which no similar image was found get the unknown class. All messages with a video as illustration also receive the unknown class. MCG-I is the only approach dealing with video processing while messages with video content represent 48.43% of the dataset.

Several phenomena can explain the poor performance of these systems. First, in the case of a slight difference between the original image (real) and the modified image (false), the images are considered as similar by the search engine. This directly impacts LK-I and MML-I that both search for similar images in databases. Secondly, the images referenced on the specialized sites are sometimes altered: image stamped with texts like 'false', 'rumor' or 'true', or with drawing (e.g. a red circle on the photoshopped area to help the reader find it). The images broadcasted on the social networks also undergo this type of editing which lower the similarity between the query image and the database image.

From these results, it also seems that the use of a similar image search (LK-I and MML-I) provides more information than the use of descriptors FOR (MCG-I).

The poor results of the image-based LK-I2 approach can be explained in part by the small size of the image database. In fact, only about 25% of the images to be classified were present in the database at the time of submission of the results for the challenge. The large number of images for which no decision has been made (class unknown) strongly impacts the results in terms of recall.

VI. FUSION STRATEGIES

A. Simple fusion of the runs

A direct fusion of the predictions of Tab. II is first studied in this section. Each message is described by the predictions true, false or unknown of the different systems, in order to learn a combination of the predictions. The fusions of the predictions are then carried out by four classification methods: 1) linear SVM; 2) decision tree;

3) Random Forest (with 500 trees of depth 2); 4) neural network (a Dropout layer, a dense hidden layer of size 20 and an output layer with sigmoid activation function). In addition to these classifiers, we indicate the results of a reference system corresponding to the majority vote on the predictions of the participants (i.e. among the predictions, the most frequently predicted class is associated with the message).

The evaluation protocol is a kind of one-leave-out but at the event level: Each classifier, is trained on all the messages of all the events except one event whose messages serve as a test set to evaluate the performances. We repeat the process, each time leaving out a different event, and average the results. The results are presented in Tab. III; asterisk denotes statistically significant results (Wilcoxon test with p = 0.05) compared to the reference system (majority vote).

It should be noted that the reference system does not allow to outperform the best predictions for the task, unlike classifiers using all the methods of the participants. This shows that not all predictions have the same importance and classifiers can learn appropriate weights for each method, or even more complex nonlinear combinations. As such, the best classifier (neural network) allows a significant increase in the rate of good classification, while offering more consistency (lower standard deviation of the performance measures).

Some messages are more difficult to classify than others. In Fig. 2, we report the distribution of messages according to the number of methods classifying them correctly. All messages are correctly classified by at least one of the participants' methods. But some messages are incorrectly classified by most of the methods; In particular, there are 263 messages for which most methods are wrong. A simple fusion strategy will then have a great chance to rely on this majority to take its decision, which will lead to a prediction error. These difficultto-categorize messages have one of three characteristics that may explain this difficulty:

1) messages written in non supported languages (information extraction failure) and making similarity calculations inappropriate (too few tweets in this language); 2) reduced URLs that hide the cited source (e.g. using short URL as goo.gl, t.co or bit.ly); 3) a large part of these messages come from events having both true and false messages and are therefore ambiguous (Paris attacks and Fuji Lenticular).

Two examples of such tweets are given in Fig. 3.

To study the contributions to the fusion of each of the methods, we can observe the produced classifiers. In the following, we focus on the Random Forest which gets both Fig. 3. Example of tweets incorrectly classified by more than 12 methods of the VMU partcipants and by the the fusion Fig. 4. Contribution (Gini index) of each system in the Random forest fusion, with the F-score of the system alone good scores and allows to study these contributions easily. The contribution of an attribute (in our case the prediction of a method) is defined as the importance according to the Gini index, also called mean decrease impurity and as defined by [START_REF] Breiman | Classification and Regression Trees, ser. Statistics/Probability Series[END_REF], averaged over all trees in the random forest and normalized between 0 and 100%. We present these contributions in Fig. 4, and we compare them with the performances of the submissions taken independently.

It is worth noting that the best systems are not the ones preferentially used in the fusion. Indeed, VMU-F1, VMU-S1 and VMU-S2 contribute for more than 60% to the fusion, while their scores are among the lowest. These three systems are the most accurate, but have a low recall (many messages classified as unknown), which explains their low overall results. The fusion allows to exploit their very high precision when they predict true or false, and to refer to other systems otherwise.

We have seen that approaches can complement one another to improve prediction scores. However, the proposed fusion uses all the predictions while the information conveyed by each classifier can be redundant (e.g. predictions MCG-T and MCG-I affect the prediction MCG-F). Moreover, we do not obtain any information on the contributions of each modality during the direct fusion. We examine these two points in the following subsections.

B. Fusion elementary predictions

The results of a direct fusion of the eight elementary predictions defined previously (LK-T, LK-I, LK-S, VMU-S1, MML-T, MML-I, MCG-T and MCG-I; see Sec. V-B) are presented in Tab. IV. The reference system is again the majority vote on the eight input predictions. In the case of equality, the unknown class is used.

Despite the withdrawal of half of the input predictions, it is still possible to properly classify 95.0% of images and their associated messages. The fusion thus still brings an absolute gain of 10% compared to the best system (LK-S in this evaluation scenario). It is also interesting to compare these results with those in Tab. III. In particular, better results are obtained with the reference system by retaining only elementary predictions. This is easily explained since majority voting is sensitive to duplicates (and more broadly to correlations) induced by runs already including fusion. Classification methods that are not very sensitive to these correlation phenomena between attributes, such as Random Forest, logically obtain equivalent results. The fusion in this case is based on systems partly different from those seen previously, as can be seen in Fig. 5, but finally offers identical performance.

C. Importance of external knowledge

Some of the eight elementary systems use knowledge external to the training data. On the one hand, (LK-S and VMU-S1) are based on the identification of sources, which rely on white or black lists of sources that have been compiled manually. On the other hand, MML-I and LK-I approaches use external image databases for image search and comparison. It is thus legitimate to question the influence of this external knowledge in the results obtained, in particular because of the great contribution of source-based approaches in the previous experiment. We give in Tab. V the results obtained by the fusion restricted to elementary approaches using no external resource to the training data.

This time, the performance is lower than previous fusion attempts, and even lower than some of the methods taken separately (Tab. II). This last point shows that the four remaining methods predict different classes (this also indicated by the scores of the reference system), and that it is difficult to find a pattern to favor one method over another (scores of fusion by learning methods lower than the majority vote). Finally, the importance of the external resources used in some participants' systems is clear, since their absence results in a 25% drop in the performance of the fusion.

D. Fusion by modality

From the set of eight elementary runs, we propose a twolevel fusion in which the first level classifies the messages according to the three modalities (text, source or image) and the second level regroups these three 1st-level predictions.

Tab. VI first presents the results of the three classifiers of first level (prediction at the level of the text, source or image). A first observation is the encouraging result of the classifier merging the text-based runs. Indeed, the results are significantly higher than those of individual systems. For the sources, the gain of the fusion is also present. Yet, the fusion of image-based approaches tends to produce worse results than the best image run.

To implement second-level fusion, we rely on neural networks, which are simple to implement and perform well in all previous fusion experiments. The architecture of the network reflects our two-level approach: the three neural networks, each corresponding to the fusion of approaches based on the same modality, serve to feed a network of second-level neurons (same architecture as in other experiences). To train this network, we test two approaches (noted training 1 and training 2 hereafter):

1) the text, image and source neural networks are trained individually, and the second-level network is then trained from their outputs; 2) the entire neural network is trained as a whole. We present the results of the second fusion level with the two training strategies in Tab. VII. As can be seen, the results in both cases are very good, but there is an interest in training the entire network all at once rather than by level. The difference is statistically significant (Wilcoxon test with p = 0.05). With the training strategy 1, the results are comparable to those of a direct fusion of all the runs.

VII. CONCLUSION

In this article, we propose and examine several fusion strategies based on the predictions made by the four teams participating in the Verifying Multimedia Use task of the Mediaeval 2016 evaluation campaign. Thus, we have seen that approaches based on the credibility of the source obtain good prediction scores but rely on external resources (white or black lists of sources) whose construction and maintenance may not seem credible in an application. on a very large scale (tweets from different countries, in different languages, for example). Approaches based on image analysis usually have disappointing individual results because of their inability to give a prediction on many cases. On the other hand, combined with other approaches, they may prove to provide complementary information improving the overall performance of a system. More generally, we have found that it is not necessarily the best individual scores approaches that contribute the most to the fusion system. The learning fusion systems we have proposed make it possible to exploit the high accuracy of certain systems while offsetting their weak recall with other methods. Finally, the main result of this article is the interest of proposing systems that combine different approaches with late fusion. The most powerful strategy seems to combine them by level, grouping first the methods working on the same type of information (text, image, source). An implementation of this two-level approach with a neural network gives indeed very good results, significantly better than the other approaches explored in this article.

Many research issues remain open after this work. We are currently developing datasets to compare existing approaches to more numerous and more various cases (tweets, but also articles of blogs or opinion sites and newspapers). These datasets are made available on the site http://hoaxdetector.irisa.fr/. From a technical point of view, future work will aim at correcting some problems, highlighted by our experiments, of systems based on images (e.g. modified images considered as similar to the original real image, images not found). We plan to expand the coverage of the image database. We are also exploring ways to improve the content comparison module to eliminate false positives, and to locate modified areas in these images [START_REF] Maigrot | Tampering detection and localization in images from social networks: A CBIR approach[END_REF]. From an application point of view, the presentation of information to the user must also be studied. It seems inappropriate that a system implements a strict censorship of messages deemed false, but the presentation of doubtful elements raises man-machine interface challenges, but also cognitive challenges (acceptance of the judgment of the machine), especially when the decision results from multiple systems combined by techniques that make it difficult to explain the final decision (especially for neural networks).

Fig. 1 .

 1 Fig. 1. Examples of two tweets from the MediaEval VMU task, on the same event (Hurricane Sandy)

  6 (31, 6) 90, 9 (23, 9) * 84.3 (28, 8) 90, 5(24, 6) * 91, 4(23, 7) * Acc. 84, 0 (28, 3) 95, 1 (11, 7) * 86, 9 (23, 0) * 95, 1(12, 9) * 96, 3(10, 5) *

Fig. 2 .

 2 Fig. 2. Histogram of messages according to the number of runs correctly classifying them

Fig. 5 .

 5 Fig. 5. Contribution (Gini index) of each system in the fusion by random forest, and their respective F-score.

TABLE II PERFORMANCE

 II OF RUNS SUBMITTED TO THE VMU TASK IN TERMS OF ACCURACY (%) AND MICRO-F-SCORE (%) (STD-DEV IN PARENTHESIS)

  TABLE III AVG. F-SCORE AND ACCURACY (%, WITH STD-DEV) OF FUSIONS OF ELEMENTARY RUNS.

TABLE IV PERFORMANCE

 IV (%) OF FUSION ON THE 8 ELEMENTARY RUNS

TABLE VI PERFORMANCE

 VI (%) OF FUSION FOR DIFFERENT LEVELS AND MODALITIES

See http://www.invid-project.eu/

See https://revealproject.eu

See https://www.pheme.eu/

https://en.wikipedia.org/wiki/List of news agencies

https://en.wikipedia.org/wiki/Lists of television channels


Vedran Vukotić Lamark

Vivien Chappelier Lamark

Teddy Furon

Are Deep Neural Networks good for blind image watermarking?

Are Deep Neural Networks good for blind image

watermarking?

Introduction

Deep Learning (DL) has completely revolutionized the field of Computer Vision. It started with image classification [14] and is now spreading to any task of Computer Vision: object recognition [START_REF] Siméoni | Unsupervised object discovery for instance recognition[END_REF], instance search [START_REF] Wu | Deep multiple instance learning for image classification and auto-annotation[END_REF], localization [START_REF] Iscen | Panorama to panorama matching for location recognition[END_REF], similar image retrieval [20,[START_REF] Gordo | Deep image retrieval: Learning global representations for image search[END_REF] . . . The transfert property is underlying this versatility: A DL network trained on some supervised task (e.g. classification) has been shown to perform well on other applications, even non-supervised tasks like similarity search.

A DL network comprises several layers. While the last layer provides a vector of probabilities that the input image belongs to the classes, the internal auxiliary data (the output of the previous layers) also happen to contain relevant information about the input image. Other tasks can indeed tap these data. Many Computer Vision works take a trained network off the shelf, keep the first layers as is, and only re-train the deepest layers for their specific task [20].

Following this trend, state-of-the-art image search algorithms tap the output of an internal layer and use them to derive a global descriptor of the image [20,[START_REF] Gordo | Deep image retrieval: Learning global representations for image search[END_REF]. This way, finding similar images boils down to finding close vectors in an Euclidean high dimensional space, where efficient fast search engines exist. It has been shown that these global descriptors are compact and discriminative while at the same time robust to valuemetric (e.g. JPEG, noise, filtering) and light geometric (e.g. cropping, rotation, scaling) distortions [START_REF] Kanbak | Geometric robustness of deep networks: analysis and improvement[END_REF].

Problem formulation

This paper investigates whether this approach is also suitable for watermarking, with the hope of benefiting from this apparent robustness. This paper only considers zero-bit watermarking. The layers of a DL network play the role of the extraction function yielding a vector to be watermarked. This raises the following challenges:

• How to invert this highly non-linear extraction function? Once the feature vector is watermarked, how to map it back into the image space?

• How to take into account a perceptual model ensuring the quality of the watermarked image?

• How to guarantee a required false positive level as there is no probabilistic modeling of these features?

Prior works

The main trend dealing with DL and watermarking is actually the protection of DL networks. It aims at proving the ownership of a network by embedding a watermark into the learned parameters of the model [START_REF] Nagai | Digital watermarking for deep neural networks[END_REF][START_REF] Darvish Rouhani | Deep-Signs: A Generic Watermarking Framework for IP Protection of Deep Learning Models[END_REF].

The connection between machine learning and image watermarking is surprisingly old. Papers [START_REF] Yu | Digital watermarking based on neural networks for color images[END_REF]13] protect images with a classical watermarking technique and a neural network is only used at the decoding side in place of a maximum likelihood decoder based on a statistical model, which may not be accurate. J. Stolarek and P. Lipiński learn a transformation [START_REF] Stolarek | Improving digital watermarking fidelity using fast neural network for adaptive wavelet synthesis[END_REF] as proposed in this paper. However, 1 this transform is dedicated to one specific host image. Paper [START_REF] Mun | A robust blind watermarking using convolutional neural network[END_REF] has a similar position but fails guaranteeing a probability of false alarm.

We also mention paper [7] which makes a very good comparison between attacks in watermarking and attacks on DL networks (a.k.a. adversarial sample -see Sect. 3.1).

Zero-bit Watermarking

Our scheme belongs to the TEMIT approach 1 : Transform, EMbed, Inverse Transform. We denote by Tr(•) the transformation extracting a feature vector, so-called the host signal x o = Tr(I o ) of dimension n, from an image I o . The embedding modifies it into x m = Emb(x o ). This function implicitly depends on a secret key. For simplicity, the paper focuses on a zero-bit watermarking scheme [START_REF] Merhav | Optimal watermark embedding and detection strategies under limited detection resources[END_REF]8,4]: we hide the presence of a secret mark, not a message. The watermarked image is given by

I m = I o + Tr -1 (x m -x o ),
where Tr -1 is the inverse transform function.

At the detection side, the image under scrutiny is I a with x a = Tr(I a ). It is deemed as a watermarked image if x a belongs to the acceptance region D ⊂ R n . The use of a secret key is implicit in these notations.

The hypercone detector

This paper focuses on one specific zero-bit watermarking scheme: the hypercone detector [START_REF] Merhav | Optimal watermark embedding and detection strategies under limited detection resources[END_REF]. This scheme is provably good [4] and the detector is blind and oblivious to the watermark, host and noise powers.

The acceptance region D is a dual hypercone of axis a ∈ R n ( a = 1) and half angle 0 < θ < π/2 defined as:

D := {x ∈ R n : |x a| > x cos(θ)}. (1)
Vector a plays the role of the secret key. We now define the following function R(•) : R n → R:

R(x) = (x a) 2 ρ(θ) -x 2 , (2)
with ρ(θ) := (tan 2 (θ) + 1). This quantity is negative when x / ∈ D, null when x lies on the boundary of D, and positive inside the dual hypercone. Indeed, when R(x) > 0, this quantity is the amount of power of Gaussian white noise to add onto x in order to push it outside the hypercone with high probability [4].

This gives a rationale for watermarking vectors at the embedding side. We push the host vector x o to a location x m deep inside D s.t. R(x m ) is as big as possible. This provably increases the robustness against a white Gaussian noise addition in the feature space. 1 a wording coined by Ton Kalker.

Linear and invertible extraction function

The hypercone detector is at the core of some image watermarking techniques [8] where the extraction function is (almost) a n × |I| Parseval tight frame T (e.g. selection of some DCT or DWT coefficients). Once the embedding modifies x o in x m , the watermarked image is obtained as

I m = I o + T (x m -x o ).
(3)

A constraint on the image Euclidean distance I m -I o 2 ≤ C (e.g. expressed in terms of MSE or PSNR) is ensured if

x m -x o 2 ≤ C.
The embedding then amounts to solve the following problem:

x m = arg max x: x-xo ≤C R(x). (4)
It is known that x m belongs to the 2D-hyperplane containing the origin, x o , and u [START_REF] Cox | Digital Watermarking. Morgan Kaufmann series in multimedia information and systems[END_REF][START_REF] Merhav | Optimal watermark embedding and detection strategies under limited detection resources[END_REF], so that (4) boils down to a line search over angle β ∈ [0, π/2] defining:

x = x o + C cos(β)u -C sin(β)v, (5)
with (u, v) a basis of this hyperplane:

u := sign(x o a)a, v := x o -(x o u)u x o -(x o u)u . (6)
Vectors x o and a are statistically independent, so that for large n, x o a ≈ 0 which yields the close form solution [4]:

β = arccos 1 -x o 2 C -2 cos 4 θ, (7)
if C ≥ x o cos 2 θ, otherwise x o cannot be watermarked (i.e. pushed into region D) because C is too small. The embedding is thus given by a close form solution or a simple line search over β. This is thanks to an invertible extraction function Tr preserving the Euclidean distance.

Deep learning feature extraction

This section considers a neural network used as image classifier over n c classes. It is denoted by function F that computes a vector from an image: p = F(I) ∈ R nc . This function is decomposed as F = S•N, where S is the softmax function ensuring that p is a probability distribution (i.e. nc k=1 p k = 1, and 0 ≤ p k , ∀k ∈ {1, . . . , n c }). Function N is the network per se, a composition of C convolutionnal layers and F fully connected layers. Each layer is a nonlinear process as it encompasses an activation function such as the well-known ReLU.

Once the parameters of these layers have been learned from a training set, the network predicts classes of images. The output of the softmax is a vector p = F(I) of estimated probabilities p k = P (k|I) that content I belongs to class k. In the end, the class of an image is given by

C(I) = arg max 1≤k≤nc p k .
(8)

Adversarial samples

Adversarial sampling is a recent trend showing that DL classifiers can easily be fooled [17,3]. The attack forges an image I a visually similar to an original image I o but with a different prediction C(I a ) = C(I o ).

In a targeted attack, a specific class is given, say k t , and an original image I o not belonging to this class. The attack finds I a as close as possible to I o s.t. C(I a ) = k t :

I a = arg min I:C(I)=kt Dist(I, I o ), (9)
with Dist a measure of distortion (usually, the L -norm of (I -I o ) with ∈ {0, 1, 2, +∞}). In a white box scenario, the attacker knows the network architecture and parameters. Problem ( 9) is impossible to be solved as the region of images classified as k t is unknown. It is replaced by

I a = arg min J λ (I) (10) J λ (I) := Loss(I, k t ) + λ.Dist(I, I o ). ( 11
)
where λ ∈ R + and Loss is a loss function measuring how far I is from being classified as class k t . For instance, the loss is a divergence (e.g. Cross-Entropy) between the predicted probability distribution F(I) and the targeted p , i.e. p k = δ kt (k), or more simply:

Loss(I, k t ) = | max k =kt (F(I) k ) -F(I) kt | + , (12)
with |a| + = a if a > 0 and 0 otherwise. This way

C(I) = k t ⇔ Loss(I, k t ) = 0. (13)

Practical solutions

The minimization problem [START_REF] Gordo | Deep image retrieval: Learning global representations for image search[END_REF] implicitly uses the domain of images of the same size h × l and same number of channels c as the original image I, say {0, 1, . . . , 255} h×l×c . A relaxation looks for a solution over the continuous set [0, 255] h×l×c . Quantization onto integers is applied on the continuous solution to obtained an image. This box-constrained optimization often uses a differentiable mapper m which is a monotonic bijection from R to [0, 255], e.g. m(x) = 255(tanh(x) + 1)/2 as in [3]. This changes (9) into a unconstrained minimization of J λ (m(X)) over X ∈ R h×l×c .

Practical attacks are mostly based on gradient descent [START_REF] Goodfellow | Explaining and Harnessing Adversarial Examples[END_REF]15]. One can back-propagate the computation of the gradient through the layers of the network. The attack initializes a random starting point X (0) (around m -1 (I)) and iteratively computes

X (i+1) = X (i) -η∇J λ (m(X (i) )). ( 14
)
It stops when the objective function no longer decreases substantially. This gradient descent converges to a local minimum. Several rounds with different initialization are competing. Advanced numerical solvers have been also used in the literature, e.g. ADAM [3]. The main difficulty is to set λ to a well chosen constant. Theoretically, there exists a range of λ values where the solution of (10) coincides with the solution of [START_REF] Goodfellow | Explaining and Harnessing Adversarial Examples[END_REF]. In practice, this range is unknown. Simpler attacks start with a given distortion budget, Dist(I a , I o ) ≤ C, and set λ = 0. The gradient descent starts at the original image, i.e. X (0) = m -1 (I o ) and iterates until either C(m(X (j) )) = k t or Dist(m(X (j) ), I o ) > C. In the latter case, the attack failed finding an adversarial sample within the distortion budget.

Application to zero-bit watermarking

The key idea of this paper is i) to use the network N or part of it (i.e. the first layers) as the extraction function Tr, ii) to modify the extracted vector as explained in Sect. 2, iii) to use an adversarial sample mechanism of Sect. 3 as Tr -1 in order to put back the marked vector into the image. This raises several issues cleared in this section.

Need of a locality transform

Our proposed system can function with any neural architecture. Denote by E the set of the first layers used for extracting an embedding of an image: e = E(I) ∈ R n . Yet, the representational coordinate system of the embedding space does not depict the feasible locations. A neural network never provides a zero-mean isotropic embedding e. The usual culprit being the asymmetrical activation functions, such as ReLU, that have a R ≥0 codomain.

This rises the need for a mapping from the original coordinate system to a local coordinate system. We empirically model a local coordinate system by providing images to the DL network and analyzing their embeddings in their representation space. We use two possible mappings:

• Centering: x = L(e) = e -ē, • PCA: x = L(e) = Σ-1/2 (e -ē),
where ē is the empirical mean and Σ is the empirical covariance matrix. The first option preserves the dimensionality, while the second may operate a reduction by keeping the biggest eigenvalues of Σ. In the end, Tr = L • E.

False alarm probability

The false alarm probability is usually defined as P fa := P(X ∈ D) where the random vector X models the feature vector of an original image. For the hypercone detector, under the weak assumption that X has an isotropic distribution (e.g. a Gaussian white distribution), this probability has a close form expression:

P fa = I cos 2 (θ) ((n -1)/2, 1/2), ( 15
)
where I is the regularized Beta incomplete function. This assumption does not hold for embeddings provided by the network. Even the empirical centering and whitening doesn't guarantee a distribution exactly isotropic.

Instead, we prefer modifying the definition of the probability of false alarm. For a given image under scrutiny, the extracted features are seen as a fixed vector. It is the acceptance region which is random: the axis direction of the hypercone is a random vector A uniformly distributed on the hypersphere. It happens that this probability of false alarm has the exact same expression as (15). For a required false positive level, (15) is inverted so as to find the corresponding half angle value θ.

Another use of this equation is to compute the p-value: for given x and a, quantity (15) is computed for a cosine equalling c = |x a|/ x . It means that if we were drawing O(1/p) random secret keys A, on expectation, one of them would give a normalized correlation bigger or equal to c.

The objective function and imposed constraints

A bad idea is to stick too closely to the watermarking described in Sec. 2.2. It amounts to first fix x m and then to use the back propagation to craft an image whose feature vector is as close as possible to x m .

Watermarking in the feature space and crafting the watermarked image are better done jointly by defining:

I w = arg min Dist(I,Io)≤C Loss(I) (16)
Loss(I) = -R(Tr(I)), (17)
and letting an adversarial sample mechanism solving this optimization problem. The distortion function Dist is for instance the Mean Square Error (expressed in dB as a PSNR). While minimizing the loss and thus iteratively generating the watermarked image, additional constraints can be applied. For instance, the watermark can be attenuated selectively in the spatial domain by a perceptual mask to make it less perceivable. We use the Structural Similarity SSIM heatmap for that purpose. The watermarking is then performed by a simple gradient descent (see algorithm 1).

Experiments

The watermarking system is both architecture and layer agnostic, meaning that any layer can a priori be used for watermarking. This paper considers one specific architecture at different layers, exploring their goodness for watermarking by evaluating robustness and perceptual quality.

Experimental protocol

The experimental part evaluates our proposed system based on the off-the-shelf VGG19 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] experiments on photos from the test set P "professional" provided as part of the CLIC challenge [START_REF]Challenge on learned image compression[END_REF]. This dataset consists of 117 high-resolution professionally taken photos.

Image quality

When the stopping criterion is expressed in terms of PSNR, the quality of the watermarked image is roughly acceptable above 40dB. Fig. 2 illustrates this with the reference Lenna image on its top row. The watermark signal in the image space looks like uniform noise when watermarking lower layers and gets more structured when processing deeper layers, as shown in Fig. 1.

A psycho-visual attenuation as specified in Algorithm 1 improves the quality of the image. Our experiments used the structural similarity (SSIM) heatmap, but any other psycho-visual model (e.g. Butteraugli [2]) can be integrated transparently. Figure 2 shows that the watermark is less perceivable on the right column thanks to attenuation in flat regions (skin, hat, wall) for a given P SN R target. This almost does not hurt robustness R(Tr(I w )).

The downside is the increased number of steps (and thus time) necessary to obtain a watermark of the same robustness. Fig. 3 shows a 3D representation of the watermarking process. The first two axis corresponds to the projection of Tr(I i ) onto the basis (u, v) (6), while the third component accounts for the energy remaining in the complimentary space. The stopping criterion was met in 10 or 13 iterations, respectively without or with perceptual attenuation.

From one layer to another

Table 1 gives the log 10 p-value for the watermarked image and after a rotation of 5 • . The first part of the table illustrates the behaviour when centering is used as a locality transform, thus retaining the original dimensionality. The second part of the table illustrates the behaviour with PCA with a reduction to 256 dimensions.

In general, watermarking at deeper layers offers more robustness. Each convolutional-pooling block adding some

Robustness

We evaluate the robustness of the first fully-connected fc1 layer of VGG19 at the expense of a lower dimension- ality (n = 4096). The stopping criterion is defined as P SN R = 42dB. To define a baseline, we watermark the same images with the linear wavelet transform as described in Sect. 2.2 with the same distortion constraint.

Figure 4 illustrates the robustness against rotations. Wavelets drop to a detection rate of around 25% after a rotation of only 0.5 • and it ceases to detect watermarks after a rotation of 1 • . On contrary, the robustness of our technique smoothly degrades with the strength of the attack.

Figure 5 illustrates the robustness against cropping. The performances again smoothly degrade with the strength of the attack: 50% of watermark detection when cropped to 90% of their original content and continues to detect some watermarks down to a crop of 50%. The system outperforms the classic approach. This is not surprising as DWT was not designed for geometric invariance. Yet, it shows that accurate synchronization is of utmost importance with linear transforms (e.g. DCT, DWT, random projection), whereas it is far less stringent in the new approach.

On top of this, we observe that the system is robust to horizontal flips! Since there are naturally and artificially occurring in the training dataset of ImageNet, the NN seems to have learned a transform invariant to this.

Figure 6 illustrates the robustness against JPEG compression. DWT performs better and we argue that this is caused by the lower dimensionality (n = 4k vs. 1M ). The rule of thumb in watermarking is to spread the watermark to gain robustness against noise addition. This conflicts with the design of a domain invariant to geometric transformation.

Conclusion

This preliminary work shows that watermarking features extracted from a DL network is feasible and relevant. Feasible because the framework of adversarial sample provides a way to create watermarked image of good quality. Relevant because the DL network provides a transformation which strike a good trade-off between invariance to geometric attacks and robustness to valuemetric attacks. However, it also raises some issues. First, the type of architecture of network has to be investigated. The Computer Vision literature keeps on improving the robustness of the image descriptor for image search [START_REF] Tolias | Particular object retrieval with integral max-pooling of cnn activations[END_REF]. Second, for a given architecture of the network, the training dataset plays also a big role. It is not clear at the moment to what extend invariance and robustness improve by learning on a training set dedicated to watermarking, e.g. comprising compressed, cropped, rotated images.

From the application point of view, this study opens the door to a single image descriptor good for both image search and watermarking. In copy detection and copyright infringement applications, image search alone yields many false positives. A watermark detection would drastically decrease the number of false recognition cases.

Figure 1 .

 1 Figure 1. The watermark signal in the image space. Left -block1 pool: last pooling layer of the first convolutional block. Right -fc1: last fully-connected layer.

Figure 2 .

 2 Figure 2. Top: P SN R = 42dB and P f a = 10 -8 without (left) and with (right) structural similarity attenuation. Bottom: 'exaggerated' watermark at P SN R = 25dB without (left) and with (right) structural similarity attenuation.

Figure 3 .

 3 Figure 3. 3D view of the iterations without (green) and with (magenta) the perceptual attenuation. The 3D shape shows the set {x : R(x) = c}. The red line shows the embedding of Sect. 2.2 providing the same robustness. The yellow line is the boundary of the hypercone. The heatmap of R(x) on plane (u, v).

Figure 4 .

 4 Figure 4. Robustness against rotation.

Figure 5 .Figure 6 .

 56 Figure 5. Robustness against cropping.

  convolutional neural network pretrained on ImageNet 2 . We perform all the 2 available at https://keras.io/applications/#vgg19 Data: input image I o , watermarking signal a Result: watermarked image I w

	i ← 0;
	I i ← I o ;
	repeat
	/ * obtain embedding	* /
	x ← Tr(I i );
	Loss(I i ) ← -R(x);
	/ * perform one step update	* /
	g ← ∇Loss(I i );
	I i+1 ← I i -ηg;
	if performing SSIM heatmap attenuation then
	h ssim ← SSIM heatmap (I o , I i+1 );
	/ * attenuate watermark with
	psycho-visual heatmap	* /
	I δ ← I i+1 -I o ;
	I i+1 ← I o + h ssim ⊗ I δ ;
	end
	/ * compute indicator values	* /
	Compute Dist(I o , I i+1 )
	i ← i + 1;
	until stopping criterion met;
	I

w ← I i ; Algorithm 1: Proposed watermarking algorithm.

Table 1 .

 1 Watermarking with different VGG19 layers with the same target P f a = 1e -6 and a stopping criterion of P SN R ≤ 42dB.

	Layer name	Emb. dim.	Average log 10 p-value marked image	Average log 10 p-value rotation 5 •
	Original dimensionality, centering
	block1 pool 802 816	-64.49	-0.68
	block2 pool 401 408	-58.32	-1.13
	block3 pool 200 704	-52.25	-3.80
	block4 pool 100 352	-21.31	-3.81
	block5 pool	25 088	-4.95	-2.24
	fc1	4 096	-5.27	-4.41
	fc2	4 096	-4.11	-1.70
		PCA to 256 dimensions
	block1 pool 802 816	-3.65	-0.93
	block2 pool 401 408	-12.30	-1.36
	block3 pool 200 704	-8.13	-3.20
	block4 pool 100 352	-8.19	-2.42
	block5 pool	25 088	-4.61	-2.41
	fc1	4 096	-5.61	-3.20
	fc2	4 096	-5.22	-1.69
	invariance to rotation and scaling, the best layer is indeed
	the first fully-connected layer. From now on, the remaining
	experiments use this layer.