Jérôme Pasquet
email: jerome.pasquet@lirmm.fr

Sandra Bringay
email: sandra.bringay@lirmm.fr

Marc Chaumont
email: marc.chaumont@lirmm.fr

STEGANALYSIS WITH COVER-SOURCE MISMATCH AND A SMALL LEARNING DATABASE

Keywords: Steganalysis, Cover-Source Mismatch, Ensemble Classifiers with Post-Selection of Features, Ensemble Average Perceptron, Clustering

Many different hypotheses may be chosen for modeling a steganography/steganalysis problem. In this paper, we look closer into the case in which Eve, the steganalyst, has partial or erroneous knowledge of the cover distribution. More precisely we suppose that Eve knows the algorithms and the payload size that has been used by Alice, the steganographer, but she ignores the images distribution. In this source-cover mismatch scenario, we demonstrate that an Ensemble Classifier with Features Selection (EC-FS) allows the steganalyst to obtain the best state-of-the-art performances, while requiring 100 times smaller training database compared to the previous state-of-the art approach. Moreover, we propose the islet approach in order to increase the classification performances.

INTRODUCTION

During the BOSS1 competition [START_REF] Bas | Break Our Steganographic System': The Ins and Outs of Organizing BOSS[END_REF], the effects of coversource mismatch were clearly observed.

A set of cover and stego image couples (18 000 images) were given to the steganalysts. The images were uncompressed 512×512 grey-level images from 7 different cameras. All the steganalysts were then given a test set of 1000 images used by the organizers to evaluate the detection capability of each competing steganalyst. Some of the images of the test set were from a camera that was not used in the learning set. Thus, the steganalysts encountered inconsistency between the image model, learned during the learning step, and the image model of the test set. This inconsistency is called the coversource mismatch [START_REF] Fridrich | Breaking HUGO -The Process Discovery[END_REF].

The cover-source mismatch phenomenon was initially reported in [START_REF] Cancelli | A comparative study of +/-1 steganalyzers[END_REF], but the only solution to manage image diversity was proposed in 2012 by Lubenko and Ker [START_REF] Lubenko | Steganalysis with mismatched covers: do simple classifiers help?[END_REF][START_REF] Lubenko | Going from small to large data in steganalysis[END_REF]. Their hypothesis is that in order to have a sufficiently descriptive model of the image, one should work on very a huge variety of images. Andrew Ker says "Google knows the model since it owns all the images". Their approach was then to use millions of images during the learning step. Evidently that kind of approach may be very time consuming. Thus, they decided to choose a steganalyzer named Ensemble Average Perceptron (EAP) [START_REF] Lubenko | Steganalysis with mismatched covers: do simple classifiers help?[END_REF] that has linear complexity. They downloaded millions of JPEG images, with a quality factor of 85, from a social network, then they embedded a message with nsF5 at 0.05 bits per non-zero DCT.

In scenarios with the largest diversity (i.e. with many different image sources), they showed that the EAP [START_REF] Lubenko | Steganalysis with mismatched covers: do simple classifiers help?[END_REF] gave the best testing accuracy (85.1%). The Ensemble Classifier (EC) [START_REF] Kodovský | Ensemble classifiers for steganalysis of digital media[END_REF] testing accuracy was 83.6%, and the Support Vector Machine (SVM) [START_REF] Chang | LIBSVM: A library for support vector machines[END_REF] testing accuracy was 80.9%. For complexity reasons, the size of the training set was not equal for each classfier: 6 000 images for SVM, 20 000 images for EC, and 1 000 000 images for EAP. In those conditions, the experimental comparison seemed to show that the EAP was better than the EC when there was cover-source mismatch [START_REF] Lubenko | Steganalysis with mismatched covers: do simple classifiers help?[END_REF].

In this paper, we refute the hypothesis that millions of images are necessary, and we show experimentally that EC with post-features selection (EC-FS) [START_REF] Chaumont | Steganalysis by ensemble classifiers with boosting by regression, and postselection of features[END_REF] allows us to obtain better results with 100 times smaller training database. Moreover, we also introduce an additional pre-processing that really overcomes the problem of cover-source mismatch. This pre-processing consists in organizing images in clusters, and associating a steganalyzer with each cluster, which thus reduces the diversity inside each cluster. We call this proposition the islet approach.

In section 2, we recall the principle of the EC-FS [START_REF] Chaumont | Steganalysis by ensemble classifiers with boosting by regression, and postselection of features[END_REF] and EAP [START_REF] Lubenko | Steganalysis with mismatched covers: do simple classifiers help?[END_REF] classifiers, that were used to carry out the experiments when there is a huge variety of images. The experimental results validate our intuition and show that the selection of features is a good tool in order to deal with cover-source mismatch phenomenon. We consider this as a contribution for the domain since it gives clues to manage the cover-source mismatch phenomenon.

In section 3, we present the islet concept. This is our second contribution, and this confirms something that has already been observed : the increase of the similarity between the learning set and the testing set implies an increase of the steganalysis performances [START_REF] Lubenko | Steganalysis with mismatched covers: do simple classifiers help?[END_REF].

In section 4, we give the experimental results and analyze them.

EC-FS AND EAP CLASSIFIERS

The classifier learning phase is performed on a database of size N cover and stego images. This database is represented by a set of couples (feature vector, class number). We note the set B = {(x i , y i )} i=N i=1 , with x i ∈ R d being a vector of dimension d characterizing the i th image, and y i ∈ {0, 1} the associated class number (-1 for a cover image and +1 for a stego image).

In the next two subsections, we present the Ensemble Classifier with Post-Selection of Features [START_REF] Chaumont | Steganalysis by ensemble classifiers with boosting by regression, and postselection of features[END_REF], denoted EC-FS, and the Ensemble Average Perceptron [START_REF] Lubenko | Steganalysis with mismatched covers: do simple classifiers help?[END_REF], denoted EAP.

Ensemble Classifier with Post Selection of Features (EC-FS)

The Ensemble Classifier with Post-Selection of Features (EC-FS) was presented at IEEE ICIP'2012 [START_REF] Chaumont | Steganalysis by ensemble classifiers with boosting by regression, and postselection of features[END_REF]. It is an extension of the EC [START_REF] Kodovský | Ensemble classifiers for steganalysis of digital media[END_REF].

The EC [START_REF] Kodovský | Ensemble classifiers for steganalysis of digital media[END_REF] is made of a set of L weak classifiers. During the learning step, each weak classifier learns separately on the same image database. A weak classifier, denoted h l with l ∈ {1, .., L}, takes the same x ∈ R d feature vector as input and returns a class number (-1 for a cover image, and +1 for a stego image):

h l : R d → {-1, +1} (1) x → h l (x)
Each weak classifier performs its learning in a space of d red dimension, with d red d. In practice, each weak classifier pseudo-randomly selects features from the feature vector of dimension d. The merging of the votes of the weak classifiers is then obtained by a majority vote, such that, for a x ∈ R d feature vector, we have:

C(x) = 0 if l=L l=1 h l (x) ≤ 0, 1 otherwise. ( 2
)
The EC with Post Selection of Features (EC-FS) compared to EC, reduces d red dimensions and also removes features that could disrupt classification process.

The idea of EC-FS is to improve the performance of each weak classifier through the selection of features. In addition, the selection of features adds an additional variability to the EC algorithm because each weak classifier selects a different number of features (and not always d red features). This additional variability enhances the classification model and leads to improved performance. To keep the complexity equivalent to the EC's one, we apply a selection process after the learning step and we do not re-run any learning step. Thus, once a weak classifier learned, it will seek to take away some features in order to reduce its probability of error.

Each weak classifier performs its learning in a space of d red dimension, with d red d. After the learning phase of a weak classifier, only a subset of the features set is kept. Five low complexity metrics evaluating the importance of a feature have been proposed in [START_REF] Chaumont | Steganalysis by ensemble classifiers with boosting by regression, and postselection of features[END_REF], and this leads to an order for the selection of features leading to the smallest probability of error.

In the article [START_REF] Chaumont | Steganalysis by ensemble classifiers with boosting by regression, and postselection of features[END_REF], we report an average gain on the recall of 1.7% in a clairvoyant scenario [START_REF] Pevný | Detecting messages of unknown length[END_REF] (cover distribution, stego ditribution, and the payload size of the message are known) without cover-source mismatch, using Boss-Base v1.00 (http://www.agents.cz/boss/BOSSFinal/), and the HUGO algorithm [START_REF] Pevný | Using High-Dimensional Image Models to Perform Highly Undetectable Steganography[END_REF] at 0.4 bpp for the embedding. Because of the difficulty of scrounging percentages during the BOSS competition [START_REF] Fridrich | Breaking HUGO -The Process Discovery[END_REF], this gain is significant. We thus decided to test how the post-selection of features may help in the case of a high diversity of images, and thus some source-cover mismatch phenomena.

Ensemble Average Perceptron (EAP)

The Ensemble Average Perceptron (EAP) is a classifier built with a set of L weak classifiers [START_REF] Lubenko | Steganalysis with mismatched covers: do simple classifiers help?[END_REF]. The EAP is constructed exactly like the Ensemble Classifier [START_REF] Kodovský | Ensemble classifiers for steganalysis of digital media[END_REF] explained in section 2.1 (see Equ. 1 and Equ. 2).

Each weak classifier is an average perceptron [START_REF] Lubenko | Steganalysis with mismatched covers: do simple classifiers help?[END_REF], h l with l ∈ {1, .., L}, defined such as:

h l : R d → {-1, +1} (3)
x → h l (x) = sign(w avg .x) with x ∈ R d being a feature vector, sign the function returning -1 or +1 depending on the sign of the input value, and w avg a vector defining the separating plane of the two classes (-1 for cover, and +1 for stego):

w avg = w sum N (4)
with N ∈ N being the number of images used for the learning step, and w sum the sum of the successive weight vectors w (i) :

w sum = N i=1 w (i) (5)
During the learning phase, for an incoming feature vector x i with a class number y i ∈ {-1, +1}, the weight vector w (i) is updated such that:

w (i) = w (i-1) if y i = sign(w avg .x i ) w (i-1) + y i .x i if y i = sign(w avg .x i ) (6)

ISLET PARTITIONING APPROACH

In a real world scenario, the image cover model is not known by the steganalyst. The diversity of the images is due to the lossy or lossless formats, the compression rates, the way images are generated (synthetics, scan, digital, numerical photos), the type of scene, luminosity, focus, etc. A classifier will manage this diversity more easily, and thus the cover-source mismatch, if we restrict its learning and classification to a set of "homogeneous" images. By "homogeneous" we mean images that have close feature vectors.

Our proposition is thus to apply a pre-processing to the image database in order to partition it into a few clusters. Then, we associate a classifier, i.e. EC-FS or EAP, to each cluster, which will learn and classify only vectors that belong to the cluster. We named this technique the islet approach.

The learning step consists of two stages. The first stage consists in running a k-means algorithm on a subset of the entire training database. The k-means algorithm is achieved on feature vectors representing the images. We are obtaining a set of K means vectors noted {µ k } k=K k=1 . The second step consists in creating K classifiers EC-FS or EAP. Conceptually, a classifier is associated to a mean vector. There is thus one classifier per cluster. In this second step, we rescan the entire training database, and for each feature vector x i , we select the closest cluster, i.e. we select the cluster numbered k, owning the smallest L2 distance between x i and µ k, with k∈{1,...,K} , and then we pass this x i feature vector to the k th classifier so that it can learned.

The classification step involves one stage. Given a feature vector x i to be classified, we first select the closest cluster, i.e. we select the cluster numbered k, owning the smallest L2 distance between x i and µ k, with k∈{1,...,K} . Second, we pass this x i feature vector to the k th classifier, for its classification into -1 for cover, and +1 for stego.

Given an input image, the classifier of which learning has been achieved on images with "similar" feature values will be less sensitive to the cover-source mismatch problem.

RESULTS

The database is obtained by downloading 1 million color images from the TwitPic website 2 mostly in jpeg format but also in uncompressed format. Then, images are decompressed, transformed in grey-levels images, cropped to 450×450, and a spatial embedding with the HUGO [START_REF] Pevný | Using High-Dimensional Image Models to Perform Highly Undetectable Steganography[END_REF] algorithm at 0.35 bits 2 http://twitpic.com per pixel is achieved. This leads to a database of 2 million images. Various payload are embedded leading to a database of 3.8 million pairs of images.

For the EAP learning, once the learning is completed, we re-run the learning step, which "virtually" leads to a learning on 7 million pairs. For each experiment, three simulations are conducted, and the database images are considered in a different order. The probability of error,

P E = P F A +P M D

2

, is obtained on a database of 40 000 images that do not belong to the learning database 3 . Let us note that the test database has been downloaded from TwitPic at a different date of the training database. This implies that the training set is made of images of which sources are different from those of the learning data-base. The cover-source mismatch phenomenon is thus present in our experiment. Finally, the average probability of error, P E , is computed by averaging the probability of error of the three simulations. In our experiment we report the prediction rate = 1 -P E .

On this large database, we evaluated the EC [START_REF] Kodovský | Ensemble classifiers for steganalysis of digital media[END_REF], EC-FS [START_REF] Chaumont | Steganalysis by ensemble classifiers with boosting by regression, and postselection of features[END_REF], and EAP [START_REF] Lubenko | Steganalysis with mismatched covers: do simple classifiers help?[END_REF]. We used personal C++ implementations, and we set parameters L and d red at the same values as [START_REF] Lubenko | Steganalysis with mismatched covers: do simple classifiers help?[END_REF][START_REF] Lubenko | Going from small to large data in steganalysis[END_REF], i.e. L = 100 and d red = 2000.

The feature vectors are rich models vectors for spatial images, of dimension 34 671, and described in [START_REF] Fridrich | Rich models for steganalysis of digital images[END_REF]. Those vectors were extracted with a personal C++ implementation. All those vectors were normalized using the maximum norm which showed an increase in the classification performance of more than 8% compared to the variance norm.

All the experiments were carried out at the Center of High Performance Computing HPC@LR4 which provided access to parallel programming, and 24 GB of RAM per node. The HPC@LR requires C++ implementations.

Comparison between EC, EC-FS and EAP

The EC and EC-FS computational complexity is O(d 2 red .N.L) and it is higher than the EAP which is O(d red .N.L). The square factor in the complexity of EC and EC-FS leads to such a huge computational time that it is impossible to train and learn on the entire database (d red has been fixed to 2000). We then tested EC and EC-FS with 20 000, 30 000, 50 000, 100 000 and 150 000 learning images (see Table 1). Moreover, the learning step for EAP was achieved on the entire database (2 × 3.8 ≈ 7 millions of images).

Table 1 and Figure 1 reveal that the EAP converge around 93%. This value exceeded the maximum value of 83% of the training size 20 000 30 000 50 000 100 000 150 000 PR * EC 81% 83% 84% 82% 50% EC. Moreover, the EC performance fell when the number of learning samples exceeded 50 000. This counter-performance may be explained by the too strong heterogeneity and thus a strong cover-source mismatch phenomenon. A high value for d red implies a counter-performance. Indeed, a linear classification is no more efficient when d red = 2000 and when the features are extremely diverse and probably noisy. Our intuition is that lots of features should not be grouped together, or should be selected for producing a weak classifier insensitive to the cover-source mismatch. This also explains why the EC-FS approach is more efficient.

The behavior of the EC-FS was completely different. We did not observe any performance collapse. Moreover, the EC-FS was more efficient than EC and EAP. Indeed, EC-FS obtained a performance of 95%, which was 2.3% higher than EAP (93%). Moreover, EC-FS converged using only 50 000 to 100 000 learning images. EC-FS required 100 times fewer images (50 000 images) than EAP (5 000 000 images) to give better results.

This result is very promising and encourages future works on feature selection, feature reduction, feature combination, etc, in the clairvoyant scenario or in a cover-source mismatch scenario. Our intuition is that the features selection acts as a denoising of the features space, or said differently the features selection creates an invariant space which is invariant to the cover-source variations, but sensitive to a message embedding. In that sense, the EC-FS approach shares some similarity to what has been proposed in [START_REF] Pevný | The challenges of rich features in universal steganalysis[END_REF], with a totally different technique.

In addition, experiments revealed that a correct cover model may be extracted without a million images. The approach is thus a more practical solution than those of Lubenko and Ker that require more than a million images [START_REF] Lubenko | Steganalysis with mismatched covers: do simple classifiers help?[END_REF][START_REF] Lubenko | Going from small to large data in steganalysis[END_REF].

In this section, we compare two very efficient classifiers for the cover-source mismatch scenario. EC-FS gave better results than EAP with 100 times fewer images. In the next section, we evaluate islet partitioning to increase the classification performances and overcome the source-cover mismatch phenomenon.

Islets experiments

Islet parameters

In the islet approach, we have to decide on the number K of islet (K vectors {µ k } k=K k=1 , and K classifiers). Increasing the number of clusters is very beneficial to overcome the cover source mismatch problem. Indeed, the higher the islet K number, the closer the image to be classified will be from a center µ k , and the more the associated classifier will be adapted to this image since, during its learning, this classifier would have learned with "similar" images. However, increasing K will necessitate a larger database since, on average, the number of learning images in each islet is N/K with N being the number images in the entire database and K the number of clusters. In order to converge, the cluster may require a sufficient number of images (50 000 to 150 000 for EC-FS, and 10 6 to 10.10 6 for EAP). This multiplicative factor (K) on the database size necessitates using specific architectures for experiments, such as the High Performance Computing Center of the Languedoc Roussillon.

Islets with EC-FS

In this section, we combine islets with EC-FS. The purpose is to obtain better results for the prediction rate than the 95% obtained at the convergence with around 50 000 images. Let us note that, practically, EC-FS is a good candidate for the islet approach since it converges rapidly. In the experiments, a k-mean was achieved on 80 000 images. Table 2 gives the prediction rate as a function of the number K of islets for 150 000 learning images. The performance has maximum for 6 islets with a prediction of 96%. When the number of islets is low, the number of images per islet is high, but this is useless because the EC-FS converges at around 50 000 images. Conversely, when the number of islets is too high, there are not enough training images, the classifiers have not yet converged, and thus the performance drops.

K islets Training size per islet

Partitioning of the learning database via the islets creates areas which are more homogeneous. On these areas, the EC-FS converge quickly and when the tested image is almost similar to the learning images, the performance of EC-FS becomes better.

With those preliminary results, we show that the gain obtained by islets is 0.67% which is not negligible and very promising. Indeed, islets further increase the performance of EC-FS.

In conclusion, EC-FS gives better results than EAP (gain of 2.3% in the prediction rate), requires 100 times fewer images and when it is combined with the islet approach, it increases the gain by about 0.7% in order to obtain a final prediction rate of 96%.

CONCLUSION

In this article, we first show that EC-FS is a very efficient tool for managing very heterogeneous data. It has good performance to limit the cover-source mismatch problem and requires a learning set 100 times smaller than EAP. Indeed, with 150 000 images, the method is 2.3% more efficient than EAP, which requieres more than one million images. Secondly, by combining EC-FS with the islet approach, we obtain a total gain of 3% compared to EAP approach. The selection of features and the partitioning are thus very promising techniques to overcome the problem of heterogeneous data, also known as the cover-source mismatch problem.

Fig. 1 .

 1 Fig. 1. Results of EC, EC-FS, and EAP, with a logarithmic scale on abscissa.

Table 1 .

 1 Comparison between EC and EC-FS.

	PR * EC-FS	92%	94%	95%	95%	95%

* PR = prediction rate

Table 2 .

 2 Results of islets with EC-FS.

BOSS (Break Our Steganography System) was the first steganalysis challenge. The challenge started on the 9th of September

and ended on the 10th of January

The goal of the player was to figure out which images contained a hidden message and which images did not. http://www.agents.cz/boss/BOSSFinal/. The steganographic algorithm was HUGO[START_REF] Pevný | Using High-Dimensional Image Models to Perform Highly Undetectable Steganography[END_REF].

There is no any automatic adjustment for EC and EAP. This adjustment is useless in the cover-source mismatch scenario for EC[START_REF] Lubenko | Steganalysis with mismatched covers: do simple classifiers help?[END_REF], and not adapted to the online behavior of EAP[START_REF] Lubenko | Steganalysis with mismatched covers: do simple classifiers help?[END_REF]. Nevertheless, in EC-FS[START_REF] Chaumont | Steganalysis by ensemble classifiers with boosting by regression, and postselection of features[END_REF], the features are selected in order to minimize the probability.

HPC@LR (High Performance Computing at Languedoc Roussillon -France) gives access to a 15-Teraflop hybrid hardware platforms consisting of 84 nodes of Intel Hexacore Dual-processors, 2 SPM servers with 80 cores, CPU/GPU Servers Nodes, 4 Cell Dual-processor Nodes, 1 Power 7 Node, Infiniband QDR Network, and Disk Storage.