Jan Butora

Senior Member, IEEE Patrick Bas

Size-Independent Reliable CNN for RJCA Steganalysis

Keywords: Independent Reliable CNN for RJCA Steganalysis Steganalysis, false positive control, JPEG, arbitrary size

come

I. Introduction

A lot of emphasis on privacy and security has been used in today's communication. Private communication tools have been thus increasingly more important in society. Steganography is one of these tools that allows the modification of a given digital image (cover image) to communicate a desired secret message. The only condition is that the modifications are statistically untedetectable [START_REF] Fridrich | Steganography in Digital Media: Principles, Algorithms, and Applications[END_REF]. Two main branches of image steganography have existed since the early beginning of the field: spatial steganography using the pixel representation of the image, and JPEG steganography using the DCT coefficients of the compressed image.

On the other hand, steganalysis aims to detect the mere presence of steganography. Many targeted attacks against early steganography have been proposed [START_REF] Böhme | Breaking Cauchy model-based JPEG steganography with first order statistics[END_REF], [START_REF] Dumitrescu | On steganalysis of random LSB embedding in continuous-tone images[END_REF], [START_REF] Dumitrescu | Detection of LSB steganography via Sample Pairs Analysis[END_REF], [START_REF] Thai | Statistical model of quantized DCT coefficients: Application in the steganalysis of Jsteg algorithm[END_REF], [START_REF] Thai | Optimal detection of OutGuess using an accurate model of DCT coefficients[END_REF]. Unfortunately, with the development of content-adaptive steganography [START_REF] Butora | Side-informed steganography for jpeg images by modeling decompressed images[END_REF], [START_REF] Cogranne | Efficient steganography in jpeg images by minimizing performance of optimal detector[END_REF], [START_REF] Guo | Using statistical image model for JPEG steganography: Uniform embedding revisited[END_REF], [START_REF] Li | A new cost function for spatial image steganography[END_REF], [START_REF] Sedighi | Content-adaptive steganography by minimizing statistical detectability[END_REF], the models targeted in the previous attacks no longer hold and are thus unusable. Researchers have hence started using machine learning classification approaches by collecting steganography-specific features [START_REF] Fridrich | Rich models for steganalysis of digital images[END_REF], [START_REF] Holub | Low-complexity features for JPEG steganalysis using undecimated DCT[END_REF], [START_REF] Kodovský | Steganalysis of JPEG images using rich models[END_REF]. More recently, deep learning (DL) detectors have been shown to be the most accurate detectors available [START_REF] Butora | How to pretrain for steganalysis[END_REF], [START_REF] Cogranne | The ALASKA steganalysis challenge: A first step towards steganalysis "Into the wild[END_REF], [START_REF] Yousfi | Improving EfficientNet for JPEG steganalysis[END_REF]- [START_REF] Yousfi | Imagenet pre-trained cnns for jpeg steganalysis[END_REF]. Despite their superior detection abilities, Figure 1: A diagram of the proposed method for sizeindependent steganalysis. Contrary to classical steganalysis, our system proposes also to control False Positive (FP) rates.

their performance guarantees are only empirical due to their supervised training. To make things worse, training convolutional neural networks (CNNs) on images of variable sizes is practically unfeasible, which forces the steganalyst to train their detector on images of fixed size. This is a major drawback of this DL strategy since in practice, we can encounter images of any possible size. Additionally, it has been shown that the CNNs suffer from the Cover-Source Mismatch (CSM) [START_REF] Ker | A mishmash of methods for mitigating the model mismatch mess[END_REF], [START_REF] Kodovský | Study of cover source mismatch in steganalysis and ways to mitigate its impact[END_REF] more than the feature-based methods [START_REF] Giboulot | Effects and Solutions of Cover-Source Mismatch in Image Steganalysis[END_REF], [START_REF] Zhang | Cover-source mismatch in deep spatial steganalysis[END_REF], which further decreases the quality of the empirical error rates of a given detector. The recent Reverse JPEG Compatibility Attack (RJCA) [START_REF] Butora | Reverse JPEG compatibility attack[END_REF], [START_REF] Butora | Extending the reverse JPEG compatibility attack to double compressed images[END_REF], [START_REF] Cogranne | Selection-channel-aware reverse JPEG compatibility for highly reliable steganalysis of JPEG images[END_REF] is a steganalysis method exploiting the structure of the JPEG images. This attack inspects the rounding errors of the decompressed JPEG files and it has been shown that steganography changes first-order statistics of this signal, which can be very accurately detected with a DL detector. Moreover, such a signal is rather insensitive to image processing operations prior to the JPEG compression, making it very robust against the CSM. On the other hand, we showed in our previous work that the attack could suffer from a JPEG compressor mismatch [START_REF] Butora | Analysis and mitigation of the false alarms of the reverse jpeg compatibility attack[END_REF].

For the steganalyst to decide if the test image is either Cover or Stego, one minimal requirement is to control the False Positive (FP) rate of the classifier1 , that is why metrics such as MD5 (miss detection rate for a FPR of 5%) were also proposed to benchmark steganographic schemes [START_REF] Cogranne | The ALASKA steganalysis challenge: A first step towards steganalysis "Into the wild[END_REF].

For a binary classifier, this means finding a decision threshold corresponding to a desired FP rate. However, with a DL detector, the only way, so far, is to empirically compute the decision threshold from the data. Another approach, without DL, uses Kolmogorov-Smirnov test [START_REF] Lehmann | Testing Statistical Hypotheses, Second Edition[END_REF] on the decompression rounding errors [START_REF] Levecque | Vers une steganalyse certifiée pour des images JPEG[END_REF], [START_REF] Levecque | Toward reliable jpeg steganalysis (at qf100)[END_REF] (see Section I-A2 for more details) but the detection power is in this case greatly reduced.

In this work, we take the RJCA further by studying the CNN detector's logits coming from cover images. This allows us to find an accurate model of the cover logit distribution, which provides us with theoretical thresholds for any desired FP rate. We also show how this distribution changes with image size, giving us the ability to steganalyze images of any size with a prescribed FP rate without the need for retraining a CNN detector trained on images of fixed size. Of course, one can always take crops of bigger images to avoid dealing with different image sizes, but this way the detector does not pool all the available information and, as we will see in Section IV it is possible to design a detector which gets more and more accurate with increasing image size.

A. Prior Art

We recall that the two main goals of this work are [START_REF] Araujo | Computing receptive fields of convolutional neural networks[END_REF] deep learning steganalysis of arbitrarily sized images and (2) control of the detector's FP rate.

Even though both of these topics have been previously studied, none of the proposed approaches is able to achieve both goals at the same time.

1) Images of Arbitrary Size:

While steganalysis of images of any size is straightforward to implement with the feature-based methods (feature vector from any image is of the same dimension), this changed with the development of DL architectures [START_REF] Boroumand | Deep residual network for steganalysis of digital images[END_REF], [START_REF] Mingxing | EfficientNet: Rethinking model scaling for convolutional neural networks[END_REF], [START_REF] Xu | Deep convolutional neural network to detect J-UNIWARD[END_REF], [START_REF] Yedroudj | Yedroudj-net: An efficient CNN for spatial steganalysis[END_REF]. Due to the minibatch-driven supervised training of the CNNs, all images in a given mini-batch must be of the same size. Despite this, training the CNNs with differently sized mini-batches is possible, but in practice would be rather cumbersome. And even then, the mini-batches are limited by GPU memory, meaning we cannot use non-trivial mini-batches of images of large sizes. In the field of computer vision, this is mitigated by simply resizing every image on the input to a desired size [START_REF] Deng | ImageNet: A large-scale hierarchical image database[END_REF]. However, such an approach cannot be used for steganalysis for two obvious reasons:

(1) resizing of the image destroys the signal of interest (the stego signal) and (2) it prevents pooling all the available information [START_REF] Ker | Batch steganography and the threshold game[END_REF].

A different, two-step approach has thus been proposed in [START_REF] Fuji-Tsang | Steganalyzing images of arbitrary size with CNNs[END_REF]. The authors first trained a slightly modified YeNet [START_REF] Ye | Deep learning hierarchical representations for image steganalysis[END_REF] on images of a fixed size (tiles), as typically done in the steganographic community. Moreover, the following change was added to the CNN architecture to better capture the information about the image size. In the majority of cases, the last layers of a CNN detector consist of a Global Average Pooling (GAP) across spatial dimensions, providing a vector of size C,2 followed by a fully-connected layer, resulting in two logits -one for the cover class and one for the stego class. This is then typically followed by a softmax activation. The authors changed this structure and extracted not only the average, but also variance, minimum, and maximum, creating thus a vector of size 4C. After having trained this so-called Tile Detector (TD), features of size 4C were then extracted from every image of interest (with potentially different sizes) and were used to train a two-layer Multi-Layered Perceptron (MLP). Such MLP then served as a steganalysis detector of arbitrarily sized images. A similar approach was later applied to SRNet [START_REF] Boroumand | Deep residual network for steganalysis of digital images[END_REF] in [START_REF] Yousfi | Breaking ALASKA: Color separation for steganalysis in JPEG domain[END_REF] during the first ALASKA competition [START_REF] Cogranne | The ALASKA steganalysis challenge: A first step towards steganalysis "Into the wild[END_REF].

2) Control of the False Positive Rate: In a typical deep steganalysis work, a CNN detector is trained to distinguish between cover and stego classes by utilizing the crossentropy loss function. The detector's performance is then measured as the minimum probability of error under equal priors

P E = min P FA P FA + P MD 2 , ( 1
)
where P FA is the probability of False Alarm (FP rate), and P MD is the probability of Missed Detection -misclassifying stego image as a cover (false negative rate). This value corresponds to a single point on the detector's ROC curve and does not inform us at all about its behavior for different values of P FA , which is impractical in an operational setting, where a steganalyst wants to set its detector's threshold to a very small value, such as 10 -3 or 10 -4 . Of course, one can set up a threshold to achieve such P FA empirically but practically this is unfeasible since we have a limited amount of training samples (usually only tens of thousands of images).

To mitigate this issue, we can see in the literature the use of statistical tests [START_REF] Cogranne | An asymptotically uniformly most powerful test for LSB Matching detection[END_REF] or p-values [START_REF] Ker | Quantitative evaluation of pairs and RS steganalysis[END_REF], [START_REF] Westfeld | Attacks on steganographic systems[END_REF]. In recent works [START_REF] Levecque | Vers une steganalyse certifiée pour des images JPEG[END_REF], [START_REF] Levecque | Toward reliable jpeg steganalysis (at qf100)[END_REF], the decompression rounding errors from cover images are split into 64 lattices (one per each pixel position in an 8 × 8 block) and two-sample Kolmogorov-Smirnov test is performed on each lattice. The p-values of every test are corrected by the Bonferroni procedure [START_REF] Lehmann | Testing Statistical Hypotheses, Second Edition[END_REF] to increase the power of the detector by aggregating different observations. In this context, the desired (even very low) P FA can be theoretically found, however, the stego detection power of such a detector is greatly lacking compared to a DL approach.

Note also that controlling the FPR can also be seen as a calibration problem [START_REF] Gawlikowski | A survey of uncertainty in deep neural networks[END_REF] in machine learning. These methods are for example used in steganalysis to fuse comparable outputs of different classifiers [START_REF] Bernard | Exploiting adversarial embeddings for better steganography[END_REF], [START_REF] Yousfi | Breaking ALASKA: Color separation for steganalysis in JPEG domain[END_REF]. However, calibration techniques are not designed to face very small FPR.

B. Our Contribution

The goal of this work is to develop a universal steganography detector for JPEG images compressed at Quality Factor 100 which can be used for analyzing images of arbitrary size, detecting potentially any steganography, and for which we have perfect control of its false positive rate. The main contributions can be summarized by the following:

• We introduce a Gaussian model of a DL detector's soft outputs (logits), allowing us to control the detector's FP rate. We verify, that the theoretical Gaussian threshold for any given FP rate corresponds to an empirical FP rate given by the data.

C. Organization of the Paper

The rest of the paper is organized as follows: In the next section, we introduce the basic concepts and notations. Section III introduces the dataset and detectors used throughout the paper. In Section IV we describe and evaluate the proposed methodology. We compare our method to previous state-of-the-art in Section V and the paper is then concluded in Section VI.

II. Preliminaries and Notation

In this Section, we introduce several concepts and notations that will be used throughout the paper.

Boldface symbols are reserved for matrices and vectors. Rounding x respectively to the nearest integer and the nearest higher integer will be denoted [x] and x . Denote ⊗ element-wise multiplication and element-wise division. Let DCT(•) denote the 2D type-II Discrete Cosine Transform (DCT) used during the JPEG compression and IDCT(•) its inverse (the 2D type-III DCT).

To compress a grayscale image using the JPEG format, we take each 8 × 8 block of uncompressed pixels x and transform them to DCT coefficients

c = [DCT(x -128) q] , ( 2
)
where q is an 8 × 8 integer-valued quantization matrix defined by the JPEG Quality Factor (QF). To decompress the DCT coefficients back to pixel values, the steps are reversed y = [IDCT(c ⊗ q) + 128] .

(3)

Note that since JPEG is a lossy compression (due to the rounding operation and numerical imprecisions in the DCT), the uncompressed image x and the decompressed image y are not necessarily the same.

A. Reverse JPEG Compatibility Attack

We briefly recall the basic idea of the RJCA [START_REF] Butora | Reverse JPEG compatibility attack[END_REF] as we will use it to build our detector. The main constraint is limiting ourselves only to QF 100. However, as previously noted in [START_REF] Cogranne | The ALASKA steganalysis challenge: A first step towards steganalysis "Into the wild[END_REF], it is also a very popular quality factor since 14% of all images uploaded to Flickr have been compressed with this quality. For lower quality factors, the variance of the signal of interest gets too big and the signal becomes useless. While Eq. ( 3) is what many decompressing libraries would provide a user with, we can avoid the rounding operation, to prevent the extra information loss. We thus decompress the DCT coefficients into non-integer pixel values

y = IDCT(c) + 128, (4)
where we left out the quantization table, as it is composed of ones at the highest QF. It has been shown in the original publication that the decompression rounding errors, denoted by e = yy,

when computed on cover images follow a Wrapped Gaussian distribution. It then follows that steganography necessarily increases the variances of this distribution, which can be very reliably detected even for small embedding payloads. The best performance is obtained with a CNN detector, where the rounding errors e are provided as inputs instead of the decompressed images. For more information about the attack, we refer the reader to [START_REF] Butora | Reverse JPEG compatibility attack[END_REF].

B. Convolutional Neural Networks

To analyze how CNN logits change w.r.t. the image size, it is important to understand some inner mechanisms of such architectures, namely the receptive field and padding which combined lead to feature poisoning. We shall see in Section IV that feature poisoning impacts the mean of the logit distribution and that the receptive field induces a subset of correlated features.

1) Receptive Field: The Receptive Field of a CNN is the region of the input that contributes to the feature after the last convolutional layer. Let k i and s i represent respectively the filter size and stride of the i-th convolution and let L be the number of convolutions in a given CNN architecture. It was shown [START_REF] Araujo | Computing receptive fields of convolutional neural networks[END_REF] that the receptive field size R at the input image can be computed as

R = L l=1 (k l -1) l-1 i=1 s i + 1. ( 6
)
Original image

Input to the global average pooling The stride of the receptive field can be computed as a product of all strides

S = L l=1 s i . ( 7
)
The overlap of the neighboring receptive fields can then be computed as

O = R -S. ( 8
)
Assuming no padding is applied in the CNN, it follows that for an image of size N 1 × N 2 , the feature size after the last convolutional layer (or equivalently before the global average pooling) is of size (N 1 -R + 1)/S × (N 2 -R + 1)/S , see Figure 2 for a graphical representation. As the feature size has to stay positive, this gives a lower bound on the input image size: N 1 , N 2 ≥ R. This constraint can be effectively mitigated by using padding in the network, but as we will see in Section IV, we do not want to use padding for the detector as it changes the distribution of the inputs. In computer vision tasks, changing the distribution with padding is typically not an issue as it can be countered by rather aggressive data augmentations, such as resizing. These augmentations are, however, not applicable in steganalysis because they would destroy the weak stego signal of interest.

In our experiments, we use only the SRNet [START_REF] Boroumand | Deep residual network for steganalysis of digital images[END_REF], for which we found R = 179, S = 16, and O = 163. We want to point out, that without padding, the size of the smallest JPEG image we could analyze with this architecture is 184 × 184 (the sizes need to be multiples of 8). Note also, that the size of the overlap of the neighboring receptive fields in SRNet is 163 × 179. For EfficientNet [START_REF] Mingxing | EfficientNet: Rethinking model scaling for convolutional neural networks[END_REF] (EFN) B0 and B4, we found R = 851, S = 32 and R = 1799, S = 32 respectively, which clearly shows why padding has to be used in these architectures since otherwise, the minimum image size would be too big. 2) Feature Poisoning: In this Section, we explain the need to discard padding with the concept of Feature poisoning. Many3 CNN architectures implicitly use some kind of padding during their convolution operations (e.g. zero padding, mirror padding, etc.) in order to preserve the feature dimensions and to avoid having the minimum image size too big as we have just seen. While in computer vision tasks, this might be justifiable, we are questioning the usefulness of padding in steganalysis. We will see in Section IV-A that padding, especially zero padding, is, in fact, undesirable because convolution even with a small 3 × 3 kernel, creates features with different properties than the rest of the image. We call these features poisoned. Figure 3 demonstrates this phenomenon with three convolutional layers, each having a kernel of size 3 × 3. Note that the stride does not have a serious effect on the feature poisoning because the padding is usually applied only on an outside layer of the image. Before the first layer, there is only the outer padding (depicted in blue) and no poisoned features. The first convolution creates a layer of poisoned features (depicted in red). Consequently, every subsequent convolution will create an extra layer of poisoned features, which is why we can see two layers (two pixels from each boundary) of poisoned features after the second convolutional layer. We will refer to the number of poisoned layers as the poisoned size.

Let us denote the poisoned size on the input to the i-th convolutional layer as P i . Next, it is rather straightforward to realize that the poisoned size dependent on the kernel stride s i-1 , padding size p i-1 (which is typically dependent on the kernel size k i-1 ) and on the previous poisoned size P i-1 . We can then find that

P i+1 = P i + p i s i , ( 9
)
where P 0 = 0 is the initial poisoned size. Unfortunately, modern CNNs, such as EfficientNet [START_REF] Mingxing | EfficientNet: Rethinking model scaling for convolutional neural networks[END_REF] or SRNet, contain many convolutions, each using padding, and many of them having stride equal to 1. In Figure 4, we show for several architectures a relative size of poisoned features in the pre-GAP feature -the last feature before the global average pooling. We can see that for image sizes smaller than the receptive field size R, all the features are poisoned. Moreover, as discussed in the previous section, we can see that the receptive field size R of the EfficientNet is much larger, essentially forcing the use of padding. Note that this larger value of R is due to larger filters (most of the convolution kernels are of size 5 × 5, adding two poisoned layers) and due to subsampling early in the network, while SRNet tends to subsample only deeper in the network.

III. Experimental Setup

Before studying the output of the trained detector, we detail how the detector is trained in this section.

A. Training and testing dataset

The dataset used throughout this work is the ALASKA 2 [START_REF] Cogranne | ALASKA-2: Challenging academic research on steganalysis with realistic images[END_REF] comprised of 80k uncompressed grayscale images of size 2048 × 2048. Using ImageMagick's convert, we first JPEG compress the dataset with QF 100 to create the cover set. We then split the dataset into training, validation, and testing sets of sizes 35k, 5k, and 40k respectively. For the stego images, we embed the cover set by simulating optimally coded ternary Least Significant Bit Matching (LSBM) at payloads 0.3, 0.05, and 0.01 bits per DCT coefficient (bpc). We chose a non-adaptive steganographic scheme, as it was shown that constant embedding costs provide the best security against the RJCA due to the smaller number of embedding changes [START_REF] Butora | Fighting the reverse jpeg compatibility attack: Pick your side[END_REF]. To test the detector on unseen, adaptive steganography, we also embedded the cover sets with UERD [START_REF] Guo | Using statistical image model for JPEG steganography: Uniform embedding revisited[END_REF] and J-UNIWARD [START_REF] Holub | Universal distortion design for steganography in an arbitrary domain[END_REF] at the same payloads. To evaluate our methodology on images of different sizes, we crop the cover set into smaller images of size N × N . We will be using two sets of image size N : S train = {184, 256, 512, 768, 1024, 1280, 1536, 1792, 2048} 4and S test = {1000, 1200, 1400, 1600, 1800, 2000}.

Note that to prevent storing a huge amount of data, we simply center crop the original images at resolution 2048 × 2048 when needed. This applies also to stego images, which should not create discrepancies for the nonadaptive LSBM. For the UERD and J-UNIWARD images, the resulting payload of the cropped images could be potentially different due to its adaptive nature, but we will consider this effect negligible.

B. Detectors

Based on the preliminary analysis in Section II, we choose the SRNet [START_REF] Boroumand | Deep residual network for steganalysis of digital images[END_REF], because of its small receptive field size. We train this detector on cover and LSBM images with 0.3 bpc payload, using only their rounding errors e (see ( 5)) as inputs. For this reason, we will refer to this detector as the eSRNet. Only crops of size 512 × 512 are used during the training of this detector. The detector is trained for 10 epochs with the rest of the hyperparameters as explained in [START_REF] Butora | Analysis and mitigation of the false alarms of the reverse jpeg compatibility attack[END_REF] (see Section 3.2). Additionally, we disable the padding in every convolutional layer, as will be explained in Section IV-A.

1) Variable Size Detector: We train another eSRNet (also without padding) on images of variable size (VS) in S train . Note that such an approach is not unrealistic, we simply need to provide images of the same dimension in every mini-batch. We do this in practice by taking cover and stego images of size 2048 × 2048 and simply centercropping the entire mini-batch into the desired size. Due to increased memory requirements, we used batch norm synchronization over 4 GPUs, each having a mini-batch size of 8. We refer to this detector as VS-eSRNet.

2) Prior Art Detector: To implement the detector from [START_REF] Yousfi | Breaking ALASKA: Color separation for steganalysis in JPEG domain[END_REF], we need to proceed in two steps, as previously explained in Section I-A1. First, an eSRNet tile detector (TD) is trained similarly as described in the previous section on images of size 512 × 512. However, we modify the SRNet structure to produce a feature vector f of size 4 × 512 instead of 512 by extracting after the last convolution the average (GAP), but also the variance, minimum, and maximum over each channel. This vector is then fed to the fully connected layer.

As mentioned in [START_REF] Yousfi | Breaking ALASKA: Color separation for steganalysis in JPEG domain[END_REF], the trained TD is then used as a feature extractor. For image crops of every size in S train , we extract their features f . Note that for every image size, f is of size 4×512. Using the same training, validation, and testing split as in the TD training, we train an MLP with 1 hidden layer of size 8×512 on these features. The batch size of the MLP is set to 100 and the other hyperparameters are kept the same. To evaluate the MLP on images of unseen sizes, that is of those in S test , we use TD to extract their features f and then feed these to the trained MLP.

IV. Studying the Logits

In this Section, we describe the proposed methodology for finding the detector's logit (soft output) distribution and using it for the steganalysis of images of any size. Note that all the detectors used in this section are fixed and trained to detect LSBM with 0.3 bpc as described in Section III-B.

To better understand the logits of our detector, we start visually inspecting its response on cover images of different sizes. For such inspection, we only use the testing images from our database. We can observe two phenomena from Figure 5 (left): with increasing image size, 1) the mean of the distribution increases, and 2) the variance decreases. While the decrease in variance is rather expected with more data, the shift in the mean is quite surprising. During our investigation, we found out that the only thing that could cause the mean shift was the feature poisoning due to padding.

A. Feature Poisoning Effect

We experimented with several different ways of padding in the CNN, such as mirror padding, zero padding, circular padding, and for every one we trained one eSRNet. Through an inspection of its logits across image sizes, we observed that while some methods were causing the meanshift in the logit distributions to be smaller, it was always present.

We thus decided to disable the padding completely in the network. To preserve the flow in the residual layers of the network with skip connections, we crop appropriately the features in the skip connections. One could argue that this leads to information loss, however, we believe that the necessary information would be preserved in the main branch of the layer. The effect of disabling the padding on the logit histogram can be seen in Figure 5 (right). As we can see, for the network trained this way, we cannot see any mean-shift anymore, even at a price of higher variance for the smaller sizes. We explain this fact by reducing the feature size before the GAP. Indeed for an image of size 184 × 184 (256 × 256), the pre-GAP feature size is 12 × 12 (16 × 16) with padding and 1 × 1 (5 × 5) without padding.

While the shift in the mean can be estimated from controlled images, we find this rather impractical. For a given image under investigation, one would have to generate a dataset of images of the same size, which could be computationally very costly, especially for large images with millions of pixels. For the rest of the paper, we will only use the eSRNet trained without padding to evaluate our proposed methodology.

Note that the prior work using the MLP [START_REF] Yousfi | Breaking ALASKA: Color separation for steganalysis in JPEG domain[END_REF] did not consider different padding strategies so we kept the TD's implicit zero padding.

B. Scaling of Variance

Now that we see that the logit distribution mean remains unchanged across different image sizes, it is time to investigate the behavior of its variance. To this end, we state the following theorem.

Theorem 1 (Variance Scaling Law). Assume a fixed JPEG compressor compressing images with QF 100, a convolutional neural network without padding, with receptive field size R and stride S, trained on rounding errors e of cover images and non-adaptive stego images. Let φ be the detector's logit from a cover image with

N = N 1 ×N 2 pixels. Denote N G = n (G) 1 × n (G) 2
the pre-GAP feature size, where n

(G) 1 = (N 1 -R + 1)/S and n (G) 2 = (N 2 -R + 1)/S . Then φ ∼ N (µ, σ 2 N ), where µ ∈ R is a constant and σ 2 N decreases with N . Specifically, if N 1 , N 2 ≥ 2R, then there exist a > 0, b ≥ 0 such that σ 2 N = a N G + b. ( 10
)
Otherwise, for N 1 ≤ N 2 , there exist a, c > 0, b ≥ 0

σ 2 N = a N G + b + cn 1 N G 1 + ψ n 2 + 1 2 -ψ n 1 + 1 2 , ( 11
) where n 1 = min{n (G) 1 , R S }, n 2 = min{n (G)
2 , R S }, and ψ(z) = Γ (z) Γ(z) is the digamma function. Moreover, the absolute term b depends on the overparametrization of the given architecture.

We refer the reader for the proof of Theorem 1 to the Appendix. We want to point out that the value µ is linked to the detectability of a given steganographic scheme, especially the embedded payload, however, a more detailed investigation is beyond the scope of this work. Moreover, the parameters a, b are specific to a given source of images (JPEG compressor) and a CNN architecture. While the assumptions in Theorem 1 are tailored to our specific steganalysis scenario, the results hold for any imaging problem where the two classes carry the same distribution everywhere, independently of the location in the image (see observations 2) and 3) in the proof). Consequently, we find the limiting behavior of the logit variance.

Corollary 2. For N → ∞, there exists b ≥ 0 such that

σ 2 N → b. ( 12
)
To experimentally verify Theorem 1, we computed the logits from all the training sizes S train and computed their respective empirical variances σ2

N . We then found the parameters a 0 , b 0 and a 1 , b 1 , c 1 by minimizing the mean relative absolute difference for the training image sizes

a 0 , b 0 = arg min √ N ∈Strain √ N ≥2R σ2 N -σ 2 N σ2 N , ( 13
)
a 1 , b 1 , c 1 = arg min √ N ∈Strain √ N ≤2R σ2 N -σ 2 N σ2 N . ( 14
)
Since for SRNet 2R = 358, we added images of size 360 × 360 to the training sizes S train as it is the closest size of a valid JPEG image.

Solving equations ( 13),( 14) with our dataset gives a 0 ≈ 36.5, b 0 ≈ 2 × 10 -4 , a 1 ≈ -4, b 1 ≈ -0.26, c ≈ 5.9. We Although, the absolute term b in Eq. ( 12) can be unexpected at first, it is not that surprising. Since the eSRNet has more than 4.5M parameters, it is extremely over-parametrized for the given task. We performed the Principal Component Analysis of the GAP features 5 and found that 98% of information can be contained in only 2 channels out of 512. We conclude that many of the convolutional kernels in different channels (within the same layer) are almost identical. As a result, the features deep in the network are greatly correlated across channels giving rise to the absolute term. See Figure 6 (bottom) for the correlation matrix of the 512-dimensional GAP feature. If the network offered no redundancy, we expect the features to not be correlated across channels, which would result in b = 0. This observation is also supported in the proof of the theorem (see Eq. ( 19)). However, compressing the architecture to avoid redundancy on a given task is out of the scope of this work and we plan to address this phenomenon in the future.

As explained in the Appendix, the terms a arises from the spatial partial correlation among the pre-GAP features (and their variance), which is, in turn, caused by overlapping receptive fields. In general, the closest odd multiple of [R/S] features are correlated in every direction. The partial correlations in a single channel are shown in the correlation matrices in Figure 6 (top). Finally, for images where one of the sizes is smaller than 2R, all the pre-GAP features are horizontally correlated if the image width is smaller than 2R and/or vertically correlated if the image height is smaller than 2R, which leads to the term c.

Based on the above discussion, we can conclude that to reduce the logit variance as much as possible (leading to better class separation), the CNN architecture should 1) have a small ratio R/S to reduce the spatial partial correlations among neighboring pre-GAP features, 2) have a small receptive field size R to prevent spatial correlations among all pre-GAP features for images of size smaller than 2R, and 3) be pruned to avoid any redundancy in order to eliminate the absolute term b. While the pruning can always be performed as a last step of the detector development, the other two points depend on the specific requirements of a steganalyst. For instance, if we only aim to steganalyze images of size at least 512 × 512, it is sufficient to consider architectures with R < 256.

C. Gaussianity of Logits and FPR control

The last step of our logit investigation deals with the gaussianity of the distribution. Indeed, having the mean and variance of the distribution would be rather useless, unless we know which type of distribution we are dealing with since ultimately we aim to use this distribution to prescribe a threshold for a given FP rate. Although there are various ways of verifying gaussianity, we use the following approach. For a given testing image size in S test , we compute the mean µ as the average mean of the logit distributions for image sizes in S train and the logit variance σ 2 N using Eq. ( 10) with a, b from the previous section. For a prescribed (Gaussian) FP rate α, we compute the Gaussian threshold as

T α = µ + σ N Q -1 (α), (15)
where Q -1 is the inverse Q-function.

We then use the same threshold to compute the empirical FP rate α emp from the logits. The eSRNet's comparison of α and α emp is shown in Figure 8. For all the testing sizes in S test , we can see a very clear match between the prescribed Gaussian FP rates and the empirical ones even for the values close to 10 -4 . Not only does this strategy verify the gaussianity of the distribution, but it also immediately demonstrates the adequation between the prescribed FP rates and the practical ones for different image sizes that do not belong to the training set. We cannot experimentally verify the relation for smaller FP rates due to the limited testing dataset.

With an accurate model of the cover class, we can use this detector as a one-class classifier, potentially detecting previously unseen steganography. We verify this statement in the next section by steganalyzing two content-adaptive steganographic algorithms.

V. Results and comparisons

We compare here three methods that perform steganalysis for different image sizes:

• eSRNet, where the training is performed only on 512 × 512 images but the distributions of the logits (see section IV-B) are derived from the different sizes provided in the training set. • VS-eSRNet, where both the training and the estimation of the logit distributions are performed with the variable image sizes provided in the training set. • the MLP (see I-A, [START_REF] Fuji-Tsang | Steganalyzing images of arbitrary size with CNNs[END_REF]), which links the decision threshold to the FPR by resorting to Monte-Carlo methods. Note that one drawback of this methodology is that the prescribed FPR relies on the size of the training set, e.g. for a FPR of 10 -n as a rule of thumb 10 n+1 images are necessary to obtain an accurate estimate of the FPR. To compare these methods, we consider two criteria: the quality of the prescribed FP rate and the probability of error (1) on unseen steganography. First, we focus on the the VS-eSRNet. Figure 9 (left) shows the logits coming from this detector and we can see that even in this case, the mean of the distribution is independent of the image size. However, we see that even in this case, the variance of the logits changes w.r.t. the image size. We thus used the same methodology as for eSRNet, proposed in the previous section. In particular, we estimated the scaling parameters a, b according to equation [START_REF] Cogranne | ALASKA-2: Challenging academic research on steganalysis with realistic images[END_REF]. Furthermore, by comparing with Figure 5, we can notice that the mean is larger than that of eSRNet, but the variance also seems to be bigger. We attribute this to the more diverse training cover set but also the stego set, because we did not adjust the embedding payload to maintain the same detectability according to the Square Root Law [START_REF] Ker | The square root law of steganography[END_REF]. Consequently, we can see in Table I that the probability of error on LSBM images is much smaller on the smaller payload for this detector that has seen a wider variety of LSBM-embedded images during training. However, results in Tables II andIII show that the performance of these two detectors is very comparable on previously unseen, content-adaptive UERD and J-UNIWARD steganographic algorithms. Yet another difference is shown in Figure 8 in which we see that the prescribed FP rates start to deviate for values below 10 -2 , suggesting that the left tail of the distribution is thicker than that of a Gaussian distribution. Overall, the VS-eSRNet only provides better performance on seen steganography, but the prescribed FP rates are not as reliable as those of e-SRNet. Moreover, the computational cost increases rapidly, requiring the use of 4 GPUs during its training.

By inspecting the logits of the MLP detector (see Figure 9 (right)), we see that the distribution changes its mean across image sizes because of the implicit padding in the tile detector, therefore we cannot prescribe a theoretical FP rate since that would require creating a dataset for every image size under investigation in order to estimate the distribution mean. Instead, for a given testing image, we take the logits of training images of the closest size in (2) train = {184, 256, 512, 768, 1024} respectively. Finally, Tables I, II, and III show that the performance of the MLP detector is inferior to the other two proposed methods, by up to 40% in terms of P E for the largest LSBM images at the smallest payload 0.01 bpc. We believe this is due to the MLP overfitting on the larger training payload 0.3 bpc. Consequently, our approach outperforms this state-of-the-art method regarding both, the detection Training false alarm α Note that while the probability of error is generally smaller on UERD images for eSRNet and MLP detectors than on LSBM images, it is the opposite for the VS-eSRNet. We attribute this to the more diverse stego training set of the VS-eSRNet, which causes it to overspecialize on the LSBM images. Furthermore, since the UERD and J-UNIWARD algorithms are content-adaptive, they create more embedding changes than the non-adaptive LSBM, giving more evidence to the detector that the images do not belong to the cover class.

VI. Conclusions

In this work, we introduced, for the first time, the concept of size-independent steganalysis with control over the false positive rate of a deep learning detector based on the RJCA. We achieve this by modeling a deep learning detector's cover class soft outputs (logits) with a Gaussian distribution. In consequence, we can use the detector as a one-class classifier separating cover images from stego images. This detector achieves state-of-the-art steganalysis performance (even on unseen steganography) and FP rate control for images of arbitrary size.

First, we showed that padding inside the convolutional layers of a CNN detector negatively affects the distribution of the input image which causes a mean-shift of the logit distribution. With the padding disabled, we proved a theorem stating that the variance of the logit distribution can be expressed as an affine function of the reciprocal number of pixels. With a detector trained on images of a fixed size, the e-SRNet, we estimated the parameters of this affine relation from several predefined datasets of images of different sizes and were able to theoretically prescribe a decision threshold for a desired false alarm rate for images of any size. We then experimentally verified that the empirical false alarm rate closely matches the prescribed one. Finally, we verified that the classifier generalizes well to detect other unseen steganography with much smaller embedding payloads than those used during the detector training.

By relying solely on a deep learning approach -providing images of different sizes during the training -we have seen that the logits even from this detector, the VS-eSRNet, follow the same trend as if trained on fixed-size images only. Using the proposed methodology for this detector revealed that the performance of the two detectors is rather similar. However training the VS-eSRNet requires more computational resources and the resulting cover logit distributions have a thicker left tail, causing discrepancies for false positive rates below 10 -2 .

While we limited ourselves only to JPEG images compressed with the highest quality, in order to exploit the decompression rounding errors, we believe that the same methodology can be applied to more classical spatial deep learning steganalysis that works on image pixels. This would allow us to steganalyze JPEG images of lower quality and perhaps even spatial domain steganography. While the padding will have a similar, perhaps even smaller, effect on the logit distribution in such a scenario, the image content and content-adaptive steganography will have a much stronger impact on the distribution.

Let further G ∈ R be the vector of GAP features and w ∈ R C×1 be the (cover) weight vector from the fullyconnected layer. We can express the cover logit as

φ = G • w = C c=1 w c G c = 1 N G C c=1 w c N G i=1 X c,i
Let us now make three key observations: 1) Convolutional neural networks are shift equivariant [START_REF] Cohen | Group equivariant convolutional networks[END_REF], [START_REF] Mcgreivy | Convolutional layers are not translation equivariant[END_REF],

2) For a given JPEG compressor, the image content under scrutiny (the rounding errors e ij ) follows the same distribution independently of the position in the image (with some exceptions in constant blocks [START_REF] Butora | Reverse JPEG compatibility attack[END_REF]),

3) Non-adaptive steganography changes the stego distribution in the same way, independently of the position in the image. We want to point out that the non-adaptivness of steganography is only required during the training of the detector.

These three observations allow us to make the following conclusions. For any channel c ∈ {1, . . . , C}, all the cover pre-GAP features have the same mean and variance

Var (X c,i ) = σ 2 c , (16)
E [X c,i ] = E [X c ] , ∀i = 1, . . . , N G . ( 17
)
In the following, we will be dropping the channel index for better readability. We note that due to the overlap of the receptive fields in the network, each row (column) of the covariance matrix of [X 1 , . . . , X N G ] contains n R,S positive correlations and N G -n R,S features that are uncorrelated, where n R,S is the number of overlapping receptive fields on a single pixel: n R,S = n 1 × n 2 . 6 The above formula dictates that if the image is too small, all the features will be correlated. In other words, if n 1 , n 2 ≤ R S , which is equivalent to N 1 , N 2 ≤ 2R -1, we obtain n R,S = N G . We now consider different situations, depending on the values n 1 , n 2 .

First, let n 1 ≤ n 2 ≤ R S . We will model the covariance between neighboring features as Cov (X i , X j ) = b d 1 (X i , X j ) , where d 1 (X i , X j ) is the 1 distance between the spatial position of X i and X j because the overlap between neighboring receptive fields decreases linearly in both directions. As such, summing the covariances of correlated features can be expressed as summing the radii of 4r points on concentric circles with radii r ∈ {1, . . . , n1 2 }. Additionally, we have to consider the points on 'circles' having one of the axes longer than the other, if n 1 < n 2 . To sum a single (any) row of the covariance matrix, we compute j Cov(X i , X j ) = σ 2 + b where ψ(z) ∼ ln(z) -1 2z is the digamma function. After some simplifications, we can write j Cov(X i , X j ) =

a + bn 1 1 + ψ n 2 + 1 2 -ψ n 1 + 1 2 . ( 18
)
In the case of n 1 ≤ R S ≤ n 2 , we simply substitute n 2 with R S in Eq. ( 18). Let us assume for the rest of the proof that n 1 , n 2 ≥ R S (N 1 , N 2 ≥ 2R). Note that this is equivalent to substituting both n 1 and n 2 with R S in Eq. ( 18). The rest of the proof for the other cases considered above would proceed in the same manner.

Since n R,S = R = a 0 N G .

It follows that

Var

(G) = 1 N 2 G Var i X i = a 0 N G .
The variance of the logit is

σ 2 N = Var C c=1 w c G c = C c,d=1 w c w d Cov (G c, G d ) = w 2 a 0 N G + C c=1 d =c w c w d Cov (G c , G d ) = a N G + b, ( 19
)
where the constant b ∈ R depends on the correlations across channels caused by over-parametrization of the architecture on a given task. Note that we were able to simplify the expression since Cov(G c , G d ) does not depend on N G for c = d.

It remains to prove that the mean of the logit does not change across different image sizes. We now express the mean as

µ = 1 N G C c=1 w c N G i=1 E [X c,i ] ,
and remind the reader that the mean in a channel is the same for every spatial position (see Eq. ( 17)). It follows that

µ = C c=1 w c E [X c ] ,
which is independent of the image size.

Figure 2 :

 2 Figure 2: Illustration of overlapping neighboring receptive fields in a convolutional neural network.

Figure 3 :

 3 Figure 3: Propagation of the feature poisoning through successive layers due to padding.

Figure 4 :

 4 Figure 4: Relative size of poisoned features for images of size N × N for different architectures with receptive field size R.

Figure 5 :

 5 Figure 5: Effect of zero-padding on histogram of eSRNet's logits for cover images of sizes N × N .

Figure 6 :

 6 Figure 6: Top: Correlation matrix of pre-GAP features across 1000 images from a single channel. Images of size 256 × 256 (left) and 512 × 512 (right). Bottom: Correlation matrix between GAP features across 1000 images of size 512 × 512. The values are cropped for better visualization.

Figure 7 :

 7 Figure 7: Logit variance as a function of the number of pixels N .

Figure 8 :

 8 Figure 8: Comparisons between prescribed FPR α and practical FPR α emp for the different schemes and images of sizes N × N (N is listed in the legend). The MLP's α emp is shown as a function of the empirical FP rate on its training images of the closest size in S train .

Figure 9 :

 9 Figure 9: Histograms of the VS-eSRNet's and MLP's logits for cover images of size N × N .

Figure 10 :

 10 Figure 10: The MLP's α emp as a function of the empirical FP rate α on insufficient training sizes. Left: testing images of size 1000 × 1000 while training image sizes are S (1) train , right: testing images of size 2000 × 2000 while training image sizes are S (2) train . In both cases, training images of the closest size were used to prescribe a decision threshold for a desired α, i.e. 1280 × 1280 and 1024 × 1024.

S 2 ,

 2 the partial correlations are constant w.r.t. the image size, and thus we can write Var iX i = i,j Cov (X i , X j ) = N G σ 2 + b (n R,S -1)

Table I :

 I P E for LSBM images of different size N × N .

	bpc	Method	N 1000 1200 1400 1600 1800 2000
		eSRNet	0.17	0.12	0.08	0.05	0.03	0.02
	0.01	VS-eSRNet 0.10 0.06 0.04 0.02 0.02 0.01
		MLP [48]	0.43	0.43	0.42	0.41	0.41	0.41
		eSRNet	0.00 0.00 0.00 0.00 0.00 0.00
	0.05	VS-eSRNet 0.00 0.00 0.00 0.00 0.00 0.00
		MLP [48]	0.08	0.06	0.04	0.03	0.03	0.02

Table II :

 II P E for unseen UERD images of different sizes N × N (bold/minimum scores take into account the value before rounding).

	bpc	Method	N 1000 1200 1400 1600 1800 2000
		eSRNet	0.15	0.11	0.07	0.04 0.02 0.01
	0.01	VS-eSRNet 0.14 0.10 0.06 0.04 0.02	0.01
		MLP [48]	0.29	0.25	0.22	0.19	0.17	0.15
		eSRNet	0.01 0.00 0.00 0.00 0.00 0.00
	0.05	VS-eSRNet 0.01 0.01	0.00	0.00 0.00 0.00
		MLP [48]	0.02	0.01	0.00	0.00	0.00	0.00
	S train . With these training logits, we then empirically find
	a threshold that corresponds to a prescribed FP rate and
	use it to compute the empirical FP rate α emp on the testing
	images. The comparison in Figure 8 indicates that this
	works quite well, however, there is one major drawback.
	For images of size bigger than 2048 × 2048, we would
	have to use the largest images in the training database to
	prescribe a decision threshold, which could lead to severe
	underperformance. A similar problem would arise if we
	were to test an image of a size smaller than in the training
	data. To demonstrate this issue, we show in Figure 10 α emp
	for images of size 1000 and 2000 if the training data was
	not optimal, e.g. S train = {1280, 1536, 1792, 2048}, and (1)
	S

Table III :

 III P E for unseen J-UNIWARD images of different sizes N × N (bold/minimum scores take into account the value before rounding).

	bpc	Method	N 1000 1200 1400 1600 1800 2000
		eSRNet		0.08	0.04	0.02	0.01	0.00	0.00
	0.01	VS-eSRNet 0.05 0.02 0.01 0.00 0.00 0.00
		MLP [48]		0.40	0.39	0.39	0.38	0.38	0.37
		eSRNet	0.00 0.00 0.00 0.00 0.00 0.00
	0.05	VS-eSRNet 0.00 0.00 0.00 0.00 0.00 0.00
		MLP [48]		0.03	0.02	0.01	0.01	0.01	0.01
	αemp
	10 -5	10 -4	10 -3	10 -2	10 -1	10 0

Misclassifying cover image as a stego image.

C denotes the number of channels after the last convolutional layer in a given architecture.

We are not aware of an architecture that would implicitly not use padding.

The smallest value corresponds to the smallest JPEG image size bigger than SRNet's receptive field size.

The feature vector resulting from the Global Average Pooling.

In our experiments with SRNet and images of size N × N , N > 2R = 358, we observed that n R,S = 11 × 11, which corresponds to the value [R/S] 2 .

Acknowledgement

This work was granted access to the HPC resources of IDRIS under the allocation 2023-AD011012855 made by GENCI. This work received funding from the European Union'Horizon 2020 research and innovation program under grant agreement No 101021687 (project "UN-COVER").

Appendix Proof of Theorem 1

Let φ be the logits of the cover class coming from a CNN with receptive field R and stride S and without any padding. Let X c,i be the vectorized pre-GAP feature entering the Global Average Pooling (GAP) at spatial position i ∈ {1, . . . , N G } in a channel c ∈ {1, . . . , C}, where N G is the number of features before GAP and C is the number of filters (channels) in the last convolutional layer (C = 512 is for SRNet, C = 1280 for EfficientNet-B0). Let us recall from Section II-B1 that


Benjamin Labbé

Romain Hérault

Clément Chatelain

Learning Deep Neural Networks for High Dimensional Output Problems

State-of-the-art pattern recognition methods have difficulty dealing with problems where the dimension of the output space is large. In this article, we propose a new framework based on deep architectures (e.g. Deep Neural Networks) in order to deal with this issue. Deep architectures have proven to be efficient for high dimensional input problems such as image classification, due to their ability to embed the input space. The main contribution of this article is the extension of the embedding procedure to both the input and output spaces in order to easily handle high dimensional output problems. Using this extension, interoutput dependencies can be modelled efficiently. This provides an interesting alternative to probabilistic models such as HMM and CRF. Preliminary experiments on toy datasets and USPS character reconstruction show promising results.

Introduction

High dimension output tasks is a challenging problem in both pattern recognition and machine learning communities. It is concerned with problems where the target y is a vector in R n and n 1 (instead of a scalar in R for a standard regression problem). For instance, in an image segmentation problem, y ∈ [1. . .255] n×m , where n × m is the size of the image. When dealing with multiple outputs, most of the time, strong relations between these outputs can be observed. For example, the label of a pixel strongly depends on the labels of its neighbors. In order to exploit these relations, a global decision overall outputs should be taken instead of multiple local decisions.

In the literature, numerous approaches have been successfully applied on real-world sequence signals, such as handwriting recognition [START_REF] El-Yacoubi | A statistical approach for phrase location and recognition within a text line: An application to street name recognition[END_REF], voice recognition [START_REF] Rabiner | A tutorial on hidden markov models and selected apllications in speech recognition[END_REF], or information extraction in text [START_REF] Bikel | An algorithm that learns whats in a name[END_REF]. These approaches are mainly based on probabilistic Hidden Markov Models, where relations between the outputs are learned. One may note that in the HMM framework, the observations are sup-posed to be independent, which is wrong in almost all applications. More recently, the Conditional Random Fields (CRF), derived the HMM to a discriminative model. CRF have been proposed to overcome this problem by modeling the conditional probability p(x|y) instead of the joint probability p(x, y).

Although efficient on sequences, only few methods, usually based on an extension of HMM or CRF, have been applied on two-dimensional signals. HMM can be adapted to 2D signals through either pseudo-2D HMM or Markov Random Field (MRF). [START_REF] Nicolas | A markovian approach for handwritten document segmentation[END_REF] provides an example of MRF for document image segmentation. CRF have also been generalized for 2D-signals with application to diagram interpretation [START_REF] Szummer | Contextual recognition of handdrawn diagrams with conditional random fields[END_REF]. To the best of our knowledge, there is no approach for signal dimension higher than 3 [START_REF] Tsechpenakis | Coupling CRFs and deformable models for 3D medical image segmentation[END_REF]. It is the limitation of these kind of approach. The reason why few probabilistic methods are applied on 2D-signals is mainly due to their decoding complexity. Even if some sub-optimal decoding methods have been proposed such as HCF or ICM, this is still an open issue. Nowadays, probabilistic methods are the state-of-the-art learning methods for tasks such as signal and image segmentation. Let us also mention the kernel dependency estimation [START_REF] Weston | Kernel dependency estimation[END_REF] and structured output SVM approaches [START_REF] Blaschko | Learning to localize objects with structured output regression[END_REF]. These approaches are based on a kernel joint projection which evaluate the co-occurrence probability of an observation x and a complex label y. Although these approaches handle complex output spaces, they suffer from the pre-image problem. They can easily give an appropriate image in a kernelized label space for a given x, but the label itself is hard to predict explicitly.

In this paper, we assume that other machine learning methods such as neural network are able to handle high dimension problems, whatever the dimension of the signal, and whatever the topology of the input or the output. Deep Neural Network (DNN) with pre-training [START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF][START_REF] Bengio | Greedy layer-wise training of deep networks[END_REF] or regularized learning process [START_REF] Weston | Deep learning via semi-supervised embedding[END_REF] have achieved good performance on difficult high dimensional input tasks where Multi Layer Perceptron (MLP) seems to failed before. These methods try to find a better representation of the input space in an unsupervised manner to ease the supervised task.

High dimensional output space may be verbose and is likely to contain a lot of internal dependencies, especially if the task is not well understood or badly formulated. Starting from this observation, we propose to retain the DNN input pre-training (or regularization) idea and adapt it to the output space. This should provide a better target representation, leading to an easier learning process.

The remaining of this paper is as follows. In section 2, we describe the main principles of neural networks and the existing optimisation procedure for deep architectures. In section 3, we explain our double embedding proposition. Preliminary experiments on toy datasets are shown in section 4, as well as texture segmentation tasks and handwritten caracter reconstruction. Finally, the section 5 concludes.

Deep Networks

In the past decades, MLP have been studied to approximate non-linear functions in many decision tasks. The standard MLP relies on the original perceptron principle and its optimisation is based on the backpropagation algorithm. MLP are usually composed of two layers which perform the following operation:

h l (x) = f l ( W l . h l-1 (x)+b l ) ∀l ∈ {1, . . ., N } , (1)
where W l is a linear transformation, b l is a vector of offsets, h 0 (x) = x, h N (x) = ŷ, and N is the number of layer in the complete network. f l is a non-linear activation function. Deep architectures with more than two layers, suit for high dimensional problems, they are called Deep Neural Networks (DNN). Two major propositions rose from the machine learning community to solve the problem of DNN. The first one stacks pre-trained auto-encoders, while the second one optimizes a regularized criterion. These two network building procedures learn embedding functions of the inputs in an unsupervised way in order to regularize the parameters of the network. An input embedding projects the input data in a better vector space where the neighborhoods are preserved. These unsupervised strategies are expected to be efficient as soon as an estimation of p(x) is useful to learn the final supervised task: estimating p(y|x).

h 1 = f 1 (W 1 .x + b 1 ) ŷ = f 2 (W 2 .h + b 2 ) x h 1 ŷ h l = f l (W l .h l-1 + b l ) x h l-1 ∈ R d h l ∈ R e

Deep layers pre-training and stacking

As deepest layers are hard to learn with a single backpropagation algorithm, [START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF][START_REF] Bengio | Greedy layer-wise training of deep networks[END_REF] propose to pre-train the parameters of these layers on an unsupervised learning with auto-encoder networks. The first layers of auto-encoder are stacked iteratively to build the complete DNN. Finally a supervised fine tuning optimizes all the parameters in the whole stacked network for the final task.

The auto-encoder architecture links a list of visible units (v, the data), a list of hidden units h and a list of estimated visible units (v, the data reconstruction), as in Fig. [START_REF] Bikel | An algorithm that learns whats in a name[END_REF]. By doing so, the auto-encoder finds a reduced representation of the data in the hidden units. The auto-encoders could be either a RBM [START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF] or a derived form of the MLP network that is an Auto-Associator neural network (AA) [START_REF] Bengio | Greedy layer-wise training of deep networks[END_REF].

Learning an Auto-Associator is similar to any MLP learning, using backpropagation. When using AA with C se criterion and no non-linear function, we obtain a PCA [START_REF] Diamantaras | Principal component neural networks Theory and applications[END_REF]. However, the data compression induced by the C se criterion might not be appropriate, when learning a binomial distribution. [START_REF] Larochelle | Exploring strategies for training deep neural networks[END_REF] propose to use the Kullback-Leibler divergence, also known as the cross entropy C ce , as a reconstruction criterion,

C ce (v, v) = - k (v log(v) + (1 -v)log(1 -v)) . (2)
The unknown and evaluated distributions v and v characterize respectively v and v, then the Kullback divergence KL(v, v) will be minimized when v = v. This means when the reconstruction error of v against v is minimum. To avoid trivial projection such as the identity function when the number of hidden unit is higher than the number of visible unit, [START_REF] Larochelle | Exploring strategies for training deep neural networks[END_REF] proposed to constrain the linear transformation matrices of the two functions f (Ux + c) and g(Vh + d) by making U = V T . Whatever the auto-encoder is, we have to split it to keep aside the embedding function f (Ux + c), and drop the re-projecting function g(Vh + d). The embedding function is used to project all the input data into the hidden space, hence the hidden units are used as inputs for a new auto-encoder training in a new layer.

During the training of a new projecting function at layer l, every parameters of the deeper layers are fixed, only the weights matrix U l and c l are modified by the backpropagation. We can add more and more layers, like in Fig. (2, right), depending on the complexity of the input space. Finally, an output layer is plugged on the top of the last layer. At that time, a full backpropagation algorithm is used to fine-tune all the parameters. Now, to maximise the capacities of the network, all the deep parameters are released so the backpropagation optimisation can reach them.

h 1 = f 1 (U 1 .x + c 1 ) x = g 1 (V 1 .h 1 + d 1 ) v = x h = h 1 v = x h 2 = f 2 (U 2 .h 1 + c 2 ) ĥ1 = g 2 (V 2 .h 2 + d 2 ) x v = h 1 h = h 2 v = ĥ1

Regularized semi-supervised deep learning

The second proposition made by [START_REF] Weston | Deep learning via semi-supervised embedding[END_REF] learns deep neural networks via a double optimisation formulation. It consists in learning the two probability distributions p(x) and p(y|x) simultaneously. Their approach optimizes a supervised criterion L 1 and the relevance of an embedding function f (.) on unsupervised input data via a criterion L 2 . The embedding is evaluated on its ability to reproduce into the projected space a similarity matrix D between the samples from the input space.

Let us define two sets of data: L, the Labeled observations, and U, the Unlabeled observations. Let I L be the set of indexes of all elements belonging to the set L, respectively I U the set of indeces for the elements of U. The multi-objective criterion is

J = i ∈ I L L 1 (f (x i ), y i ) + λ (i,j) ∈ (I L ∪I U ) 2 L 2 (f k (x i ), f k (x j ), D k (i, j)) . ( 3
)
where λ is a balancing parameter to tune, and k is the layer at which the capacity of the embedding is evaluated.

At each iteration of the batch algorithm, they compute the stochastic gradient of this multi-objective function, until they reach a minima. This nice formulation of the embedding optimisation with an hinge loss as L 1 and a margin loss as L 2 (see [START_REF] Weston | Deep learning via semi-supervised embedding[END_REF]) provides competitive results to the state of the art methods as Deep Belief Network and DNN, on the MNIST dataset.

Embedding the Output Space

For high dimensional output tasks, we propose here to apply a similar embedding on the output space too. Some decision tasks, such as 2D-signal regression, or image segmentation, occurs in an high dimensional output space. This output space might be verbose and is likely to contain internal dependencies, especially if the task is badly formulated. Embedding on the output space might remove these dependencies and provide a better representation of the output. We propose here to learn embeddings for both the input data and the output labels. This allows us to optimize the decision function in a much smaller space which is adapted to the hidden complexity of the output. That is, an estimation of p(x) and p(y) will provide a good start to estimate p(y|x). We will describe both the stacked pre-trained encoders strategy and the multi-objective optimisation strategy. Nevertheless, described experiments are implemented through the stacking strategy.

Building the stacked network We learn two embedding in an self supervised manner, one out of the input space, and a second one out of the output space. The input reduction is learned as in section 2.1, the AA are stacked iteratively. The projecting activation function f l , the weights matrices U l and the offset b l are kept aside to initialize the W matrices of the final network (Fig. 2). The output reduction is done symetrically by an AA. Here, the output AA are identified with negative notations -l. The re-projecting function g -l , the weights matrices V -l and the offset d -l are stored, and are stacked from the output units (Fig. 3). To sum up, the following inititialization of the complete network is done using :

h -1 = f -1 (U -1 .y + c -1 ) ŷ = g -1 (V -1 .h -1 + d -1 ) v = y h = h -1 v = ŷ h -2 = f -2 (U -2 .h -1 + c -2 ) ĥ-1 = g -2 (V -2 .h -2 + d -2 ) v = h -1 h = h -2 v = ĥ-1 ŷ
W l ← U l ; b l ← c l ; l ∈ [1, T -1] W N -l ← V -l ; b N -l ← d -l ; l ∈ [0, N -T -1] , ( 4
)
where T is the layer where the projected input space is linked to the projected output space, and N is the total number of layer. For the final supervised tuning, the W T matrix is randomly initialized, and all the W l =T matrix are released to permit backpropagation optimisation on the entire network .

Note that the regularised semi-supervised deep learning [START_REF] Weston | Deep learning via semi-supervised embedding[END_REF] can be extended through the addition of a third objective loss L 3 in equation 3 which evaluates the capacity of the output embedding.

Experiments

To demonstrate the validity of our proposition, we have run 4 types of experiments on 3 different datasets. We will first describe each dataset, then each type of experiment and finally we present the results of our experimentations. We generated three kinds of toy datasets for three different tasks: de-noising, simple phase recognition and texture recognition. All problems take as input an image and target the label of each pixel.

Task 1 Each example consists in a noisy image of a rectangle as an input, and the clean image of the rectangle as the target (64 × 64).

Task 2 Each example consists in a representation of the function sin(i/8 + φ) + cos(j/8 + φ), as an input, where i and j are respectively the line number and the column number in the image, φ = 0 on all the image except in a rectangle where φ = π/2; the target is a white over black image of the rectangle (64 × 64).

Task 3 Each example consists in, as image input, a composition of two textures taken from the Brodatz texture archive 1 ; the background is taken from Texture 64, a rectangle is drawn upon it with Texture 65. The target is a white over black image of the rectangle (128 × 128).

In all the datasets, input values are in between 1 and -1 and exactly 1 or -1 for the target. For each task, a train set of 1000 examples and a validation set of 100 examples have been generated. The first example of each validation set are shown on Table [START_REF] Bengio | Greedy layer-wise training of deep networks[END_REF]. A black and white rectangle image is really verbose and a lot of internal dependency occurs. Therefore a better representation of the output space should be available and pre-training on output should improve the results of the supervised task.

Experimental setup.

All the experiments have been run thanks to the Torch5 library [START_REF] Collobert | [END_REF]. We have built a DNN with two hidden layers. Each hidden layer consists in twice as much units as the size of one dimension of the input (64 × 2 pixels for the first dataset). Four experiments have been tested, each differs in the way the network is pre-trained:

Exp. 2

The first hidden layer is pre-trained by using an auto-associator on the input data. Standard supervised backpropagation is then used.

Exp. 3

The first and the second hidden layer are pretrained by using, respectively, an auto-associator on the input data and an auto-associator on the output data. Standard supervised backpropagation is then used.

Exp. [START_REF] Collobert | [END_REF] The first and second hidden layers are pre-trained by using respectively an auto-associator on the input data and on the output data. The supervised learning is decomposed into two steps: on the first step, the link between the first and the second hidden layer is learned by backpropagation using the hidden state of the second auto-associator as target. On the second step, a standard supervised backpropagation is then performed on the whole DNN.

The pre-training for each auto-associator is done by batch gradient stopped at 50 iterations. The first step of the two supervised learning steps is also done by batch gradient with an early stopping at 50 iterations. Batch gradient with early stopping at 300 iterations is performed for the final supervised learning in all case. The learning rate for all the experiments is the same and has been fixed according to the first experiment (No pre-training). The four experiments have been undertaken on the 3 tasks. 3. Output image evolution for the first validation example according to the number of iterations.

Results and discussions

Table [START_REF] Bikel | An algorithm that learns whats in a name[END_REF] shows that, the validation error of the DNN is not improved significantly by input pre-training (Exp. 2), in comparison to standard learning (Exp. 1). However, it shows that with output pre-training (Exp. 3 and 4), DNN achieves better performance. Table (3) clearly shows that the output is already gathered at iteration 10 in a rectangle on Exp. 3 and 4, this is due to the fact that the DNN already knows about internal dependencies on the target thanks to pre-training. On Exp. 1 and 2, output looks more like a cloud at iteration 10. We need to wait until iteration 200 to start to see a rectangle.

According to the MSE at iteration 300, the 2-step supervised training in Exp. 4 does not seem to improve significantly the results in comparison to 1-step supervised training in Exp. 3. Nevertheless, the drop of the validation error is faster. Actually most supervised learning has been done during the first step. One may argue that pre-training is time consuming, so we may compare the result for fewer iterations on Exp. 3 and 4 than for Exp. 1 and 2; but even with less iterations, output pre-training does achieve better performances.

Comparison to KDE, the image reconstruction problem

After these experimentations based on toy datasets, we have compared our proposition to Kernel Dependency Estimation [START_REF] Weston | Kernel dependency estimation[END_REF] on the image reconstruction problem. This relies on a kernel based loss.

A kernel function, K(., .) : R d×2 → R, can be used as a loss function that qualifies the efficiency of a supervised task. Effectively, a distance in an induced Hilbert space can be formulated thanks to its associated kernel K(., .), L(y, ŷ) = K(y, y) -2 * K(y, ŷ) + K(ŷ, ŷ) .

(5) For example, the RBF-loss, L RBF , based on the RBF kernel, is expressed as

Original Image Input

L RBF (y, ŷ) = 2 -2 * exp -||y -ŷ|| 2 2σ 2 . ( 6
)
Highlighting this properties, [START_REF] Weston | Kernel dependency estimation[END_REF] proposed to build two kernels, one for the input space and one for the output space. The authors then learn the dependency between these two kernels to achieve a supervised task. This is similar to our methodology, first of all, learn the unconditional probabilities of inputs and targets, p(x) and p(y) then try to learn the conditional probability of the target p(y|x). We reproduced the same image reconstruction task on USPS dataset presented in [START_REF] Weston | Kernel dependency estimation[END_REF]. Each character image of 16×16 pixels is split into two smaller images, one corresponding to the top 8 lines and the other one corresponding to the bottom 8 lines. 6) with σ = 0.35, qualifies all the applied methods, KDE, k-NN, Hopfield-net and DNN. The results of the 3 first methods are reported from [START_REF] Weston | Kernel dependency estimation[END_REF].

The DNN architecture has been fixed to 2 hidden spaces, h 1 and h -1 , that is 3 computational layers. Both hidden spaces belongs to R 16 . The learning rate and the number of iterations have been fixed according to the results of backprogation without pre-training. On pre-trained DNN, the first computational layer, from x to h 1 , is initialized thanks to an Auto-Associator on the input space (top 8 lines); the last computational layer, from h -1 to y, is initialized thanks to an Auto-Associator on the output space (bottom 8 lines). is greater when using DNN. Indeed, neural networks are prone to instability induced by random initialization. Pretrained DNN leads to less deviation. Nevertheless their performance are similar to the standard backpropagation without pre-training. This observation can be explained by the fact that DNN without pre-training already performs well on this task and may have reached the limit of the given architecture. To emphasize the benefit of the pre-training, not only on the stability of the performance but also on the performance itself, we reproduced the experience but this time on a 10-fold scheme (100 examples in training and 900 examples in testing), making the task harder to achieve. Here, pre-training improves the RBF-loss L RBF and its standard deviation compared to standard DNN.

Conclusion

In this study, a new approach for high dimensional problems such as image segmentation has been presented. The main contribution is to embed the inputs and the outputs of a given problem through a deep architecture in order to benefit from their internal dependencies.

A standard DNN can achieve good performances but seems to slowly converge. It may also fall into unwanted local minima. Input pre-training, by making a better representation of the input space, ease the supervised learning [START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF][START_REF] Weston | Deep learning via semi-supervised embedding[END_REF]. Extending this idea, our proposition is based on output pre-training in order to also model output relations. This method achieves promising results, not only on toy-data image segmentation, but also on real image reconstruction task.

When compared with probabilistic methods (2D-HMM, 2D-CRF), the decision process of the proposed approach is really fast, but is limited to fixed input and output size problems. For instance, probabilistic models better suit variablelength signals such as human voice, handwriting, etc. Our future work will try to overcome this issue without falling into the pre-image curse.

ŷFigure 1 .

 1 Figure 1. (left) a shallow MLP network, (right) a Deep Neural Network. For the layer l, d is the number of hidden units in h l-1 and e in h l ; hence, W l ∈ R d×e , b l ∈ R e .

Figure 2 .

 2 Figure 2. (left) An Auto-Associator learned on the input x. (right) an Auto-Associator learned on the hidden units h 1 .

Figure 3 .

 3 Figure 3. (left) An auto-Associator learned on the output y (right) An auto-Associator learned on the h -1 units.

Figure 4 .

 4 Figure 4. Decomposition of a USPS character in input image and target image.

  Fig. (4) shows this decomposition. Knowing the top lines, the inferning algorithm has to reconstruct the bottom lines. The first thousand examples of the database are taken into account in a 5-fold cross validation scheme. In an unusual manner, each training set consists in one fold (200 examples) and the corresponding testing set consists in the remaining 4 folds (800 examples). The RBF-loss, L RBF , as defined in equation (

Table 1 . Image input and image target for the first validation example of the 3 tasks.

 1

	Task	1	2	3
	Input
	Target
	Task Exp. 1 Exp. 2 Exp. 3 Exp. 4
	1	0.154	0.153	0.068 0.065
	2	0.133	0.132	0.051 0.050
	3	0.167	0.167 0.045 0.045

1 http://www.ux.uis.no/ ˜tranden/brodatz.html

Table 2 . MSE at iteration 300 of the final su- pervised training on the validation set for the 3 tasks, for the 4 experiments.

 2

	Exp. 1 No pre-training, supervised learning by back-
	propagation is directly done.

Table ( 4

 ( ) displays original and reconstruction characters produced by a DNN learned with pre-training.The results presented in table[START_REF] Diamantaras | Principal component neural networks Theory and applications[END_REF] clearly shows the efficiency of Deep Architecture on this particular image reconstruction tasks. On the one hand, for the original experimentation scheme, DNN outperforms KDE, k-NN and Hopfield net. On the other hand, the L RBF standard deviation

	Character	Ex. 1	Ex. 30 Ex. 95 Ex. 101
	Original
	Reconstruction

Table 4 . Characters reconstruction by DNN.

 4

	Methods	# folds Pre-train	L RBF
	Hopfield net *	5	-	1.2190 ± 0.0072
	k-NN *	5	-	0.8960 ± 0.0052
	KDE *	5	-	0.8384 ± 0.0077
	DNN	5	N	0.8000 ± 0.0167
	DNN	5	Y	0.8070 ± 0.0154
	DNN	10	N	0.9489 ± 0.0258
	DNN	10	Y	0.9338 ± 0.0225

Table 5 . RBF-loss of DNN, with or without pre- training, compared to KDE, k-NN and Hopfield Net (

 5

* results reported from

[START_REF] Weston | Kernel dependency estimation[END_REF]

).