Raphaël M J I Larsen

Marc-Oliver Pahl

Gouenou Coatrieux

Authenticating IDS Autoencoders Using Multipath Neural Networks

An Intrusion Detection System (IDS) is a core element for securing critical systems. An IDS can use signatures of known attacks, or an anomaly detection model for detecting unknown attacks. Attacking an IDS is often the entry point of an attack against a critical system. Consequently, the security of IDSs themselves is imperative. To secure model-based IDSs, we propose a method to authenticate the anomaly detection model. The anomaly detection model is an autoencoder for which we only have access to input-output pairs. Inputs consist of time windows of values from sensors and actuators of an Industrial Control System. Our method is based on a multipath Neural Network (NN) classifier, a newly proposed deep learning technique. The idea is to characterize errors of an IDS's autoencoder by using a multipath NN's confidence measure c. We use the Wilcoxon-Mann-Whitney (WMW) test to detect a change in the distribution of the summary variable c, indicating that the autoencoder is not working properly. We compare our method to two baselines. They consist in using other summary variables for the WMW test. We assess the performance of these three methods using simulated data. Among others, our analysis shows that: 1) both baselines are oblivious to some autoencoder spoofing attacks while 2) the WMW test on a multipath NN's confidence measure enables detecting eventually any autoencoder spoofing attack.

I. INTRODUCTION

Intrusion Detection Systems (IDSs) are the primary security tools for Industrial Control Systems (ICSs) [START_REF] Hu | A survey of intrusion detection on industrial control systems[END_REF]. IDSs date back to the early 80's. They are systems whose objective is to detect malicious activity and policy violations regardless of "whether they are initiated by outsiders who attempt to break into the system or insiders who attempt to misuse the privileges of their accounts" [START_REF] Denning | Requirements and model for IDES-a real-time intrusion-detection expert system SRI International Menlo Park[END_REF]. While early IDSs were based on signatures of known attacks only, recent solutions rely on anomaly detection models that also detect unknown attacks.

The used models are typically unsupervised: they do not need examples to be trained to detect anomalies, such as functional failure or attacks. This is important as attack examples are rare, especially for Cyber-Physical Systems (CPSs). ICSs often control several CPSs. At the same time, CPSs are more likely to be faced with unknown attacks than Information and Communications Technology systems [START_REF] Mitchell | A survey of intrusion detection techniques for cyber-physical systems[END_REF].

This work was supported by the Cyber CNI Chair and IMT Atlantique, Bretagne-Pays de la Loire.

Anomaly detection for ICS follows three directions [START_REF] Wressnegger | Content-based Anomaly Detection for Industrial Control Systems[END_REF]: 1) protocol, 2) network and 3) payloads, i.e., sensors measurements, automata commands and actuators responses, referred to as sensor data. Protocol-based anomaly detection identifies anomalies on the basis of deviation from normal packet data and protocol rules. For network-based anomaly detection, a sequence of network packets is used, modeling the communication relationships between devices. For payloadbased anomaly detection, it is the modeling of the physical relationships between devices as well as the normal range of system behavior that are considered.

The use of Deep Learning techniques to build an anomaly detection model for the security of CPSs, including ICSs, has increased in recent years [START_REF] Wickramasinghe | Generalization of deep learning for cyber-physical system security[END_REF], [START_REF] Berman | A survey of deep learning methods for cyber security[END_REF], [START_REF] Luo | Deep learning-based anomaly detection in cyber-physical systems: Progress and opportunities[END_REF], [START_REF] Mitchell | A survey of intrusion detection techniques for cyber-physical systems[END_REF]. The anomaly detection method used for monitoring an ICS is a critical tool that might be targeted by attackers. The security of IDSs themselves matters.

Yet, [START_REF] Axelsson | Research in intrusion-detection systems: A survey[END_REF] reports that security of most of IDSs is low and only one paper [START_REF] Ptacek | Insertion, evasion, and denial of service: Eluding network intrusion detectionSecure Networks inc Calgary Alberta[END_REF] that "concerns itself with the nature of attacks against intrusion detection systems themselves" [START_REF] Axelsson | Research in intrusion-detection systems: A survey[END_REF] is cited. While this survey dates from 1998, a decade later, in 2009, [START_REF] Garcia-Teodoro | Anomaly-based network intrusion detection: Techniques, systems and challenges[END_REF] stated that "more significant efforts should be done to improve intrusion detection technology in this aspect [security]". In the case of Deep Learning techniques, this concern is even more relevant since they are recent techniques whose security is not completely understood yet. The first survey on attacks against IDS dates back from 2013 [START_REF] Corona | Adversarial attacks against intrusion detection systems: Taxonomy, solutions and open issues[END_REF]. It distinguishes six attacks against IDS, including Response Hijacking-"a pattern is crafted to generate an incorrect alert description and mislead the IDS response mechanism (either automatic or performed by a security operator)" [START_REF] Corona | Adversarial attacks against intrusion detection systems: Taxonomy, solutions and open issues[END_REF]-which our study focuses on. In 2016, another survey [START_REF] Gardiner | On the security of machine learning in malware C&C detection: A survey[END_REF], about the evasion resilience of IDSs' machine learning methods, shows that the security aspect of many of their anomaly detection models is an oversight. To the best of our knowledge, there is no method that tackles the security of an anomaly detection model used by an IDS. This study is restricted to the security of payload-based anomaly detection models monitoring an ICS. The research question is how to authenticate these models.

In this paper, the anomaly detection method used by the IDS to monitor the ICS physical process is based on an autoencoder. We propose a method to verify that the genuine autoencoder is working by probing its outputs. Otherwise, we assume an autoencoder spoofing attack. Our method bases on a statistical test on the confidence measure of a multipath Neural Network (NN) classifier [START_REF] Larsen | Hyper-Neurons and Confidence Through Path Validation[END_REF]. If the test indicates a likely change of distribution of a multipath NN's confidence measure, an alert is raised. Results show that our method can detect any autoencoder spoofing attack with an arbitrary False Positive Rate (FPR) fixed in advance, yet the lower the FPR, the longer the detection.

In the rest of the paper, we first present in Section II necessary background knowledge for understanding the paper which comprises 1D-Convolutional Neural Network (CNN) autoencoder, multipath NN and Wilcoxon-Mann-Whitney test. Section III presents our method. Section IV presents the dataset used for evaluation, details two baselines to be compared to our method, exposes and discusses the results.

II. BACKGROUND AND ASSUMPTIONS

A. Motivated assumptions

We assume that an IDS anomaly detection model gets an input x (sensor data) and outputs a security signal y based on which the security team can take a decision. For instance, if the anomaly detection model is an autoencoder, the security signal y is the reconstruction of x and the security team decide whether there is an anomaly based on the differences between x and y. Too much difference suggests an anomaly in sensor data x. We would like to be sure that the security signal comes from the genuine model and no one is spoofing the model, by providing a security signal y attacker trusted to be from the IDS, to prevent the security team to be aware of a threat.

A trivial solution is to replicate the autoencoder of the IDS so that the security team just has to compare the output of their autoencoder y team to the one of the monitoring autoencoder y. If y = y team , then the monitoring autoencoder is spoofed. However, this solution adds a course of action for a malicious actors who now has two ways to learn about the anomaly detection model: through the IDS and through the replicated autoencoder. Not to mention that this implies that the security team has access to the anomaly detection model, including its architecture and weights in case a neural network is used.

Another solution is to add classical security tools to authenticate the IDS's anomaly detection model. For example, one could use a cryptographic message authentication code tag IDS (y), also known as tag, so that with the proper secret key, they can verify that tag IDS (y) = tag team (y). But solutions like this require computational resources to be added to the IDS to get tag IDS (y). Furthermore, if an attack is committed by an insider, by simply replacing the monitoring autoencoder of the IDS by another, one cannot detect this attack by the means of tags anymore.

In contrast, the solution we propose does not imply access to the IDS anomaly detection model nor additional computational resources. Instead its constraints are of another kind, it requires several input-output pairs (x,y) to authenticate the IDS anomaly detection model and its authentication is not strict but is rather given by a level of confidence. In other words, a security operator can only probe the output of the autoencoder, thus adding a layer of security, the authentication of the anomaly detection model, without compromising the previous one, the anomaly detection itself.

B. 1D-CNN autoencoder

In this paper, we consider several kinds of NNs. In particular, the anomaly detection model of the IDS is a 1D-CNN autoencoder.

NNs consist of several layers, each composed of several neurons. Given couples of input and target values, NNs are trained to predict the target from a particular input using a loss function. The loss function is applied on the output layer and a backpropagation algorithm enables changing the weights that link neurons between two different layers.

Autoencoders are NNs whose input and output have the same number of dimensions and they are trained to reconstruct their inputs. Because of some constraints on the hidden layers, such as dimension reduction, an autoencoder better reconstructs points close to the training distribution than points far from it. In an anomaly detection setting, the reconstruction error is used as an anomaly score. If the average error of reconstruction of an input is above a certain threshold, the input is considered dissimilar enough to those of the training set to raise an alert.

Autoencoders can be of the form of a Convolutional Neural Network (CNN). CNNs are NNs with drastically fewer weights than an FC (Fully Connected) NN. More specifically, a nD-CNN is composed of convolutional layers and pooling layers. It takes inputs with n spatial dimensions. Typically, a time window has one spatial dimension (n = 1), generally called temporal dimension. An image has two spatial dimensions (n = 2), height and width. CNNs pooling layers change the size (i.e., number of features) of the next layer without adding learnable parameters. A convolutional layer consists of sets of filters that are sets of kernels (tensors of order n). They apply an operation similar to the well-known convolution, namely the cross-correlation, so as to capture spatial patterns along n dimensions. In a layer after the input layer, the channel dimension corresponds to the number of filters that extract potentially different patterns. The channel dimension of the input layer depends on the input shape. For example, in a 1D-CNN, for a k-dimensional time series, there are k input channels. Details about CNNs can be found in [START_REF] Andrej | Convolutional neural networks (cnns/convnets)[END_REF].

In this paper, the IDS's anomaly detection model is a 1D-CNN autoencoder. Its inputs are time windows of a kdimensional time series of physical process data, with k the number of sensors and actuators of the observed ICS.

C. Multipath Neural Network

An important building block of our method to authenticate a 1D-CNN autoencoder is the multipath NN model.

A multipath NN is a NN with at least one layer composed of Dissector Layers (Diss-Layers) [START_REF] Larsen | Hyper-Neurons and Confidence Through Path Validation[END_REF]. A Diss-Layer is a computational unit whose output has the same number d of dimensions as the input and that is defined by a simple relation that uses 3 × d + 1 learnable parameters. This computational unit's sum of activations is a surrogate for the left-hand side of the equation of a hyperplane d i=1 a i × x i + a 0 = 0 so as to benefit from a property of supervised NN classifiers shown in [START_REF] Alain | Understanding intermediate layers using linear classifier probes https://openreview.net/forum?[END_REF]. This property is that the level of linear separability increases monotonically when we go to deeper layers. Diss-Layers are typically in the last layers.

Several Diss-Layers compose a Hyper-Neuron. A multipath NN classifier forces, during the training phase, the information to go through the dedicated Hyper-Neuron depending on the input class. During inference, it provides a confidence measure that verifies that the information goes through the right Hyper-Neuron. Its confidence measure is shown to be robust to transformations of the input distribution that mainly occur in non-discriminative features, i.e., features not responsible for the class membership.

In this paper, a multipath NN's confidence measure characterizes an autoencoder's errors using a statistical test.

D. Wilcoxon-Mann-Whitney test

The second building block of our method to authenticate a 1D-CNN autoencoder is the Wilcoxon-Mann-Whitney (WMW) test. This test is distribution-free, i.e., data are not assumed to follow a particular family of distributions, which makes our method practical for any ICS.

The WMW test [START_REF] Wilcoxon | Individual Comparisons by Ranking Methods[END_REF], [START_REF] Mann | On a test of whether one of two random variables is stochastically larger than the other The annals of mathematical statistics[END_REF] allows to check whether two samples X 1 , . . . , X n and Y 1 , . . . , Y m are from the same distribution. The assumption of this test is that the observations from both samples are independent of each other. The null hypothesis H 0 is that the distributions are the same, and the alternative hypothesis H 1 considered herein is that the distributions are different. The statistic U of this test is the minimum between the number of times a Y precedes a X, U 1 and the number of times a X precedes a y, U 2 = nm -U 1 , in the ordered sequence composed of X's and Y 's. More formally, U = min(U 1 , U 2 ) and, with 1 the indicator function:

U 1 = n i=1 m j=1 1 Xi>Yj + 1 2 n i=1 m j=1 1 Xi=Yj
The test is consistent-i.e., the probability to reject H 0 when H 1 is true converges towards 1 when n, m → +∞-

only if P(X > Y | H 1 ) = P(Y > X | H 1 )
, with P the notation for probability. For samples of size greater than 20, U 1 and U 2 are approximately normally distributed, so one can decide an appropriate acceptable False Positive Rate (FPR) α, such that H 0 is rejected whenever the p-value-probability under H 0 that |U | > |u|, with U the statistic and u the observation of this statistic-is lower than α.

In this paper, we propose to perform the WMW test on the confidence measure of a multipath NN. A sample S 1 of confidence measures known to be derived from the genuine autoencoder of the IDS serves as a reference to test if a second sample S 2 is from the same distribution.

III. AUTHENTICATION OF A 1D-CNN AUTOENCODER

Our method to authenticate an 1D-CNN autoencoder that monitors the time series of an ICS's sensors and actuators values relies on three steps. The first step consists in choosing moments in the ICS cycle that serve to characterize the autoencoder based on time windows corresponding to these moments. The second step is the training of a multipath NN classifier that must predict the moment when given the matrix of errors of the autoencoder that processed the corresponding time window. The third step is a hypothesis testing that aims at verifying that the distribution of the multipath NN's confidence measure has not changed.

A. Preprocessing of time series of sensors and actuators values

Before explaining the first step in detail, we need to introduce some notation. We consider an ICS composed of k sensors/actuators, now referred to as items. The ICS's physical process data form a k-dimensional time series, i.e., a time series that yields k values at each time step. We denote R c i,j the value of the i-th item at the j-th time step within the c th ICS cycle. We assume that the duration between two time steps is always the same. For the sake of simplicity, we do not consider a particular cycle instance c anymore but the random variable R i,j instead, i.e., the function associating R c i,j to c. Now, we can explain the first step, i.e., choosing moments in the ICS cycle. Let us first assume that the ICS executes the same sequence of actions each cycle. In this case, the time window of the n-th moment of interest can simply be R n = ((R i,j ) 1≤i≤k ) pn×L/l ≤j< pn×L/l +s , with L the variable length of the cycle, l = min(L), p n positions parameters and s the size of time windows. In other words, the cycle is simply divided into windows whose positions p n × L/l are proportional to the cycle length as depicted in Figure 1. If the size of the time windows is big enough with respect to the cycle variability, one can assume that a moment defined as in Section III-A will correspond to a same set of ICS's high-level actions from one cycle to another.

High-level actions are actions that an ICS is programmed to perform in its physical environment. For example, drilling an object or the manipulation of an object by a robotic arm. In other words, results of the working of an ICS at the human level. In this way, time windows from the same moment in different cycles are similar enough for a classifier to predict the moment given a corresponding time window. It enables training a classifier to predict the moment given the autoencoder's matrix of errors of reconstruction of a corresponding time window. This is needed for the next step.

In general, an ICS does not necessarily executes the same sequence of actions in each cycle. As an example, a car manufacturing system equipped of a single production line can produce vehicles with different features. The ICS will conduct different sequences of actions depending on the car to produce. In this case, the system's knowledge is necessary to define moments that correspond to some high-level sets of actions. However, the purpose is the same as before, that is one must T ime Item Fig. 1. Two ICS cycles with fixed size time windows from the time series of sensors/actuators values (items) represented by gray dots. Time windows are linked to 6 distinct moments whose positions in the cycle are proportional to its length. Due to the ICS's working variability, the cycle length is variable. be able to predict the moment given the autoencoder's matrix of errors of reconstruction of a corresponding time window. Once this verification done, one can proceed to the next step.

B. Training of a multipath NN classifier

The second step, after having specified moments in the ICS cycle as explained in the previous section, consists in training a multipath NN to predict the moment based on the matrix of errors-element-wise squared difference between the input and the output-of the monitoring autoencoder that processed a time window corresponding to this moment. This NN comes with a confidence measure that is robust to some transformations of the input (cf. Section II-C). The idea is to characterize errors of an IDS's autoencoder thanks to a multipath NN's confidence measure. Once m moments have been chosen, a multipath NN classifier is defined such that its last layer is a fully connected layer with m output features and its first layers are convolutional layers. Then, one needs a balanced-about the same number of examples for each class/moment-set S of matrices of reconstruction errors from the monitoring autoencoder that are normalized with the minmax normalization. To this set of size |S| are added |S|/m random matrices from the unit hypercube (cf. [START_REF] Larsen | Hyper-Neurons and Confidence Through Path Validation[END_REF]): consider a matrix as a vector and draw each dimension's value from the uniform distribution between 0 and 1. The set obtained is the training set of the multipath NN. Once the training set defined, one uses the multipath NN's loss detailed in [START_REF] Larsen | Hyper-Neurons and Confidence Through Path Validation[END_REF] that ensures that the multipath NN's confidence measure is properly learned. This confidence measure is simply the softmax output's max value multiplied by a term which allows verifying that the path of the information leading to the prediction is the expected one.

C. The Wilcoxon-Mann-Whitney test on the multipath NN's confidence measure

The third step is a Wilcoxon-Mann-Whitney (WMW) test on the confidence measures of the multipath NN. One has to decide the size of the tested sample and the size of the sample of reference. The latter is the confidence measures' sample arising during the normal operation of the ICS (without attack or dysfunction) and with the genuine autoencoder. Typically, the sample of reference is large (≥ 100), and to be able to use any FPR α (cf. Section II-D) manageable for the security team, the size of tested samples has to be greater than 20 as explained in Section II-D. The more subtle is the attack, the larger has to be the tested samples. Thus, if the multipath NN is used online, one can perform several tests on increasingly larger samples to rapidly detect rather obvious attacks and eventually detect more subtle ones.

One has to build a sample of reference in the same way as the set S from Section III-B-but different from S-except that one has to use the same normalization parameters as before (cf. Section III-B) instead of recomputing them based on the new matrices. The same holds for tested samples, but, of course, one does not know whether matrices of errors are from the genuine autoencoder. An alert is raised whenever the WMW test rejects the null hypothesis.

IV. EXPERIMENTATION

The experimentation consists in testing our method in several settings, that is with different 1D-CNN autoencoders and multipath NNs, as well as two baselines introduced in Section IV-A. After presenting the data in Section IV-B and the results in Section IV-C, we compare the ability of each method to distinguish normal samples from abnormal ones with Receiver Operating Characteristic (ROC) curves, which plot True Positive Rate (TPR) against FPR.

A. Baselines and hyperparameters

Our method aims to characterize an autoencoder's reconstruction errors by providing to the WMW test a confidence measure on its ability to retrieve the moment within the ICS cycle so as to detect spoofing attacks. One can ask if there is not a better variable to summarize an autoencoder's errors. The first baseline is the use the autoencoder's mean squared error for the WMW test. The second baseline involves the confidence measure of a classical NN classifier, that is its softmax's maximum value, as summary variable for the test.

The architectures of the classical NNs and of the multipath NNs, all trained in batches of size 16 and in 20 epochs, are:

• [START_REF] Brand | Beyond authorship: attribution, contribution, collaboration, and credit[END_REF][START_REF] Wressnegger | Content-based Anomaly Detection for Industrial Control Systems[END_REF]). F lat is a layer, without learnable parameters, used to reshape the features into a vector so that one can use a fully connected layer afterwards. Conv and Dense are respectively convolutional and fully connected layers and 5DissLayers is a layer with 5 Diss-Layers [START_REF] Larsen | Hyper-Neurons and Confidence Through Path Validation[END_REF].

Down(3)([1, 1, 2, 2, 3, 3]) =
The autoencoders' layout is: [ [START_REF] Axelsson | Research in intrusion-detection systems: A survey[END_REF][START_REF] Axelsson | Research in intrusion-detection systems: A survey[END_REF], [START_REF] Mann | On a test of whether one of two random variables is stochastically larger than the other The annals of mathematical statistics[END_REF][START_REF] Corona | Adversarial attacks against intrusion detection systems: Taxonomy, solutions and open issues[END_REF],45,72, [START_REF] Axelsson | Research in intrusion-detection systems: A survey[END_REF][START_REF] Axelsson | Research in intrusion-detection systems: A survey[END_REF]]. Output shapes (couples (a, b) or numbers c) are obtained with omitted sampling, flattening and cropping layers. A couple (a, b) indicates a convolutional layer (or the input layer at the beginning) with b channels, each with a features. A unique number c indicates a fully connected layer with c features.

B. Data

In this study, data are of two types: 1) the ICS physical process data and 2) the matrices of reconstruction errors of an 1D-CNN autoencoder that monitors the ICS physical process data. Data of the first type are simulated so that we can gather enough data of the second type to have representative results. Indeed, time series from real-world ICSs would require a large stochastic model-both from the point of view of the autoencoder and the model that authenticates the autoencoder. Yet, since we deal with stochastic models, we need several trained instances to perform a thorough analysis of the performance of the model that authenticates the autoencoder.

Data of types 1 and 2 are created as follows:

1) Simulated ICS physical process data are a binary Multivariate Markov Chain with 15 chains (for 15 sensors/actuators values) that operates a cycle in about 50 time steps in average and comprises 4000 cycles for the training set and 1000 for the test set. This requires much smaller stochastic models than real-world ICSs that typically have tens of sensors/actuators and cycles with hundreds of time steps. Moreover, real-world ICSs also have continuous sensors/actuators data, but for the sake of simplicity we simulate a binary Multivariate Markov Chain where the ICS executes the same sequence of actions in each cycle, and we select five moments (cf. Section III-A).

2) As the method's performance depends on both the genuine autoencoder and the spoofing autoencoder, not only we have to repeat the procedure for several genuine autoencoders, but each genuine autoencoder must be faced with attacks from several spoofing autoencoders. Ten 1D-CNN autoencoders are trained. Then, they alternately play the role of the (single) genuine autoencoder and the 9 spoofing autoencoders, resulting in 90 spoofing attacks which is enough to aggregate representative results. For each autoencoder, a multipath NN classifier and a traditional NN classifier are trained. Finally, samples derived from matrices of reconstruction errors are tested against a sample derived from matrices of errors from the genuine autoencoder only. More precisely, it is a summary False Positive Rate Fig. 4. ROC curves from 1 -p-values of the WMW test on the summary variable of the three methods (one per column) for samples of size 25 tested against a reference sample of size 1000 (see Figure 2 for the summary of the ROC curves per methods). Each row corresponds to a genuine autoencoder, each graph contains 9 ROC curves, one for each spoofing autoencoder.

First baseline

Second baseline

Our method

True Positive Rate False Positive Rate Fig. 5. ROC curves from 1 -p-values of the WMW test test on the summary variable of the three methods (one per column) for samples of size 25 tested against a reference sample of size 1000 (see Figure 2 for the summary of the ROC curves per methods). Each row corresponds to a genuine autoencoder, each graph contains 9 ROC curves, one for each spoofing autoencoder.

of the matrix of errors that is the variable of the WMW test, the confidence measure in the case of the multipath NN classifier and in the case of the second baseline, as well as the mean squared error in the case of the first baseline. Matrices of errors summarized into single values that compose a tested sample are all, either from the genuine autoencoder, either from a unique spoofing autoencoder. This way, a ROC curve can be plotted to visualize how well a genuine autoencoder is distinguished from a particular spoofing autoencoder, but of course this does not change the end result which is the average performance of the model that authenticates the autoencoder.

C. Results

To see which summary variable best distinguishes an abnormal sample (from a spoofing autoencoder) from a normal one (from the genuine autoencoder) with the WMW test, we plot ROC curves from 1 minus the p-value. Each point of a curve comes from a particular anomaly threshold 1 -α, with α the corresponding FPR (cf. Section II-D). Indeed, 1 minus the p-value can be seen as an anomaly scoring function whose input is a tested sample since, the more abnormal the sample, the lower the p-value due to the consistency of the test (cf. Section II-D). The more the ROC curve is near the top left corner of the graph-or similarly the bigger the Area Under the Curve (AUC)-, the better.

Results of the 90 ROC curves (see Section IV-B) for each the three methods with tested samples of size 25 are shown in Figure 2: 1st baseline's AUC = 0.6877 ± 0.16; 2nd baseline's AUC = 0.8012 ± 0.17; our method's AUC = 0.8953 ± 0.14. Similar graphs for tested samples of size 200 are shown in Figure 3: 1st baseline's AUC = 0.8650 ± 0.18; 2nd baseline's AUC = 0.9198 ± 0.16; our method's AUC = 0.9736 ± 0.08.

D. Discussion

The multipath NN's confidence measure allows the WMW test to detect a spoofing attack with less examples. When used online, it thus detects attacks faster than the baselines.

Moreover, when a ROC curve is too close to the first bisector for tests with 25 examples (Figure 2), the detection of the corresponding spoofing autoencoder is difficult, if not impossible, with more examples (Figure 3). Yet, each ROC curve of the graph related to our method move to the top left corner with the increased size of the tested samples (from Figure 2 to Figure 3), while some ROC curves from the baselines are stuck on the first bisector. This means that some spoofing autoencoder will never be detected by the baselines. In contrast, since the WMW is consistent under a reasonable assumption already mentioned (cf. Section II-D), our method will eventually detect every spoofing autoencoder, even if it takes a large sample and thus a long time for a few of them.

Figures 4 and5 show the details of Figure 2, that are, for each of the 10 autoencoders, the ROC curves corresponding to the other 9 trying to spoof them. They show that 1) in each case, our method is better than the first baseline and that 2) in only two cases out of the ten cases, the second baseline outperforms our method but only by a small margin.

Finally, it is worth noting that, in practice, when the test on the multipath NN's confidence measure rejects the null hypothesis-meaning that the distribution of the tested sample is likely to be different from the reference sample's distribution-it only indicates there is an anomaly occurring in the matrices of errors of the autoencoder, but it does not tell where this anomaly comes from. It can be from the physical process data or from a spoofing autoencoder. One can only be sure that the first case is wrong when the first baseline does not detect any change in distribution while our method detects a significant change in distribution.

V. CONCLUSION

In this paper, we propose a method to authenticate the autoencoder of an IDS monitoring an ICS by the use of inputoutput probes (Section III). This allows to add a layer of security (authentication) without compromising the previous one (anomaly detection). Our method can detect any autoencoder spoofing attacks (Section IV-D) with an arbitrary FPR, yet with a trade-off between the FPR and the time of detection.

AUTHORS CONTRIBUTIONS

Following the taxonomy from [START_REF] Brand | Beyond authorship: attribution, contribution, collaboration, and credit[END_REF]: Raphaël M.J.I Larsen's contributions: 1) Conceptualization (raised the article's problem: autencoder authentication, conceived the idea of using of the multipath Neural Network to treat this problem) 2) Methodology 3) Investigation 4) Formal Analysis 5) Software 6) Visualization 7) Validation 8) Writing (Original Draft & Editing).

Marc-Oliver Pahl's contributions: 1) Validation 2) Supervision (gave precise guidelines for the first draft presentation that is best for the topic treated) 3) Writing (Review & Editing). Gouenou Coatrieux's contributions: 1) Conceptualization (conceived the idea of applying some digital forensics principles for authentication) 2) Supervision (helped defining the notion of traceability, which is a notion that led to the idea of autoencoder authentication) 3) Funding acquisition.

  Classical NN classifier InputLayer(15,15) → Conv(15,19) → U p(4)(60,19) → Down(5)(12,19) → Conv(12,22) → U p(4)(48,22) → Down(5)(10,2) → Conv(10,24) → U p(4)(40,24) → Down(5)(8,24) → Conv(8,25) → F lat(200) → Dense(5) • multipath NN classifier (simplest form: one Diss-Layer per dedicated Hyper-Neuron) InputLayer(15,15) → Conv(15,19) → U p(4)(60,19) → Down(5)(12,19) → Conv(12,22) → U p(4)(48,22) → Down(5)(10,2) → Conv(10,24) → U p(4)(40,24) → Down(5)(8,24) → Conv(8,25) → F lat(200) → Dense(100) → 5DissLayers(100) → Dense(5) A couple or a number after layers U p(n), Down(m), Conv, Dense, F lat or 5DissLayers indicates the layer's output shape. U p(n) repeats data n times (for example, U p(2)([1, 2, 3]) = [1, 1, 2, 2, 3, 3]), and Down(m) operates a max-pooling with strides m and pool size 2 (for example,

Fig. 2 .

 2 Fig.2. ROC curves from 1 -p-values of the WMW test on the three summary variables (Section IV-A) for samples of size 25 tested against a reference sample of size 1000. First 3 graphs: 90 ROC curves in each, details in Fig.4 and 5. Fourth graph: mean curves (ROC1/2: 1st/2nd baseline, ROC3: our method).

Fig. 3 .

 3 Fig. 3. Same as in Figure 2 except that the tested samples' size is 200.

  AUC = 0.596426) ROC 2 (AUC = 0.512524) ROC 3 (AUC = 0.464492) ROC 4 (AUC = 0.539816) ROC 5 (AUC = 0.625890) ROC 6 (AUC = 0.501597) ROC 7 (AUC = 0.784801) ROC 8 (AUC = 0.553250) ROC 9 (AUC = 0.942415) Mean ROC1; AUC = 0.6135 ± 1.46E-01 ± 1 std. dev. visualization; AUC median = 0AUC = 0.628748) ROC 2 (AUC = 0.820850) ROC 3 (AUC = 0.895782) ROC 4 (AUC = 0.860498) ROC 5 (AUC = 0.919087) ROC 6 (AUC = 0.617445) ROC 7 (AUC = 0.907677) ROC 8 (AUC = 0.672261) ROC 9 (AUC = 0.616213) Mean ROC1; AUC = 0.7710 ± 1.27E-01 ± 1 std. dev. visualization; AUC median = 0AUC = 0.857573) ROC 2 (AUC = 0.957957) ROC 3 (AUC = 0.977972) ROC 4 (AUC = 0.974970) ROC 5 (AUC = 0.987866) ROC 6 (AUC = 0.854484) ROC 7 (AUC = 0.985175) ROC 8 (AUC = 0.679704) ROC 9 (AUC = 0.885230) Mean ROC1; AUC = 0.9067 ± 9.56E-02 ± 1 std. dev. visualization; AUC median = 0AUC = 0.675416) ROC 2 (AUC = 0.520569) ROC 3 (AUC = 0.503399) ROC 4 (AUC = 0.503240) ROC 5 (AUC = 0.666789) ROC 6 (AUC = 0.490653) ROC 7 (AUC = 0.865505) ROC 8 (AUC = 0.597224) ROC 9 (AUC = 0.967713) Mean ROC1; AUC = 0.6434 ± 1.62E-01 ± 1 std. dev. visualization; AUC median = 0AUC = 0.793746) Mean ROC1; AUC = 0.6700 ± 1.07E-01 ± 1 std. dev. visualization; AUC median = 0.6626 AUC = 0.508391) ROC 2 (AUC = 0.685297) ROC 3 (AUC = 0.628495) ROC 4 (AUC = 0.695705) ROC 5 (AUC = 0.794987) ROC 6 (AUC = 0.600241) ROC 7 (AUC = 0.685119) ROC 8 (AUC = 0.597557) ROC 9 (AUC = 0.510445) Mean ROC1; AUC = 0.6340 ± 8.72E-02 ± 1 std. dev. visualization; AUC median = 0AUC = 0.990147) ROC 2 (AUC = 0.999936) ROC 3 (AUC = 0.998748) ROC 4 (AUC = 0.998262) ROC 5 (AUC = 0.999474) ROC 6 (AUC = 0.995404) ROC 7 (AUC = 0.994897) ROC 8 (AUC = 0.999233) ROC 9 (AUC = 0.940165) Mean ROC1; AUC = 0.9903 ± 1.81E-02 ± 1 std. dev. visualization; AUC median = 0AUC = 0.981046) ROC 2 (AUC = 0.998380) ROC 3 (AUC = 0.995263) ROC 4 (AUC = 0.996343) ROC 5 (AUC = 0.997202) ROC 6 (AUC = 0.997419) ROC 7 (AUC = 0.999651) ROC 8 (AUC = 0.977177) ROC 9 (AUC = 0.986465) Mean ROC1; AUC = 0.9919 ± 7.86E-03 ± 1 std. dev. visualization; AUC median = 0AUC = 0.544750) ROC 2 (AUC = 0.746052) ROC 3 (AUC = 0.524717) ROC 4 (AUC = 0.694564) ROC 5 (AUC = 0.907965) ROC 6 (AUC = 0.749742) ROC 7 (AUC = 0.519136) ROC 8 (AUC = 0.809414) ROC 9 (AUC = 0.550447) Mean ROC1; AUC = 0.6719 ± 1.34E-01 ± 1 std. dev. visualization; AUC median = 0AUC = 0.556104) Mean ROC1; AUC = 0.8371 ± 1.32E-01 ± 1 std. dev. visualization; AUC median = 0.8752 AUC = 0.963793) ROC 2 (AUC = 0.997680) ROC 3 (AUC = 0.986459) ROC 4 (AUC = 0.989847) ROC 5 (AUC = 0.995926) ROC 6 (AUC = 0.849774) ROC 7 (AUC = 0.959566) ROC 8 (AUC = 0.981957) ROC 9 (AUC = 0.998284) Mean ROC1; AUC = 0.9691 ± 4.43E-02 ± 1 std. dev. visualization; AUC median = 0AUC = 0.970670) ROC 2 (AUC = 0.999368) ROC 3 (AUC = 0.998209) ROC 4 (AUC = 0.955294) ROC 5 (AUC = 0.996753) ROC 6 (AUC = 0.992721) ROC 7 (AUC = 0.997192) ROC 8 (AUC = 0.999364) ROC 9 (AUC = 0.867148) Mean ROC1; AUC = 0.9749 ± 4.09E-02 ± 1 std. dev. visualization; AUC median = 0AUC = 0.991084) ROC 2 (AUC = 0.998006) ROC 3 (AUC = 0.997266) ROC 4 (AUC = 0.999708) ROC 5 (AUC = 0.999893) ROC 6 (AUC = 0.975867) ROC 7 (AUC = 0.998996) ROC 8 (AUC = 0.976291) ROC 9 (AUC = 0.997465) Mean ROC1; AUC = 0.9924 ± 9.23E-03 ± 1 std. dev. visualization; AUC median = 0AUC = 0.615376) ROC 2 (AUC = 0.845812) ROC 3 (AUC = 0.853094) ROC 4 (AUC = 0.780395) ROC 5 (AUC = 0.955428) ROC 6 (AUC = 0.860148) ROC 7 (AUC = 0.558165) ROC 8 (AUC = 0.902962) ROC 9 (AUC = 0.676759) Mean ROC1; AUC = 0.7831 ± 1.29E-01 ± 1 std. dev. visualization; AUC median = 0AUC = 0.761117) Mean ROC1; AUC = 0.9073 ± 9.79E-02 ± 1 std. dev. visualization; AUC median = 0.9495

  AUC = 0.614298) ROC 2 (AUC = 0.713482) ROC 3 (AUC = 0.658932) ROC 4 (AUC = 0.759217) ROC 5 (AUC = 0.519867) ROC 6 (AUC = 0.734737) ROC 7 (AUC = 0.570578) ROC 8 (AUC = 0.520051) ROC 9 (AUC = 0.802026) Mean ROC1; AUC = 0.6548 ± 9.86E-02 ± 1 std. dev. visualization; AUC median = 0AUC = 0.709572) ROC 2 (AUC = 0.511939) ROC 3 (AUC = 0.495213) ROC 4 (AUC = 0.508287) ROC 5 (AUC = 0.730376) ROC 6 (AUC = 0.489282) ROC 7 (AUC = 0.905116) ROC 8 (AUC = 0.624289) ROC 9 (AUC = 0.971965) Mean ROC1; AUC = 0.6607 ± 1.72E-01 ± 1 std. dev. visualization; AUC median = 0AUC = 0.728660) ROC 2 (AUC = 0.530915) ROC 3 (AUC = 0.518155) ROC 4 (AUC = 0.534118) ROC 5 (AUC = 0.539571) ROC 6 (AUC = 0.732708) ROC 7 (AUC = 0.896325) ROC 8 (AUC = 0.694590) ROC 9 (AUC = 0.967378) Mean ROC1; AUC = 0.6825 ± 1.58E-01 ± 1 std. dev. visualization; AUC median = 0AUC = 0.492040) ROC 2 (AUC = 0.597622) ROC 3 (AUC = 0.685882) ROC 4 (AUC = 0.624408) ROC 5 (AUC = 0.722489) ROC 6 (AUC = 0.507682) ROC 7 (AUC = 0.702986) ROC 8 (AUC = 0.594221) ROC 9 (AUC = 0.830140) Mean ROC1; AUC = 0.6397 ± 1.01E-01 ± 1 std. dev. visualization; AUC median = 0AUC = 0.950447) Mean ROC1; AUC = 0.6351 ± 1.44E-01 ± 1 std. dev. visualization; AUC median = 0.5877 AUC = 0.901293) ROC 2 (AUC = 0.856995) ROC 3 (AUC = 0.781862) ROC 4 (AUC = 0.855063) ROC 5 (AUC = 0.902173) ROC 6 (AUC = 0.936901) ROC 7 (AUC = 0.784235) ROC 8 (AUC = 0.885804) ROC 9 (AUC = 0.508249) Mean ROC1; AUC = 0.8236 ± 1.22E-01 ± 1 std. dev. visualization; AUC median = 0AUC = 0.521700) ROC 2 (AUC = 0.654874) ROC 3 (AUC = 0.558657) ROC 4 (AUC = 0.538052) ROC 5 (AUC = 0.850356) ROC 6 (AUC = 0.514839) ROC 7 (AUC = 0.561840) ROC 8 (AUC = 0.735872) ROC 9 (AUC = 0.752558) Mean ROC1; AUC = 0.6321 ± 1.15E-01 ± 1 std. dev. visualization; AUC median = 0AUC = 0.560688) ROC 2 (AUC = 0.669419) ROC 3 (AUC = 0.682282) ROC 4 (AUC = 0.826857) ROC 5 (AUC = 0.869355) ROC 6 (AUC = 0.893401) ROC 7 (AUC = 0.497182) ROC 8 (AUC = 0.611723) ROC 9 (AUC = 0.493141) Mean ROC1; AUC = 0.6783 ± 1.45E-01 ± 1 std. dev. visualization; AUC median = 0AUC = 0.816964) ROC 2 (AUC = 0.993256) ROC 3 (AUC = 0.988613) ROC 4 (AUC = 0.998232) ROC 5 (AUC = 0.998312) ROC 6 (AUC = 0.981747) ROC 7 (AUC = 0.991624) ROC 8 (AUC = 0.941366) ROC 9 (AUC = 0.785426) Mean ROC1; AUC = 0.9438 ± 7.84E-02 ± 1 std. dev. visualization; AUC median = 0AUC = 0.525461) Mean ROC1; AUC = 0.8090 ± 1.29E-01 ± 1 std. dev. visualization; AUC median = 0.8737 AUC = 0.922447) ROC 2 (AUC = 0.815222) ROC 3 (AUC = 0.896571) ROC 4 (AUC = 0.943788) ROC 5 (AUC = 0.992210) ROC 6 (AUC = 0.951064) ROC 7 (AUC = 0.973371) ROC 8 (AUC = 0.879130) ROC 9 (AUC = 0.934651) Mean ROC1; AUC = 0.9230 ± 5.05E-02 ± 1 std. dev. visualization; AUC median = 0AUC = 0.704161) ROC 2 (AUC = 0.885668) ROC 3 (AUC = 0.919514) ROC 4 (AUC = 0.657438) ROC 5 (AUC = 0.985457) ROC 6 (AUC = 0.858301) ROC 7 (AUC = 0.844255) ROC 8 (AUC = 0.951317) ROC 9 (AUC = 0.694091) Mean ROC1; AUC = 0.8333 ± 1.13E-01 ± 1 std. dev. visualization; AUC median = 0AUC = 0.516259) ROC 2 (AUC = 0.922228) ROC 3 (AUC = 0.647273) ROC 4 (AUC = 0.515374) ROC 5 (AUC = 0.599721) ROC 6 (AUC = 0.857666) ROC 7 (AUC = 0.564041) ROC 8 (AUC = 0.699724) ROC 9 (AUC = 0.583747) Mean ROC1; AUC = 0.6562 ± 1.37E-01 ± 1 std. dev. visualization; AUC median = 0AUC = 0.996440) ROC 2 (AUC = 0.999286) ROC 3 (AUC = 0.998498) ROC 4 (AUC = 0.997570) ROC 5 (AUC = 0.999442) ROC 6 (AUC = 0.998731) ROC 7 (AUC = 0.999974) ROC 8 (AUC = 0.990077) ROC 9 (AUC = 0.925594) Mean ROC1; AUC = 0.9891 ± 2.28E-02 ± 1 std. dev. visualization; AUC median = 0AUC = 0.611998) Mean ROC1; AUC = 0.9243 ± 1.14E-01 ± 1 std. dev. visualization; AUC median = 0.9574