Arnaud Rosay
email: arnaud.rosay@st.com

Florent Carlier
email: florent.carlier@univ-lemans.fr

Pascal Leroux
email: pascal.leroux@univ-lemans.fr

MLP4NIDS: an efficient MLP-based Network Intrusion Detection for CICIDS2017 dataset

Keywords: Machine Learning, Multi-Layer Perceptron, Network Intrusion Detection, CICIDS2017 Dataset

More and more embedded devices are connected to the internet and therefore are potential victims of intrusion. While machine learning algorithms have proven to be robust techniques, it is mainly achieved with traditional processing, neural network giving worse results. In this paper, we propose usage of a multi-layer perceptron neural network for intrusion detection and provide a detailed description of our methodology. We detail all steps to achieve better performances than traditional machine learning techniques with a detection of intrusion accuracy above 99% and a low false positive rate kept below 0.7%. Results of previous works are analyzed and compared with the performances of the proposed solution.

Introduction

In recent years, IoT is growing in all areas. This is typically the case for smart agents (IoT-a) that proliferates to provide complex functionalities. Such smart agents rely on communication to cooperate [START_REF] Carlier | IoT-a, embedded agents for smart internet of things. application on a display wall[END_REF]. This is also true for cars that shall embed a cellular connection for emergency call. This is mandatory in Europe from April 1st, 2018 for all passengers cars and light commercial vehicles [START_REF] Parliament | [END_REF] and also in Russia from January 2017. The presence of a modem enables the emergence of new services requiring data transfer through this cellular connection. In such a context, attacks against IoT-a may lead to loss of functionality or even worse open access to other elements of the network potentially compromising privacy. Cars become potential victims of hackers able to run remote attacks on entire fleet of vehicles. In response to these risks, a first approach is to use network intrusion detection system (IDS) tools such as Snort (open source IDS tool) to analyze network traffic and extract signatures that can be compared to known attack signatures. This type of approach has two limitations: on the one hand, the signature of an attack must already be known and on the other hand, it generates a high false positive rate [START_REF] Garg | Performance analysis of Snort-based Intrusion Detection System[END_REF]. An alternative approach is to use Machine Learning (ML) techniques. These techniques include supervised learning to classify network flows into different categories and unsupervised learning to detect anomalies.

Supervised learning is only possible by having a dataset available, in this case the recording of network frames containing normal traffic and attacks. Cellular or WLAN conecctivity are quite common in IoT and connected cars and we can consider that the intrusion is similar to the attack that would be carried out on a conventional computer network. Common network intrusion detection datasets like KDD-Cup99 [START_REF] Lee | Mining in a data-flow environment: Experience in network intrusion detection[END_REF], NSL-KDD [START_REF] Tavallaee | A detailed analysis of the KDD CUP 99 data set[END_REF] or CICIDS2017 [START_REF] Sharafaldin | Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization[END_REF] containing different types of attacks against computers or servers can be used for IoT and connected cars.

Traditional machine learning algorithms and neural network-based techniques (also called deep learning) can be used as supervised learning methods. The former group contains algorithms like Decision Tree, Random Forest, or SVM. Multi-Layer Perceptron (MLP), Convolutional Neural Network (CNN) or Recurrent Neural Network (RNN) are examples of the latter group. In IDS researches, many papers cover deeply the traditional approaches while a smaller amount of papers concentrate on neural network techniques.

Our main contributions consist in two parts. Firstly, we propose an approach based on multi-layer perceptron exercised on recent dataset. CICIDS2017 contains intrusion attacks and traffics that are representative of current network usage. All steps are detailed from dataset analysis to tuning of neural network. Secondly, we analyse previous works on the same dataset and compare their performances with our experimental results. This paper presents in section 2 the related work. Our proposed methodology is described in section 3. Section 4 provides the results of our approach using several metrics and compares it to previous works. A link to the source code is provided so that it can be used to reproduce results and for further improvements. Finally, section 5 concludes this paper and identifies ideas for future work.

Related work

Several studies have been conducted on multiple network intrusion detection datasets using various methods. Dhanabal et Shantharajah [START_REF] Dhanabal | A Study on NSL-KDD Dataset for Intrusion Detection System Based on Classification Algorithms[END_REF] analyzed the NSL-KDD dataset content and studied several classifiers from the traditional machine learning techniques. They obtained pretty good intrusion detection with accuracy around 99% with SVM and J48 (C4.5) decision tree. Tang et al. [START_REF] Tang | Deep learning approach for Network Intrusion Detection in Software Defined Networking[END_REF] used a deep learning approach on the same dataset. They proposed a small neural network with a very limited number of features in input and reached an accuracy of almost 76%.

With the release of CICIDS2017, Sharafaldin et al. [START_REF] Sharafaldin | Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization[END_REF] provided performances of intrusion detection on their dataset. A significant gap is observed on precision, recall and f 1 -measure between all traditional ML algorithms and multi-layer perceptron. K-Nearest Neighbors and Quadratic Discriminant Analysis outperformed MLP by almost 20%. Feature selection, data pre-processing and details about the classifiers are not described and therefore results cannot be reproduced. In their paper, Jiang et al. [START_REF] Jiang | ALDD: A Hybrid Traffic-User Behavior Detection Method for Application Layer DDoS[END_REF] focused only on denial of service, 4 classes among the 14 of CICIDS2017 dataset. They proposed new features and compared the results of a neural network with features provided in original CICIDS2017 CSV files. A two-level model was proposed by Ullah et Mahmoud [START_REF] Ullah | A Two-Level Hybrid Model for Anomalous Activity Detection in IoT Networks[END_REF]. The first level classes traffic either as normal or attack with a decision tree and the second one identifies the attack type with a random forest after data augmentation based on synthetic minority oversampling technique [START_REF] Chawla | SMOTE: Synthetic minority over-sampling technique[END_REF] and edited nearest neighbors. The method has been tested on CICIDS2017 and UNSW-NB15 datasets. Ustebay et al. [START_REF] Ustebay | Intrusion Detection System with Recursive Feature Elimination by Using Random Forest and Deep Learning Classifier[END_REF] proposed a two-step approach for CICIDS2017 dataset. Recursive Feature Elimination (RFE) based on Random Forest is used to identify the most useful features that are then injected in a neural network.

Those works either obtain lower performances with neural network or when achieving good results do not consider all types of attacks. We propose a detailed methodology to achieve high performance with neural network without removing any type of attacks.

Our methodology and MLP solution

After selecting a dataset, we propose an approach consisting in training a multilayer perceptron neural network in order to quantify the benefits and potential limitations of using deep learning in IDS. Fig. 1 give the overall framework.

Choice of dataset

KDD-Cup99 is a dataset for IDS publicly released in 1999. It is derived from DARPA-98 dataset that contains raw data corresponding to captured network frames with TCPDUMP. The training set is composed of 24 types of attacks. Raw data have been processed to produce 41 features in KDD-cup99. A first critique of this dataset has been carried out by McHugh [START_REF] Mchugh | Testing Intrusion Detection Systems: A Critique of the 1998 and 1999 DARPA Intrusion Detection System Evaluations As Performed by Lincoln Laboratory[END_REF]. Tavallaee et al. [START_REF] Tavallaee | A detailed analysis of the KDD CUP 99 data set[END_REF] provided a detailed analysis and proposed a derived dataset referred to as NSL-KDD with the goal to solve some of the shortcomings of KDD-Cup99 described McHugh. NSL-KDD dataset has been widely used for IDS since 2009. Despite that attacks has evolved over the time, KDD-Cup99 and NSL-KDD remains a subject of study [START_REF] Riyaz | An Intelligent Fuzzy Rule based Feature Selection for Effective Intrusion Detection[END_REF] [15] [START_REF] Zyad | Improve R2L Attack Detection Using Trimmed PCA[END_REF] . We can consider that most of current attacks are not present in a dataset relying on traffic recorded 20 years ago.

In 2015, Moustafa et Slay proposed a new dataset called UNSW-NB15 [START_REF] Moustafa | UNSW-NB15: A comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set)[END_REF] that has been generated by simulation and representing 31 hours of traffic. As a major difference with NSL-KDD, it contains low footprint and modern attacks retrieved from CVE site3 that are grouped in nine different families of attacks. UNSW-NB15 is composed of 49 features. A part has been extracted from packet headers while some others have been generated specifically.

Gharib et al. [START_REF] Gharib | An Evaluation Framework for Intrusion Detection Dataset[END_REF] reviewed different datasets and evaluated them with respects to 11 criteria: complete network configuration, complete traffic, labeled dataset, complete interaction, complete capture, available protocols, attack diversity, anonymity, heterogeneity, feature set, metadata. On top of elements already identified in [START_REF] Tavallaee | A detailed analysis of the KDD CUP 99 data set[END_REF], KDD-Cup99 and therefore its NSL-KDD derivative suffer from the lack of some important protocols like HTTPS.

The University of New Brunswick's Canadian Institute for Cybersecurity released the CICIDS2017 dataset based on the framework defined in [START_REF] Sharafaldin | Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization[END_REF] intending to solve shortcomings of previous datasets. Raw data in the form of PCAP files are provided together with a set of 84 features in CSV files. Network traffic has been recorded over 5 days. The first one only contains normal traffic while 14 types of attack appear for the other days.

Due to the shortcomings of KDD-cup99 and NSL-KDD, we restricted the choice to either UNSW-NB15 or CICIDS2017. Table 1 presents 7 parameters. The first one is year of creation of each dataset. Then the number of features gives us the quantity of input data and labels characterizing each record. The number of records and the distribution between normal traffic and data are important information for dataset selection.

We observe that CICIDS2017 is more recent, include more features and contains more instances than UNSW-NB15. Date of creation and quantity of data are key elements. Firstly, old dataset are no longer representative of current network traffic. As example NSL-KDD does not contain HTTPS exchanges while it represents more than 70% of traffic today. Secondly, more data leads to better learning. For these reasons, CICIDS2017 dataset is selected for the evaluation of our intrusion detection solution.

Data Preparation

Original dataset Clean-up A series of operations has been carried out to detect the presence of empty lines, redundant features and non numeric values in numeric fields. This systematic verification allowed to drop more than 280,000 instances whose all features were empty. One column corresponding to forward header length appearing twice, one instance has been removed. Most of features are numeric but some values are not a number. Such cases appear in 6 different traffic types: BENIGN, FTP-PATATOR, DoS Hulk, Bot, PortScan and DDoS. As the number of instances containing 'NaN' or 'Infinity' is negligible in each traffic type, these instances have simply been removed. Table 2 provides the number of records for each traffic type and the number of 'NaN'/'Infinity'. Each instance is characterized by its source and destination port and IP address, its protocol and a flow identifier containing the same information. As 'FlowID' feature is redundant with other features, it has been removed from the dataset.

Our model will be trained with 2 feature sets in order to ease comparison with previous works on the same dataset. Variant-1 contains 73 characteristics: all features except those listed above. In variant-2, source/destination IP address and source port have be dropped, resulting in 70 features.

Standardization Data pre-processing is essential to prepare the dataset for an efficient training. In particular, the neural network can better learn when all features are scaled within the same range. This is especially useful when the inputs are on very different scales. Several normalization techniques exist. In our implementation, Z-score normalization (also called standardization) has been selected as it presents better results than other techniques on the selected dataset. For each feature F j , the transformation of each value x i is given by equation 1 where µ (Fj ) and σ (Fj ) are respectively the mean and standard deviation values of feature F j .

x

(Fj ) i = x (Fj ) i -µ (Fj ) σ (Fj ) (1)

Model creation

Model description Multi-layer perceptron is a fully connected, feed-forward neural network classifier. Figure 2 shows the architectural design of our model. Inputs correspond to the normalized values of the selected features of the original dataset. Both hidden layers contain 256 nodes. The classifier has 15 outputs for the 14 types of attacks and the benign traffic. Dropout is used as regularization technique for hidden layers to prevent over-adjustment on training data by dropping units randomly in the MLP with a probability keep prob.

Neural network output h (3) is calculated by chaining outputs of the different layers according to equations 2, 3 and 4 in which x, W (i) and b (i) are respectively the input vector containing selected features, the matrix of weights and the vector of biases for layer i.

h (1) = g 1 W (1)T .x + b (1)
(2)

h (2) = g 1 W (2)T .h (1) + b (2) (3)
h (3) = g 2 W (3)T .h (2) + b (3) (4)
Activation functions g 1 and g 2 bring non linearity to neural network. As MLP does not provide an intrinsic normalization of its outputs, scaled exponential linear unit given in equation 5 is used as activation function for hidden layers with values λ = 1.0507 and α = 1.6733 as defined in [START_REF] Klambauer | Self-normalizing neural networks[END_REF] to take advantage of self normalization. Output layer is a softmax activation function g 2 as defined in equation 6 so that each output can be interpreted as the probability of predicting a given class. The predicted label ŷ is given by ŷ = argmax h (3) . is optimized with Adam algorithm. Three parameters α, β 1 and β 2 described in [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] allow to configure this optimizer.

L(w) = -[y. log (ŷ) + (1 -y) . log (1 -ŷ)] (7)
Two different models corresponding to the two variants described in section 3.2 have been trained.

Evaluation Metrics

Several key information can be extracted from a confusion matrix as depicted in Table 4: True Positive (TP) is the number of attacks correctly predicted as attacks, True Negative (TN) is the number of normal instances classified as normal traffic while False Positive (FP) and False Negative (FN) are respectively the number of normal instances classified as attacks and the number of attacks predicted as normal traffic. Different metrics can be derived from the information contained in the confusion matrix. As some existing papers use only a subset of metrics, we propose to cover more metrics in order to enable comparison of our work with future studies on the same dataset.

T N rate given in equation 8 is the percentage of traffic classified as benign over the actual number of benign instances and F P rate in equation 9 is the percentage of benign traffic classified as attacks.

T N rate = T N T N + F P (8)
F P rate = F P T N + F P

P recision and Recall given in equations 10 and 11 correspond respectively to the percentage of attacks correctly detected over the number of instances predicted as attacks and the percentage of attacks correctly detected over the total number of actual attacks.

P recision =

T P T P + F P (10)

Recall = T P T P + F N (11)
Accuracy measures the proportion of total number of correct classifications.

Accuracy = T P + T N T P + T N + F P + F N (12)
f 1 score is the harmonic mean of P recision and Recall

f 1 score = 2 × precision × recall precision + recall (13)
Matthews Correlation Coefficient (MCC) is an interesting measure taking into account all elements of the confusion matrix in a correct way for imbalanced dataset as opposed to the accuracy which may report high value even when the whole minority class is wrongly classified. As intrusion detection datasets are intrinsically imbalanced, this is an key metric for such application. MCC provides a value between -1 and +1. A perfect prediction corresponds to M CC = 1. At the opposite, M CC = -1 denotes a full disagreement between predictions and actual classes. A random prediction would result to M CC = 0.

M CC =

T P × T N + F P × F N

(T P + F P ) × (T P + F N ) × (T N + F P ) × (T N + F N ) (14)
4 Experimental results

Performance evaluation

After training a golden model, the test set has been used to measure the MLP performance as a 15-class classifier. The resulting confusion matrices for the 2 variants are simplified in Tables 5 and6 to show the estimated class in 3 columns: benign, correct attack and other attacks. In multi-class classification, an attack can be predicted as another attack. This denotes an issue if the correct class is required. In intrusion detection, such a misclassification is acceptable as it is still considered as an attack. Consequently, predictions of all attack types have been merged in one single class to get a binary classifier to calculate metrics in Table 7.

Table 5 shows that MLP performs well on all classes except 'Infiltration'. This specific issue is most likely due to the extremely low number of instances. The neural network didn't succeed in learning from the only 18 examples available in the training set. A significant difference can be observed in Table 6. On the one hand, 1244 normal traffic instances are classified as error, generating false positive detection. On the other hand, more attack instances are classified as normal traffic for almost all classes, meaning that the classifier failed to detect some attacks. In particular, 'infiltration' and 'WebAttack' are not successfully detected. Again, it corresponds to classes with low number of instances. It confirms the well-know fact that the amount of data is a key point for success in machine learning.

The best results are obtained with variant-1 but we can guess the MLP learns the IP address of the machine conducting the attack. By removing IP addresses and source ports in variant-2, the model requires a longer training to reach good performances without being as good as the previous one. This clearly show the importance of these 3 features. Even if the accuracy is above 99%, the recall falls from 99.99% to 99.34% and F P rate goes up from 0.08% to 0.62%. Our classifier provides an MCC value close to 1 and therefore indicate a pretty good performance.

All results can be fully reproduced by using the source code available on Github (https://github.com/ArnaudRosay/mlp4nids) to generate the training, cross-validation and test sets, build the golden model and obtain the values of all the metrics described in this document.

Comparison with prior results

Table 8 compares results with reference papers. As different metrics are used, it is not possible to fill in all cells of the table. In addition to neural network solutions, traditional machine learning algorithms are also considered for this comparison.

Our model outperforms all the results reported by Sharafaldin et al, both for neural network and traditional machine learning techniques. Jiang et al. achieves a slightly worse performances than our solution. It should be noted that the type of attacks has been limited to application layer DoS (slowloris, slowhttptest, hulk, DDos GoldenEye) and with a very limited amount of instances (4171). Therefore it is difficult to make an exact comparison.

A two-stage approach using decision trees proposed by Ullah and Mahmoud [START_REF] Ullah | A Two-Level Hybrid Model for Anomalous Activity Detection in IoT Networks[END_REF] reports metrics with an average of 100%, a closer look at the details reveals that 4 attack types are not perfectly detected by the decision trees. These are the exact same classes for which our classifier encounter some difficulties. The lack of significant digits and averaging method in [START_REF] Ullah | A Two-Level Hybrid Model for Anomalous Activity Detection in IoT Networks[END_REF] do not allow a fine-grain analysis but the overall comparison shows that our MLP-based solution reaches the same performance level as traditional machine learning algorithms.

The neural network based solution of Jiang et al. [START_REF] Jiang | ALDD: A Hybrid Traffic-User Behavior Detection Method for Application Layer DDoS[END_REF] achieved results similar to our proposed method but only cover DoS attacks. As shown in Table 5 and 6, thoses attacks are not the most difficult to detect. We can expect a drop of their overall performance when all other attacks are taken into account.

Ustebay et al. [START_REF] Ustebay | Intrusion Detection System with Recursive Feature Elimination by Using Random Forest and Deep Learning Classifier[END_REF] also used multi-layer perceptron but did not achieved high performance. Nevertheless, their results cannot be directly compared with [START_REF] Jiang | ALDD: A Hybrid Traffic-User Behavior Detection Method for Application Layer DDoS[END_REF] and [START_REF] Ullah | A Two-Level Hybrid Model for Anomalous Activity Detection in IoT Networks[END_REF] as the latter use IP addresses of the machines conducting attacks. In a real life scenario, addresses of hackers are not known and cannot be used for intrusion detection. It should be noted that once source IP address is removed, RFE reports the source port as the most important feature. More generally, this proves that usage of source/destination ports and addresses features may improve intrusion detection on CICIDS2017 but may not be realistic for application to a real network.

Conclusion

This paper proposed an approach based on multi-layer perceptron for network intrusion detection system covering analysis of the dataset, definition of the MLP and its training. As in any machine learning project, cleaning the dataset and selecting features is an important step before optimization of a neural network. The experiment has shown that MLP is a viable solution reaching top performance. Our approach provides better results than previous implementations with neural networks. It should be noted that IP addresses and destination port are important features, helping to detect intrusion detection. Nevertheless this is not suitable for real world implementation. Without these features, our approach reaches high performance at the cost of a longer training phase. Deep learning techniques are computationally expensive. We only focused on performance but in a constrained system it may be a drawback pushing to use traditional machine learning algorithms. This may evolve with the increase of hardware accelerators for deep learning in many electronic devices. In future work, neural network in supervised learning may be improved by data augmentation techniques. Number of instances in classes that are not well detected is clearly a point to address to obtain better performances. Beyond this, supervised learning does not allow detection of attacks unseen during the training phase. Unsupervised learning methods may be considered to overcome this limitation.

Fig. 1 .

 1 Fig. 1. Overview of the proposed approach.

Fig. 2 .

 2 Fig. 2. MLP architectural design.

g 1 Training

 1 (z) = λ. α.(e z -1) for z < 0 z for z For model implementation and training, we have used python and Tensorflow as deep learning framework. Training set is divided in mini-batch of 32 instances. MLP learns classification by tuning the weights w between neural network nodes in order to reduce the cross-entropy loss function L(w) as defined in equation 7 where y the ground truth label and ŷ the predicted class. L(w)

Table 1 .

 1 Comparison of UNSW-NB15 and CICIDS2017.

	Parameters	UNSW-NB15 CICIDS2017
	Year of creation	2015	2017
	Features	49 (+ 2 labels)	84 (+ 1 label)
	Attack families	9	14
	Duration	31 hours	5 days
	# of instances	2,540,044	2,830,743
	# of normal instances	2,218,764	2,273,097
	# of attack instances 321,283 (12.65%) 557,646 (19.70%)

Table 2 .

 2 CICIDS2017 Traffic Instances.Training set, cross-validation set and test set creation As shown in Table2, the dataset is highly imbalanced at two different levels. First, the normal traffic (BENIGN) accounts for 80% and second, some attacks (Heartbleed, Infiltration, WebAttack SQL injection) are represented by a very limited number of instances. This is known to be a challenge for supervised learning. The proposal to create dataset for MLP training and testing consists in choosing randomly 50% of each attacks for the training set, 25% for the cross-validation set and 25% for the test set, ensuring that each instance is used only once. Then each set is completed by adding randomly selected instances of normal traffic. This results in a dataset that is balanced in term of normal traffic versus attacks but still imbalanced in term of attack types. The exact composition of the training, cross-validation and test set is provided in Table3.

	Traffic	original instances NaN/Infinity after clean-up
	BENIGN	2,273,097	1,777	2,271,320
	Bot	1,966	10	1,956
	DDoS	128,027	2	128,025
	DoS GoldenEye	10,293	0	10,293
	DoS Hulk	231,073	949	230,124
	DoS Slowhttptest	5,499	0	5,499
	DoS Slowloris	5,796	0	5,796
	FTP-PATATOR	7,938	3	7,935
	Heartbleed	11	0	11
	Infiltration	36	0	36
	PortScan	158,930	126	158,804
	SSH-PATATOR	2,897	0	2,897
	WebAttack BruteForce	1,507	0	1,507
	WebAttack SQL Injection	21	0	21
	WebAttack XSS	652	0	652

Table 3 .

 3 Dataset Split.Feature selection Feature selection is one of the fundamental concepts of machine learning that greatly influences the model performance. Irrelevant or partially relevant features can have a negative impact and lead to a decrease in accuracy. An analysis of the dataset revealed 8 features that are not informative as their value is constant whatever the traffic types. Consequently, features 'Bwd PSH Flags', 'Bwd URG Flags', 'Fwd Avg Bytes/Bulk', 'Fwd Avg Packets/Bulk', 'Fwd Avg Bulk Rate', 'Bwd Avg Bytes/Bulk', 'Bwd Avg Packets/Bulk', 'Bwd Avg Bulk Rate' have been dropped. As we don't want the model to learn when attack occurs, the 'Timestamp' feature cannot be considered as informative.

	Traffic	Training set Cross-validation set Test set
	BENIGN	278,274	139,135 139,135
	Bot	978	489	489
	DDoS	64,012	32,006	32,006
	DoS GoldenEye	5,146	2,573	2,573
	DoS Hulk	115,062	57,531	57,531
	DoS Slowhttptest	2,749	1,374	1,374
	DoS Slowloris	2,898	1,449	1,449
	FTP-PATATOR	3,967	1,983	1,983
	Heartbleed	5	2	2
	Infiltration	18	9	9
	PortScan	79,402	39,701	39,701
	SSH-PATATOR	2,948	1,474	1,474
	WebAttack BruteForce	753	376	376
	WebAttack SQL Injection	10	5	5
	WebAttack XSS	326	163	163
	Total	556,548	278,270 278,270

Table 4 .

 4 Confusion matrix.

			Predicted labels
			Attacks	Normal
	Actual	Attacks	TP	FN
	labels	Normal	FP	TN

Table 5 .

 5 Simplified confusion matrix for variant-1.

				Predictions
			BENIGN Correct attack Other attack
		BENIGN	139,017	-	118
		Bot	5	472	12
		DDoS	1	31,994	11
		DoS GoldenEye	0	2,544	29
	Actual classes	DoS Hulk DoS Slowhttptest DoS Slowloris FTP-PATATOR Heartbleed Infiltration PortScan	0 0 0 0 0 4 0	57,529 1,345 1,434 1,969 2 3 39,675	2 29 15 14 0 2 26
		SSH-PATATOR	0	1,459	15
		WebAttack BruteForce	0	349	27
		WebAttack SQL Injection	0	0	5
		WebAttack XSS	0	0	163

Table 6 .

 6 Simplified confusion matrix for variant-2.

				Predictions
			BENIGN Correct attack Other attack
		BENIGN	138,277	-	858
		Bot	193	296	0
		DDoS	30	31,970	6
		DoS GoldenEye	33	2,537	3
	Actual classes	DoS Hulk DoS Slowhttptest DoS Slowloris FTP-PATATOR Heartbleed Infiltration PortScan	45 18 7 12 0 7 43	57,486 1,348 1,420 1,967 2 0 39,639	0 8 22 4 0 2 19
		SSH-PATATOR	18	1,455	1
		WebAttack BruteForce	340	0	36
		WebAttack SQL Injection	2	0	3
		WebAttack XSS	162	0	1

Table 7 .

 7 Performance Results.

	Metrics	Variant-1 (73 feat.) Variant-2 (70 feat.) Training Test Training Test
	TP	278,256	139,125	276,444	138,225
	FP	257	118	1,630	858
	FN	18	10	1,830	910
	TN	278,017	139,017	276,644	138,277
	T Nrate (%)	99.91	99.92	99.41	99.34
	F Prate (%)	0.09	0.08	0.59	0.62
	Recall (%)	99.99	99.99	99.34	99.35
	Precision (%) 99.91	99.92	99.42	99.38
	Accuracy (%) 99.95	99.95	99.38	99.36
	F1-score (%)	99.95	99.95	99.38	99.36
	MCC	0.9990	0.9991	0.9876	0.9873

Table 8 .

 8 Performance Comparison.

	Paper	Algorithm Accuracy Precision Recall F Prate MCC
			(%)	(%)	(%)	(%)
	our work -variant-1	MLP	99.95	99.92	99.99 0.08 0.9991
		(45 epochs)
	our work -variant-2	MLP	99.36	99.38	99.35 0.62 0.9873
		(715 epochs)
	Sharafaldin et al. [14]	MLP	-	77	83	-	-
		Quadratic	-	97	88	-	-
		Discriminant
		Analysis
		K-Nearest	-	96	96	-	-
		Neighbors
	Jiang et al. [6]	MLP	99.23	99.87	99.60 0.77	-
	Ullah and Mahmoud Decision Tree +	-	100	100	-	-
	[18]	Random Forest
	Ustebay et al. [19]	MLP	91	-	-	-	-

http://cve.mitre.org/


Diogo Ferreira

Carlos Senna
email: cr.senna@av.it.pt

Paulo Salvador

Luís Cortesão

Cristina Pires
email: cristina-j-pires@alticelabs.com

Rui Pedro
email: rui-d-pedro@alticelabs.com

Susana Sargento
email: susana@ua.pt

Root Cause Analysis of Reduced Accessibility in 4G Networks

Keywords: Cellular networks, root cause analysis, machine learning

The increased programmability of communication networks makes them more autonomous, and with the ability to actuate fast in response to users and networks' events. However, it is usually a difficult task to understand the root cause of the network problems, so that autonomous actuation can be provided in advance. This paper analyzes the probable root causes of reduced accessibility in 4G networks, taking into account the information of important Key Performance Indicators (KPIs), and considering their evolution in previous time-frames. This approach resorts to interpretable machine learning models to measure the importance of each KPI in the decrease of the network accessibility in a posterior time-frame. The results show that the main root causes of reduced accessibility in the network are related with the number of failure handovers, the number of phone calls and text messages in the network, the overall download volume and the availability of the cells. However, the main causes of reduced accessibility in each cell are more related to the number of users in each cell and its download volume produced. The results also show the number of PCA components required for a good prediction, as well as the best machine learning approach for this specific use case.

Introduction

In communication networks, root cause analysis of network problems or failures is essential, so that a fast reaction to these failures, or even an anticipation and prevention of these failures can take place. However, usually it is difficult to assess the cause of reduced network accessibility, since it may happen due to a large number of issues, and impacting in a large number of metrics simultaneously. Knowing the causes that lead to such events can help to prematurely detect them. Besides, when such events do happen, it is easier to know where and how to autonomously actuate in the network to mitigate the failure.

With the increased requirements proposed for the 5G networks, e.g. 1 000 000 devices per km 2 , 20 Gbit/s of download peak data rate [START_REF] Kennedy | D2.6 Final report on programme progress and KPIs[END_REF], the new generation of cellular networks promises to handle more traffic than ever before. The incorporation of network slicing, as well as Software-Defined Networking (SDN) and Network Function Virtualization (NFVs) in the 5G architecture, overly increases the management complexity of those networks. With so many metrics to monitor, it is becoming harder to detect the cause of an event due to the complex combinations of various Key Performance Indicators (KPIs). Traditional approaches to detect the root cause of failures, with a knowledge base and a set of rules, is becoming obsolete due to the flexibility of the network. With the advances in machine learning, it is easier to indirectly analyze dependent variables with reduced complexity, but with increased uncertainty.

This work identifies the KPIs that may cause reduced accessibility in 4G networks, using machine learning techniques. Knowing those KPIs helps to create a proactive management of the network, detecting an eventual future drop in network accessibility and having the possibility to avoid it by acting on the network, adjusting the resources that have the most impact on those KPIs. In this work, two different approaches for root cause analysis using machine learning techniques are explained and discussed. The first approach measures the feature importance using internal calculations in the model to determine the importance of each KPI in a reduced accessibility event. However, due to the high number of combinations of the KPIs, it is not feasible to test all possibilities. It is then important to perform feature selection. The second approach proposes a dimensionality reduction algorithm to reduce the number of features and apply the machine learning algorithms. In the evaluation results we present the most important KPIs that are able to predict if the number of E-UTRAN Radio Access Bearer (E-RAB) setup failures is above a specific threshold. Then, we present the most important KPIs that are able to predict if the number of E-RAB establishment failures has high variations, and therefore, are highly correlated to the reduction of the network accessibility.

The remaining of the paper is organized as follows. Section II addresses the relevant related work. Section III discusses how KPIs impact the network accessibility. Section IV presents the proposed approaches for root cause analysis, while section V discusses the results of the approach to root cause detection. Finally, section VI presents the conclusion and the future works.

Related Work

Understanding the root cause of an observed symptom in a complex system has been a major problem for decades. The main question is how to find appropriate real-time mechanisms to determine root causes [START_REF] Solé | Survey on models and techniques for root-cause analysis[END_REF]. Most root cause analysis approaches for network operators is currently based on Bayesian Networks [START_REF] Mfula | Adaptive root cause analysis for self-healing in 5g networks[END_REF]. They have the capability of representing network metrics and events in nodes, and their relations represent the dependencies, along with a conditional probability. To obtain the most probable cause of an error, a probabilistic inference can be done. In [START_REF] Bennacer | Scalable and fast root cause analysis using inter cluster inference[END_REF][START_REF] Bennacer | Selfdiagnosis technique for virtual private networks combining bayesian networks and case-based reasoning[END_REF], the authors argue that a Bayesian network is not suitable for large-scale systems with a large number of components, because the complexity of inference increases exponentially with the number of nodes and dependencies between them. To solve that, they combine the Bayesian network with case-based reasoning techniques to prune the nodes needed to analyze in the network. The results show that the technique used reduces drastically the inference time, as well as the need for human intervention.

In [START_REF] Yan | G-RCA: A generic root cause analysis platform for service quality management in large IP networks[END_REF], a generic framework for root cause analysis in large IP networks was proposed. To determine the root cause of events, two reasoning engines are included: Bayesian inference and rule-based reasoning. The authors discuss that rule-based logic is often preferred over Bayesian inference, because it is easier to configure, it has an easier interpretation of results and it is effective in most applications. However, Bayesian networks are preferred when the root cause is unobservable (no direct evidence can be collected). In our work, the root cause of reduced accessibility is mostly unobservable.

The proposed solution in [START_REF] Chigurupati | Root cause analysis using artificial intelligence[END_REF] computes what are parameters that are most relevant across all different types of failure modes, and use them to build a Bayesian Network to model the cause-effect relationship between the degradation parameters (cause) and failure modes(effect) that occur on the field. Two real life field issues are used as examples to demonstrate the accuracy of the network once it is modelled and built. This paper shows that accurately modelling the hardware system as a Bayesian Network substantially accelerates the process of root cause analysis.

A self-healing method based on network data analysis is proposed to diagnose problems in future 5G radio access networks [8]. The proposed system analyzes the temporal evolution of a plurality of metrics and searches for potential interdependence under the presence of faults. The work in [START_REF] Gonzalez | Root cause analysis of network failures using machine learning and summarization techniques[END_REF] is the only one, that we are aware, that uses the concept of "variable importance" ("feature importance" in our work) to measure how much a feature contributes to predicting an objective variable on a machine learning model. The influence of each variable is then represented in an influence matrix, that represents the influence that each variable has for each event. However, only the Random Forest algorithm is used to create a model. The variable importance is then calculated using the permutation feature importance approach.

Problem Statement

The objective of this work is to understand what are the most important KPIs to anticipate reduced accessibility in a 4G network. A 4G network is chosen, since it is a running network with real data. The data available from 5G networks, because they are still not widely implemented, is scarce and can be unbiased by the users (only users with newer phones with 5G available can use it.)

In this work, a 31-day data set with 4G network KPIs was used. The interval period between each measurement of a KPI is 1 hour, which means that there are 24 values for each KPI for each cell every day. A metric must be chosen to represent reduced accessibility. After that, with time-series analysis, it is possible to calculate the importance of each KPI to the reduced accessibility metric.

The metric used to indicate low accessibility in the network is the number of E-UTRAN Radio Access Bearer (E-RAB) setup failures per hour in the network. The E-RAB setup in a 4G network is a major KPI for accessibility. The E-RAB is a bearer that the User Agents (UEs) need to establish communications in the network. Figure 1 shows the E-RAB setup phase. After the UE has established a connection with the E-UTRAN Node B (eNB), it is needed to setup a context with the Mobility Management Entity (MME), to enable the UE to communicate and send data to the network. When the MME sends a context setup request, it is called an E-RAB setup attempt. After some configuration messages between the UE and the eNB, the context setup response from the eNB to the MME is called an E-RAB setup success. There are more E-RAB setup attempts than E-RAB setup successes. When the network is congested, the difference between the E-RAB setup attempts and the E-RAB setup successes is higher, because some messages after the E-RAB setup attempt are lost due to network problems, such as congestion.

A new accessibility metric is used to measure the accessibility of the network: the number of E-RAB setup failures. If the number of E-RAB setup failures has a high value, the network is congested. The E-RAB setup failure formula is presented in Equation 1, where esf is the number of E-RAB setup failures, esa is the number of E-RAB setup attempts and ess is the number of E-RAB setup successes.

esf (t) = esa(t) -ess(t) (1)
The E-RAB setup failure is depicted in Figure 2. The biggest congestion happened on the 17 th day. The objective of this paper is now to understand which KPIs most contribute to the forecast of this metric.

Methodology for Root Cause Detection

In this section we describe two different approaches for root cause analysis using machine learning techniques. One approach, that measures feature importance using internal calculations in the model, is used to determine the importance of each KPI in a reduced accessibility event. Our second approach proposes a dimensionality reduction algorithm to decrease the number of features (feature selection). There are two disadvantages of analyzing the KPIs' importance with the Pearson correlation coefficient. For each KPI, it is only measured its linear contribution to the number of E-RAB setup failures. Besides, combinations of KPIs that can be important are not being taken into account, because it is assumed that the KPIs are independent of each other.

There are other approaches to measure the importance of input features to an output value that takes into account these considerations. These approaches take advantage of machine learning techniques, and they can be divided into two categories: some approaches take into account the error of the model in a test set to calculate the importance of the input features; other approaches measure the feature importance with an internal calculation (algorithm-specific) of the coefficients associated with each input feature.

Approaches considering the model error: One of the approaches that fall into the first category is the drop column feature importance. In this approach, the importance of a feature is measured by comparing the test error of a model when all features are available as input, with the test error of a model when one feature is dropped for training. The higher the error for the model with one feature dropped, the more importance is given to that feature.

A big disadvantage of this approach is that, for each feature, it is needed to train a new model with that feature dropped, which causes the approach to be inefficient for many features, or for models that take a significant training time. In the permutation feature importance approach [START_REF] Fisher | All models are wrong but many are useful: Variable importance for black-box, proprietary, or misspecified prediction models, using model class reliance[END_REF], a similar approach is used without the need to re-train the model for each feature. Instead of dropping the feature, it is applied random shuffling to the test values of that feature among the various examples, to preserve the distribution of that variable. If the model error is almost unchanged, the feature is not much important for the forecast. However, if the model error increases much, it is a sign that the feature is important for the forecast.

Both previous approaches have some advantages. It is possible to apply them to black-box models, given that the feature importance is measured by the model error. They also take into account all interactions between features, which is an advantage when compared with the correlation tests.

In the permutation feature importance approach, a disadvantage is that the results are dependent on the shuffling of the features. If the tests are repeated, the results may vary.

Approaches considering feature importance using internal calculations: The approaches that measure the feature importance by inspecting the internals of the models are algorithm-dependent. For some algorithms, like neural networks or Support Vector Machines (SVMs), it is impossible to calculate the importance of each feature, due to the non-linear transformations applied. However, for other algorithms such as Logistic Regression5 , Extra Trees, Random Forest6 , Gradient Boosting7 or AdaBoost8 , it is possible to estimate the importance of each feature. For linear regression, the importance of each feature can be measured by the absolute value of the coefficient associated with each input (if all features are within the same scale). For the other four treebased ensemble algorithms (Extra Trees, Random Forest, Gradient Boosting and AdaBoost), the feature importance can be calculated with resort to the mean decrease impurity (or Gini impurity 9 ). The importance of a node j in a decision tree is computed as described in Equation 2, where w j is the weighted number of samples in node j, C j is the impurity of this node, and lef t(j) and right(j) are the respective children nodes.

ni j = w j C j -w lef t(j) C lef t(j) -w right(j) C right(j) (2)
The feature importance of feature i across all nodes is computed as described in Equation 3 10 .

F i i = j:node j splits on feature i ni j j∈all nodes ni j (3)
For this approach, it is important that the features are all normalized within the same scale, and it is recommendable that they are from the same type (continuous/categorical) for better importance estimation.

The advantage of the approaches that measure the feature importance by inspecting the internals of the models is that they do not depend on the test set, only on the model. If the model is accurate and it is not underfitting or overfitting, the feature importance can be calculated more reliably than the previous methods. Otherwise, the feature importance will be highly biased. The biggest challenge is to create accurate models that do not overfit the train set. This second category of approaches (measure feature importance with an internal calculation algorithm-specific) will be used to measure the importance of the KPIs.

Feature Selection

If all KPIs are included as features, the performance of the model will be degraded, because some features are uncorrelated with the output and do not contribute to the output classification. It is then important to perform feature selection. The ideal scenario is to test all combinations of features in the input, and determine the best features by the test error. However, it is not feasible to test all combinations, due to their high number.

The approach chosen was to use a dimensionality reduction algorithm to reduce the number of features. Principal Component Analysis [START_REF] Pearson | Liii. on lines and planes of closest fit to systems of points in space[END_REF] will be used as the algorithm to perform dimensionality reduction. The five algorithms chosen to train the model and to calculate the feature importance were the following: Logistic Regression, Extra Trees, Random Forest, Gradient Boosting and Ad-aBoost. The feature importance of each model will calculate the importance of each PCA component. Each component importance is then multiplied by the PCA coefficients, to get the KPI importance for each component. Finally, all the importances of the same KPI are added, as described in Equation 4, where f i j is the feature importance of the j PCA component and pca coef f icient(j, i) is the PCA coefficient i for the component j, where i is the number of a KPI and j is a PCA component.

kpi i i = n pca components j=0 f i j * pca coef f icient(j, i) (4)
To get the best possible model, the number of PCA components cannot be too small, because relevant features can be lost; however, the number cannot be too big either, with the risk of creating overfitted models. The number of PCA components tested will vary from 1 to the number of KPIs, for the five algorithms. The model with a lower test error will be used to calculate the KPI importance.

Defining the input and the output of the algorithms

The input values will be based on the KPIs. There will be two types of input values: normalized values and normalized variations. For each KPI, the normalized value of the previous hour will be used as input. The number of lags could be increased besides one hour, but in this test it is considered that the low accessibility indicators appear at most one hour before the congestion. For each KPI, the normalized variation will be calculated according to the Equation 5.

The normalized variations are added as features because the network accessibility can depend, not only on the previous values, but also on sudden variations of other KPIs.

normalized variation(t) = value(t -1) -value(t -2) value(t -2) (5)
Two types of classification problems will be used to calculate the importance of the KPIs. It is important to understand which KPIs are more important when forecasting the possibility of a low accessibility event in a network. It is also important to understand which KPIs are more important to forecast sudden increases and decreases in the network accessibility. For each case, a different output value will be calculated. For the first case, it will be set a threshold in the 90 th percentile of the data, with all the values above the threshold being classified as one, and all other values as zero, turning the problem into a binary classification problem. For the second case, the output will be classified as one if the absolute difference between two consecutive values is higher than the 90 th percentile of the data, and zero otherwise. Doing that, it is possible to analyze which KPIs are most important for classifying low accessibility events and also for forecasting bigger increases and decreases in data, which can be important for resource allocation.

The two classification problems will be applied in two different ways, according to the data split. First, the tests will be done with aggregated data. The network KPIs will be added and will be taken into account in the whole network. In this way, it is possible to forecast low accessibility of the network. Another way of performing the tests is to split the data per cell. In this approach, the data will not be aggregated, and 75% of all cells in the network will be used to train, with the other 25% cells to test. The threshold for the tasks will be defined using the specific values of each cell (each cell will have a different threshold, based on its values). With this approach, it is built a model that is capable of detecting low accessibility per cell. This case is expected to perform worse than the aggregated network, because forecasting low accessibility per cell is harder than forecasting low accessibility in the network. However, for a network operator it is very important to forecast low accessibility per cell for various reasons. For a cell in a crowded region, it can be made management adjustments to avoid the low accessibility of the cell, such as the installation of another temporary cell, or the allocation of resources for that cell. Besides, it can be made a time-series analysis about the future accessibility of the cells in a region, for expansion purposes.

Performance metric

For both classification problems, a performance metric must be used. Since in the problems previously described the number of positive samples is lower than the number of negative samples, the accuracy performance metric would give similar cost to the false negative and false positive errors. The performance metric used will be the F1-score (Equation 6). The F1-score is the harmonic mean of precision (Equation 7) and recall (Equation 8), and it encompasses the False Negative and False Positive errors, weighted according to the number of samples of each class.

F 1 = 2 * precision * recall precision + recall (6)
precision = true positives true positives/f alse positives [START_REF] Mfula | Adaptive root cause analysis for self-healing in 5g networks[END_REF] recall = true positives true positives/f alse negatives (8)

Root Cause Detection Results

Traffic monitoring is essential to provide a good quality do the services in communication networks. Network conditions such as bandwidth, packet loss, delay and jitter are important for traffic engineering to track the quality needs of applications. Therefore, it is important not only to measure the network conditions, but also to analyze them to understand the causes of reduced accessibility in cellular networks. In the following, we show how machine learning techniques can help to understand which network KPIs can indicate a low accessibility event that will happen in the future.

Aggregated network tests

In this subsection, the results for the aggregated network tests will be presented.

As explained before, two scenarios are proposed. First, it will be presented (i) the most important KPIs for predicting if the number of E-RAB setup failures is above a threshold. Second, it will be presented (ii) the most important KPIs for predicting if the number of E-RAB establishment failures has high variations.

In Figure 3 it is shown that the best model for scenario (i) is the Extra Trees algorithm, achieving an F1-score of 86.6%, with 23 PCA components. With a higher number of PCA components, the performance of most of the algorithms starts to deteriorate.

Table 1 shows the ten KPIs that are considered more important for predicting if the number of E-RAB setup failures is above a threshold (i). The handover failure is considered to be the most important KPI, both inter-and intra-frequency. The third KPI is the Circuit Switched FallBack (CSFB) preparation success. The CSFB is a technology to create circuit-switched calls over a 4G network that does not support LTE voice call standard (VoLTE), which has to fall back on the 3G network. This indicates that the number of phone calls and SMS messages has a high impact on the network accessibility. The Packet Data Convergence Protocol (PDCP) download transmission time, the variation of the inter-frequency handovers and the cell available time are also important KPIs to predict variations in the number of E-RAB setup failures. Finally, the ERAB normal release, handovers intrafrequency success and attempt, and upload traffic volume have also a significant importance. 1. Ten KPIs that were considered more important for the scenario (i) in the aggregated network.

The best model for scenario (ii) is achieved with the AdaBoost algorithm, achieving an F1-score of 40.0%, with 24 PCA components. The F1-score is lower than in scenario (i) because the task of predicting variations is harder than the one of predicting if the value is above a threshold. The number of PCA components is almost the same as in scenario (i); however in Figure 4 it can be seen that the algorithms' F1-score is very unstable, and that it is hard to build a model to predict accurately the variations of the number of E-RAB establishment failures in the aggregated network.

Table 4 shows the ten KPIs that are considered more important for the scenario (ii). Like in the previous scenario, RRC setup success ratio and E-RAB setup success rate are the most important KPIs for the output. In this task, other KPIs have also similar importance, such as the variation of the cell availability, the average connected subscribers and the average number of radio bearers. A radio bearer is a connection between the eNB and the UE at layer 2, and defines the communication configurations for upper layers. 4. Ten KPIs that were considered more important for the scenario (ii) in the individual cells tests.

Discussion

Aggregated network tests: From the results of both aggregated network tests, it can be concluded that most KPIs that cause the number of E-RAB setup failures in a network to be above a threshold are the same that cause it to have high variations. Those KPIs are the number of failed handovers (intraand inter-frequency), the CSFB preparation success (number of phone calls and SMSs in the network), the PDCP download volume and the cell availability. The maximum number of active subscribers (downloading subscribers and overall subscribers) also causes the number of E-RAB setup failures to vary.

Interpreting the KPIs, the results achieved are according to the intuition about lower network accessibility. When the number of failure handovers is high, the cells are crowded with user sessions and cannot accept any more sessions, which leads to lower network accessibility in the next hour. The high number of CSFB preparation success shows that there is a clear relationship between the high number of phone calls and SMS messages in the network with its lower accessibility. The KPIs of PDCP download volume and the maximum number of active subscribers also show that the number of active subscribers and their download volume influence the network accessibility (more than the number of connected subscribers). Finally, the cell availability indicates that, if many cells are unavailable in the current hour, it is likely that the network accessibility will be lower in the next hour.

Individual cells tests:

The results of the forecasts of low network accessibility in individual cells had higher error than the results with the aggregated network. In the tests done, the best models needed more data than the aggregated network models to achieve the best result, because more features were needed to be able to generalize the predictions for different cells.

The results show that, just like in the aggregated network tests, the most important KPIs for forecasting the number of E-RAB setup failures in a cell to be above a threshold are the same that cause it to have high variations. However, the most important KPIs that cause lower accessibility in a network are different from the KPIs that cause lower accessibility in a cell. The two most important KPIs are the RRC and the E-RAB setup success rate. These KPIs cannot be understood as the cause for lower network accessibility, but as a consequence. Because they are intrinsically related to the E-RAB setup failure, being themselves accessibility metrics, they can be understood as an indicator of the high autocorrelation between consecutive hours. These results show that the network accessibility per cell is highly dependent on the network accessibility of that cell in the previous hour. If a cell has lower network accessibility in an hour, it is likely that the network accessibility in the next hour for that cell is still low, and vice-versa. It is essential for a network operator to monitor the right KPIs for the different tasks: forecast lower accessibility in a network or forecast lower accessibility per cell.

As opposed to the results for the aggregated network, the important KPIs for low network accessibility are not related with the CSFB preparation success, or with any of the handover metrics. The results by cell show that, besides the RRC and the E-RAB setup success ratio, the most important KPIs are counters related to the number of users in a cell and its utilization: maximum number of connected subscribers, average number of connected subscribers, average number of active subscribers download data, variation of the maximum number connected subscribers, variation of the maximum number of active download subscribers, average number of radio bearers or variation of the average number of radio bearers. It is expected that, as these KPIs have higher values, the accessibility of a cell decreases.

Conclusion

Understanding the causes of events in a network, such as low accessibility, helps the network operators to forecast and avoid them to happen, by adjusting network resources that influence their causes. In this work, the goal was to determine the causes of reduced network accessibility in 4G networks, using only historic data.

Two different analysis were made. Besides analyzing the causes of reduced accessibility in the whole network, it was also analyzed the causes of reduced accessibility for each cell. The results showed that the causes of reduced accessibility for each analysis are very different. While for the overall network, the KPIs that most influence the accessibility are the number of failure handovers, the number of phone calls and SMSs in the network, the overall download volume and the availability of the cells, the KPIs that most influence the accessibility of each cell are related with the number of users in a cell and its download volume. For a network operator, it is important to know if it is important to monitor low accessibility in a cell, in a network, or in both, to make the right measurements in the network.

As future work, the next step will be to detect the patterns of those KPIs that indicate future low accessibility, to be able to predict it and adapt the network to prevent it to happen. For example, if the network operator knows that the network accessibility will be lower in the next hour when the number of handover failures intra-frequency and the maximum connected users both exceed a threshold, he can take proactive measures to adapt the network and avoid the low accessibility.

Fig. 1 .

 1 Fig. 1. Sequence diagram indicating the E-RAB setup phase.

Fig. 2 .

 2 Fig. 2. Number of E-RAB setup failures.

4. 1

 1 Approaches to measure the feature importance A simple test to check if any KPI can accurately forecast the number of E-RAB setup failures is to calculate the Pearson correlation coefficient 4 between the number of E-RAB setup failures and all other KPIs. Since the goal is to understand which KPIs can forecast low accessibility before the number of E-RAB setup failures increases, the KPI values will be shifted (lagged) behind the number of E-RAB setup failures by one hour (the values are sampled hourly; one hour is the minimum time interval).

Fig. 3 .

 3 Fig. 3. F1-score with different algorithms varying the number of PCA components for the task (i) in the aggregated network.

https://www.statisticssolutions.com/pearsons-correlation-coefficient/

https://www.statisticssolutions.com/what-is-logistic-regression/

https://towardsdatascience.com/an-intuitive-explanation-of-random-forest-andextra-trees-classifiers-8507ac21d54b

[START_REF] Mfula | Adaptive root cause analysis for self-healing in 5g networks[END_REF] https://towardsdatascience.com/understanding-gradient-boosting-machines-

9be756fe76ab 8 https://towardsdatascience.com/understanding-

adaboost-2f94f22d5bfe 9 https://victorzhou.

com/blog/gini-impurity/ 10 https://stats.stackexchange.com/questions/311488/summing-feature-importancein-scikit-learn-for-a-set-of-features/

Acknowledgments This work is supported by the European Regional Development Fund (FEDER), through the Regional Operational Programme of Lisbon (POR LISBOA 2020) and the Competitiveness and Internationalization Operational Programme (COMPETE 2020) of the Portugal 2020 framework [Project 5G with Nr. 024539 (POCI-01-0247-FEDER-024539)], and by FCT/MEC through national funds and when applicable co-funded by FEDER -PT2020 partnership agreement under the project UID/EEA/50008/2019.

Table 2 shows the ten KPIs that are considered more important for predicting if the number of E-RAB establishment failures has high variations (ii). The handover failures are still important, but they are not the most important KPI. In this case, the CSFB preparation success is the most important KPI to predict the variations in the number of E-RAB setup failures. Similar to the task (i), PDCP download transmission time, the variation of the inter-frequency handovers, the number of active subscribers, download and upload ones, and the cell available time are also important KPIs. 2. Ten KPIs that were considered more important for the scenario (ii) in the aggregated network.

Individual cells tests

In this subsection, the results for the individual cells will be presented. In this case, 75% of the cells in the network will be used for training and the other 25% for testing. Just like in the previous subsection, first it will be presented (i) the most important KPIs for predicting if the number of E-RAB establishment failures is above a threshold, and then it will be presented (ii) the most important KPIs for predicting if the number of E-RAB setup failures has high variations.

The best model for scenario (i) achieved an F1-score of 30.79%, using the Gradient Boosting algorithm, with 66 PCA components. For the cell prediction, more information is needed to obtain the best model when compared with the previous subsection. Figure 5 shows the F1-score varying with the number of components and the different algorithms. In the PCA component 62 there is an increase in the F1-score of all algorithms. Table 3 shows the ten KPIs that are considered more important for the task (i). Two KPIs have much more importance than all others: Radio Resource Control (RRC) setup success ratio ( RRC setup success RRC setup attempt ) and E-RAB setup success ratio ( ERAB setup success ERAB setup attempt ), since they are directly correlated with the network accessibility.

The best model for scenario (ii) has an F1-score of 20.39%, using the Ad-aBoost algorithm with 68 PCA components. The F1-score for different PCA components is similar to the scenario (i), where the F1-score improved its performance significantly after 60 PCA components (Figure 6).