Tobias Sundqvist

Monowar H Bhuyan
email: monowar@cs.umu.se

Johan Forsman
email: johan.forsman@tietoevry.com

Erik Elmroth
email: elmroth@cs.umu.se

Boosted Ensemble Learning for Anomaly Detection in 5G RAN

Keywords: Anomaly detection, AdaBoost, LSTM, Radio Access Network (RAN), 5G, System logs, Functional area

The emerging 5G networks promises more throughput, faster, and more reliable services, but as the network complexity and dynamics increases, it becomes more difficult to troubleshoot the systems. Vendors are spending a lot of time and effort on early anomaly detection in their development cycle and majority of the time is spent on manually analyzing system logs. While main research in anomaly detection uses performance metrics, anomaly detection using functional behaviour is still lacking in depth analysis. In this paper we show how a boosted ensemble of Long Short Term Memory classifiers can detect anomalies in the 5G Radio Access Network system logs. Acquiring system logs from a live 5G network is difficult due to confidentiality issues, live network disturbance, and problems to repeat scenarios. Therefore, we perform our evaluation on logs from a 5G test bed that simulate realistic traffic in a city. Our ensemble learns the functional behaviour of an application by training on logs from normal execution time. It can then detect deviations from normal behaviour and also be retrained on false positive cases found during validation. Anomaly detection in RAN shows that our ensemble called BoostLog, outperforms a single LSTM classifier and further testing on HDFS logs confirms that BoostLog also can be used in other domains. Instead of using domain experts to manually analyse system logs, BoostLog can be used by less experienced trouble shooters to automatically detect anomalies faster and more reliable.

Introduction

The 5G Radio Access Network (RAN) is a complex large-scale system where parts are both virtualized and distributed. Detection and diagnostics of system issues (e.g., faults, outages, degradation) are of great importance due to enduser experience and brand reputation. A key to success for companies is to find anomalies fast and early in the development cycle [START_REF] Damm | Early and Cost-Effective Software Fault Detection : Measurement and Implementation in an Industrial Setting[END_REF], this reduces the cost and time to market when new features are developed. Finding a way to speed up analytics and make it easier to analyze RAN traffic would therefore be very useful since it would both make the system more reliable and also earn money for the company. When it comes to anomaly detection in RAN, the main research has focused on using performance counters, e.g., network traffic and Key Performance Indicators [START_REF] Fernández Maimó | A self-adaptive deep learning-based system for anomaly detection in 5G networks[END_REF], [START_REF] Iyer | Automating diagnosis of cellular radio access network problems[END_REF], [START_REF] Szilagyi | An automatic detection and diagnosis framework for mobile communication systems[END_REF], [START_REF] Al Mamun | Anomaly detection and classification in cellular networks using automatic labeling technique for applying supervised learning[END_REF]. A foreseen area is the functional domain where system logs can be used to determine the behaviour of the applications. System logs are today mainly used to troubleshoot systems when testing or when live traffic fails, in those cases it is very time consuming to manually analyze logs since they are huge.

In this paper, we use machine learning to develop a system that will allow trouble shooters in the Telecom area to find anomalies faster and with less system knowledge. By tracking and separating call sessions, we turn system logs into sequential time series that is suitable for anomaly detection. A model that has shown to be very useful in predicting time series is the Long Short Term Memory (LSTM) model [START_REF] Hochreiter | Long short-term memory[END_REF], [START_REF] Sun | Adaboost-lstm ensemble learning for financial time series forecasting[END_REF], [START_REF] Karevan | Transductive lstm for time-series prediction: An application to weather forecasting[END_REF], [START_REF] Bian | Internet traffic forecasting using boosting lstm method[END_REF]. One advantage of the LSTM model is that it can learn and remember long term dependencies which is useful when long sequences of events are analysed. By training on sequences of events in system logs, the LSTM model can also grasp the application behaviour without having any deep knowledge of the system. Recent research use the predictions made by the LSTM model to detect anomalies [START_REF] Malhotra | Lstm-based encoder-decoder for multi-sensor anomaly detection[END_REF], [START_REF] Chalapathy | Deep learning for anomaly detection: A survey[END_REF], [START_REF] Chniti | E-commerce time series forecasting using lstm neural network and support vector regression[END_REF]. The common idea in these articles is that a LSTM model is used to detect single deviations from normal behaviour. The problem with these approaches are that they rely on single deviations and cannot detect multiple small deviations from the normal behaviour. Another aspect is that these anomaly detection methods are performed on numerical data, when it comes to analysing functional behaviour using system logs very little has been done using machine learning. In a recent survey of anomaly detection in system logs [START_REF] Ajith | A survey on anomaly detection methods for system log data[END_REF], the most matured method DeepLog [START_REF] Du | Deeplog: Anomaly detection and diagnosis from system logs through deep learning[END_REF], also the LSTM model and it shows the potential of the technique. In large scale, real time systems such as RAN, there are many concurrent threads that causes events to occur in different order from time to time. This makes it very difficult for a single LSTM model to predict the next state and as in the DeepLog case, one LSTM models are trained for each unique sequence. To overcome the real time challenges we propose an adaptive boosting ensemble [START_REF] Polikar | Ensemble based systems in decision making[END_REF], that uses LSTM models in a novel way to detect both single anomalies but also multiple deviations from the normal functional behaviour. The main contributions of this work are as follows:

-The design of an adaptive boosting ensemble, BoostLog, which outperforms single LSTM models and automatically detects anomalies in system logs. -A novel, single LSTM classifier that can detect both multiple small deviations and also single larger ones in system logs. -Evaluation of BoostLog and single LSTM classifier using real-time and benchmark data including the 5G test bed and HDFS log set.

This paper is organized as follows. Section 2 presents how machine learning can be used to find anomalies in system logs and why this is important in RAN. In Section 3 we propose our method and in section 4, we outline how our experiments were carried out. Finally our conclusions and how to proceed in future work are presented in section 5.

Background

Recurrent Neural Network

Instead of using predefined log rules as in analysis tools such as Logsurfer [START_REF] Prewett | Analyzing cluster log files using logsurfer[END_REF], Swatch [START_REF] Hansen | Automated system monitoring and notification with swatch[END_REF], or SEC [START_REF] Rouillard | Real-time log file analysis using the simple event correlator[END_REF], we want a model that can use a system log to learn the normal behavior without deep knowledge about the system. The state events in the log, are typically sequential and one model that has been proven to be useful for such time series is the Recurrent Neural Network (RNN) [START_REF] Zhang | Automated it system failure prediction: A deep learning approach[END_REF], [START_REF] Du | Deeplog: Anomaly detection and diagnosis from system logs through deep learning[END_REF]. RNN has the advantage of possessing an internal memory and can remember its input. This has been found very useful in areas such as speech, text, financial data, audio, video, weather and much more. An extension of the RNN is the Long Short Term Memory (LSTM) model [START_REF] Hochreiter | Long short-term memory[END_REF], which expands the memory of the RNN to be able to remember the inputs over a long period of time. Just like RNN, the LSTM also uses a chain of repeating modules, in which each module takes an input, produces an output, and also forward some information to the next module. Instead of having just one neural network in each module as in the normal RNN case, the LSTM uses four neural networks that interacts with each other. The main idea of the LSTM is to allow each module to select its input using an input gate, delete the information that is not important using a forget gate, and finally let the information in each module control the output using an output gate, see Fig. 1.

tanh ¡ ¡ X t ht tanh x x + ¡ x X t+1 ht+1 X t-1 ht-1
Fig. 1: LSTM architecture, X and h are input respectively output for a time step

Detecting anomalies

The first part in detecting an anomaly using a LSTM model is to make a prediction of the next state, given previous data. The prediction is then compared with the state that followed and if they differs it can be classified as an anomaly, see also Section 3.1. The normal procedure when using time series to make predictions of future events is to re-frame the events into a supervised learning problem. By using previous events in the sequence, the next time step can be predicted and compared with the next event. The number of time steps to look back are very important when the data is sequential [START_REF] Du | Deeplog: Anomaly detection and diagnosis from system logs through deep learning[END_REF]. If several sequence chains are similar, more time steps are needed in order to predict the correct outcome and if too many time steps are used it will be difficult for the model to separate the sequence chains form each other. In Fig. 2, the importance of using many time steps, when predicting the events can be seen. In case A, only one time step is used and it is impossible to tell if 7, 8, or 9 is going to follow after event 6. In case B, two time steps are used and it is possible to tell the difference between 3-6-7 and 2-6-7 but it is still impossible to predict the 2-6-9 sequence. Only after looking at three time steps as in case C, we can tell the difference between all three sequences.

RAN metrics

The 5G generation of wireless telecom networks are designed to increase the speed, throughput, and reliability. An important part of the 5G network is RAN. It consists mainly of base stations and antennas and its task is to make sure that the User Equipment (UE) can communicate with the core network, see Fig. 3. In RAN there are three categories of metrics that are interesting to consider:

-Configurations, describes how the hardware and software are configured.

-Performance measurements, metrics that describes the performance of the system, often during a certain time period. Some performance measurements are used as quality indicators of the system and are often referred as Key Performance Indicators. -System logs, contains time series of application specific events.

System logs contains most useful information when it comes to anomaly detection and root cause analytics. A faulty configuration can cause software to malfunction but logs are needed to find the configuration issues. Performance measurements can also hint that something is faulty during a certain time period but the logs are often needed to locate the fault. System logs are used to understand the behavior of the application, the vendor decides how often and what to log and it can contain start up sequences, important data, special events or alarms, etc. The best way to locate an anomaly in RAN is to use all three categories of data but we will limit our work in this article to look at how time series in system logs can be used to find the anomalies.

3 BoostLog: Proposed Method

LSTM classifier

RAN system logs contains a mix of events from thousands of session calls that occurs in parallel of each other. Such concurrency causes a random distributions of sequence events and makes it impossible to learn the behavior for any model. As in [START_REF] Yu | Cloudseer: Workflow monitoring of cloud infrastructures via interleaved logs[END_REF] a better approach is to separate work flows or in this case sessions, into separate sequences and let the model train on each sequence instead of the whole list of events. In Fig. 4 it can be seen how metrics are collected from each Base Station and stored in the metric collector. The system log are then filtered and separated into UE sessions before the ensemble of LSTMs uses them for training or prediction. Even if it helps to separate the call sessions, all sessions will not look the same if sequence of events are analyzed. This is due to the nature of real time systems where several applications work in parallel to perform a task and depending on the load of applications, some events may occur in different order from time to time. This means that even if a LSTM model is trained on a large data set there will most likely be sequences that the model has not seen before. But even if the combined variations creates a sequence that has not been seen before, the single variations might be seen in other sequences that are used for training. This will also make the LSTM model more forgiving and sequences not seen before can be predicted normally if they only contains those small variations. Since sequences vary a lot we cannot use the most probable state from the LSTM model as the prediction, this would make the model to always choose the state that it has seen most times during training. Instead we let the model to output a vector with predictions for all states and look at the probability for the state it is trying to predict.

Fig. 5 shows the output probabilities predicted by a LSTM model after it has trained on several different sequences. The probability for a state to occur is written above the state (1.0 means 100% probability), states receiving 0 probability are not shown. To determine if a particular sequence contains any anomaly or not, the probability for each state is first compared with the highest probable state:

p(act) i p(max) i = c i (1)
p(act) i is the probability for the state, p(max) i is the maximum probability among all states predicted, and c i is the comparative value. Then all state comparatives is multiplied with each other resulting in a comparative value for the whole sequences:

c 1 • c 2 • ... • c n = C (2)
Where C is the product of all comparatives for the whole sequence, we call this the sequence error. The top sequence in Fig. 5 has the highest probability and would produce a product of 1•1•1•... = 1. If a sequence shown in the bottom of the Fig. 5 occurs then it would get a product of 1•1•(0.1/0.7)•(0.1/0.7)•(0.1/0.7)•1•1 = 0.0029. The sequences can then be classified as anomaly if the sequence error is below a certain limit, this is defined as the error threshold. The advantage of using an error threshold for the whole sequence is that it will both capture single events that are very unlikely and also sequences that has several minor deviations from the normal behaviour.

Filtering data

The RAN data used in this article contains several thousands of sequences for calls between users. Many of these sequences are similar but small differences in order of events causes a large number of unique sequences. If a LSTM model would train on the whole data it would learn to predict the most probable state sequences. Sequences that occur very seldom would be treated as an anomaly since they deviate from the normal behaviour. In order to address this issue the data is filtered so that the LSTM model only train on sequences it has not seen before.

AdaBoost ensemble of LSTM classifiers

As the single LSTM model is trained on more and more unique sequences, the probability for a certain event to occur will decrease since there are more events for the LSTM model to choose between. In these cases it would be more beneficial to use several models that trained on sequences that are similar to each other. Sequences can be compared in many ways using the length of the sequence, number of matching events, how close the events are to each other, the time difference between them, etc. Even if we divided the sequences based on these parameters, the LSTM model would perhaps not treat the sequences as similar anyway. The best would be to let the LSTM model determine if the sequences are equal or not and divide the sequences into groups and then retrain the models. This is similar of how the AdaBoost ensemble works [START_REF] Freund | A decision-theoretic generalization of on-line learning and an application to boosting[END_REF], where multiple weak learners are trained in sequence. The first learner trains and classifies the data used for training and then the next learner trains more on the data that were misclassified by the previous and so on until all learners have trained and classified the data. All the learners can then be used to classify new data by using a voting among the learners. The better a learner has been during training the more importance is given to the vote. AdaBoost in combination with LSTM models has also recently proven to be useful for predicting numerical time series [START_REF] Sun | Adaboost-lstm ensemble learning for financial time series forecasting[END_REF], [START_REF] Xiao | Short and mid-term sea surface temperature prediction using time-series satellite data and lstm-adaboost combination approach[END_REF], [START_REF] Bian | Internet traffic forecasting using boosting lstm method[END_REF]. The procedure we propose to train the AdaBoost LSTM ensemble for our purpose are the following:

1. Initialize an equal weight to all sequences used for training:

ω t (i) = 1 N (3)
2. The weights are used to select which sequences to train on, the higher number the more likely it is that the sequences are chosen for training. 3. A new LSTM model trains on the selected sequences. 4. The same model then predicts if there is an anomaly in any of the sequences. 5. If the accuracy is less then 0.55 then step 3-5 is repeated until the accuracy is larger than 0.55. If the accuracy does not reach 0.55 within 10 times the best model is chosen and we continue to step 6. 6. When all sequences is classified a value α is calculated based on the accuracy of the LSTM model, (t is the iteration of step 2-7):

α t = 1 2 ln 1 -t t ( 4
)
where t is the sum of the weighted error for the prediction:

t = m i=1 ω t (i)[y i = h(x i )] (5)
7. α is the used to increase the weights for sequences classified as anomaly and decrease weights if classified as normal:

ω t+1 (i) = ω t e -α t h t (x)y(x) (6)
ω(i) is then normalised so that the sum of all ω(i) is 1:

ω t (i) = ω t (i) N i=1 ω t (i) (7)
8. step 2-7 are repeated for all models selected to be part of the ensemble. 9. To detect anomalies, all models classifies a sequence and a voting procedure is used to determine the outcome. For each model, α is used to tell the importance of a vote, if the model had high accuracy during the training the vote will be more important.

During training of the ensemble it is also important that the training sequences does not contains any anomalies. If anomalies are present the LSTM models would also learn the anomaly behaviour and could not differ between normal and abnormal behaviour.

Anomaly feedback loop

During the continuous product development cycle, the software is constantly updated with new features and the behavior might change slightly for each software update. The update may cause the LSTM model to predict normal sequences as abnormal, this might also happen if the data contains a lot of sequences that has not been seen before (discussed in Section 3.1). The false positive (FP) sequences, can then easily be used to train the LSTM model and its weights will be adjusted during the training so that it recognizes the FP sequences. By analyzing all sequences that are classified as anomaly the FP cases can be separated and sent into the LSTM model for extra training, see Fig. 4.

Evaluation

RAN data

RAN system logs can contain data that are sensitive for users and vendors and are hard to obtain. Instead of using logs from commercial live systems we used a 5G test bed created by TietoEvry, which can simulate a one day scenario for a small city with 10 000 inhabitants. The advantage in using a test bed is that we can control the scenarios, repeat them, and also choose what to include in the logs without disturbing any live system. Two data sets containing 7458 respectively 7456 session calls were collected from the one day scenario in which all traffic were normal. By letting the test bed skip or add extra events in the system log, a third data set containing 26 individual anomalies were created. The anomalies were: missing event, extra event, impossible state transitions, events not seen before, multiple events missing, and multiple extra events. The session calls in the logs were separated into event sequences and the key values for each event were extracted and encoded to be suitable for the machine learning problem.

To address the imbalanced data we discussed in Section 3.2 the two normal data sets were filtered to create two lists with unique sequences. The third data set was filtered to only contain the anomaly sequences. The three data sets will later be referred to as training (173 seq), test (45 seq) and error data set (26 seq).

Benchmark data

To evaluate how BoostLog performs in other domains than RAN and also to compare our findings with recent research, a second data set was retrieved from a Hadoop Distributed File System (HDFS), in which 200 Amazon's EC2 nodes been running Hadoop-based map-reduce jobs. The block sequences in the data have been labeled by domain experts as anomaly or normal and contains 11 175 630 events in 575 061 sequence chains. This data is described more in detail in [START_REF] Zhu | Tools and benchmarks for automated log parsing[END_REF] and has been used for research in anomaly detection by [START_REF] Xu | Detecting large-scale system problems by mining console logs[END_REF] and [START_REF] Du | Deeplog: Anomaly detection and diagnosis from system logs through deep learning[END_REF]. In similar way as for the RAN data set only the unique sequences were extracted and split into train (12 406 seq), test (5317 seq) and anomaly data set (4375 seq). This data set is much larger than the RAN data set and in order to reduce time to tune the model and make it similar to the RAN log we only used 10 % of the normal data while we kept all the anomalies.

Performance measures

The main idea of this research was to find new ways to automatically analyze the system logs in order to find anomalies faster without being expert on the domain. One important aspect is also to be able to trust the model, for trouble shooters in RAN it are very important to not classify normal data as an anomaly. If the model keeps classifying normal data as faulty then the same thing as happens in the legendary Aesop's fable The Boy Who Cried Wolf will occur, the users will stop relying on the model. To make the model useful and trustworthy it need to fulfill two things, it should be able to find many anomalies and seldom classify normal behavior as faulty.

The LSTM model

The single LSTM model used has 2 hidden layers with about 100 neurons, some models had fully connected layers and some layers used dropout. In the AdaBoost ensemble it is more beneficial to use many different classifiers and 15 LSTM models with a range from 2-3 hidden layers and 32-200 neurons were used. For all LSTM models, the keras [START_REF] Chollet | Keras[END_REF] back-end with the implementation of the LSTM layer was used.

Parameter tuning

Single LSTM classifier. There are two important parameters to choose for the single LSTM classifier, firstly what error threshold that should be used and secondly how many states it should look back on in order to predict the next state.

To see the impact of the error threshold used for classifying the sequences we choose to set the look back to 4 and test error thresholds in range or 1 to 1e -9. The model is first trained on the training data set and then the accuracy is measured on the training, test and error data sets, the result can be seen in Fig. 6a. When the error threshold is high all sequences will be classified as anomalies and if the error threshold is very low it will miss some anomalies but it will classify the normal sequences more correctly. It seems that the threshold can be lowered down to 1e -5 before the true positive (TP) rate starts to drop and we still get a quite high accuracy for the normal sequences. This can also be seen in the precision-recall curve in Fig. 6a.

Next parameter to tune is the number of look backs, ranging the look backs from 1 to 12, we tested both the error threshold at 1e -5 where the true positive and true negative (TN) rate was high and also at 1e -9 were TN rate was highest. For 1e -9 the accuracy for the normal sequences stayed stable at the same level as in Fig. 6a but the accuracy kept decreasing with the increasing amount of look backs, see Fig. 6b. The drop in accuracy is explained by the increasing certainty of the model when the number of states to look back on are increased, a higher threshold is needed to compensate for the more precise model but then the accuracy would also drop for the normal sequences. The BoostLog. To start with, we used the same error threshold and look back parameter in BoostLog as for the single LSTM classifier. To see how the AdaBoost classifier improves with the amount of learners, the accuracy was calculated for the three data sets, see Fig. 7a. As we can see the accuracy are getting stable at 10 learners. When comparing with the single LSTM model we can also see that the accuracy is better for both test and error sequences. This is very promising since the accuracy improved for those sequences not seen yet. Normally it is better to use weak learners that achieve an accuracy just above 0.5 in an AdaBoost ensemble. As expected the ensemble improved even more when increasing the error threshold and decreasing number of look back to get learners that were not equally good as the best single LSTM model. Final result can be see in Table . 1.

Training using feedback loop

Both the single LSTM model and BoostLog classified some of the normal sequences as an anomaly. These anomalies are considered false positive (FP) and as we discussed in Section 4.3 and 3.4, we want to minimize the FP rate to make the model more trustworthy and useful. The models classifies these sequences as anomalies since they have not seen these variations before or at least very seldom. When retraining the single LSTM classifier with the normal sequences not seen before, the overall accuracy did not change, in mean 13±2% of the normal sequences were still classified as anomaly. As we discussed earlier in When the same procedure was performed for the BoostLog the accuracy stayed the same for the anomaly sequences and increased from 84% to 100% for the sequences it had not seen before, see Fig. 7a.

Among the false negative cases, there were only two sequences that both the single LSTM and BoostLog classified as normal and both these cases missed one state in the sequence. These anomalies are difficult for the LSTM classifier to detect since the single missed state will not cause a very large drop in sequence error. If the LSTM predicts the probability for time step t+1 it will also know what state that is likely to occur in time step t+2. The probability for the state to occur in t+2 will be slightly higher than other states and if the state for time step t+1 is missing, the second highest probability is likely to be the state for t+2.

Evaluation

In this work we have limit our models to only look at one feature from the system logs. The main focus have been to investigate how the anomaly detection can be improved in system logs using an ensemble of LSTM models.

To do this we have compared the single LSTM model with the new BoostLog ensemble. Precision, Recall, and F-score was calculated for each model before and after training on RAN testbed, see Table . 1, the models are called feedback model when training has been performed on normal sequences not seen before. The single LSTM model improved in detecting anomalies (the recall value), but instead it also classified more normal sequences as anomalies (the precision value). In total there was no improvement in the F-score. The BoostLog ensemble outperforms the single LSTM model and adapted very well to the new data while it kept the recall rate high. After training on test data it was very reliable and did not point out one single normal sequence as anomaly and still found 24 of the 26 anomalies.

The same models were then tuned for the HDFS data set and tested in the same way as for the RAN data, see Table It seems that it is much easier for the single LSTM model to learn the sequence behaviour of the HDFS block sequences compared to the RAN call chains, it can also more easily find the anomalies. A quick check with fast Dynamic Time Warping (fastDTW) [START_REF] Salvador | Toward accurate dynamic time warping in linear time and space[END_REF], confirms that the unique sequences in RAN data set is much more similar to each other than for the HDFS data. There is also a larger difference between the normal and anomaly sequences for HDFS data compared with the RAN data. Similar sequences will make it more difficult to predict the outcome for the single LSTM and if the anomaly sequences also are more similar to normal sequences it will get even more difficult to detect the anomalies. This might explain why the single LSTM model can detect the anomalies easier in the HDFS data. We can also see that, as in the RAN data set, there is no improvement for the single LSTM model when training extra on normal sequences not seen before. Since the single LSTM model already performs very well on the HDFS data set, the results cannot be improved much using an ensemble, we can only see a slight improvement using the BoostLog ensemble.

Our results cannot be directly compared with previous research that has used the HDFS data set [START_REF] Du | Deeplog: Anomaly detection and diagnosis from system logs through deep learning[END_REF], [START_REF] Xu | Detecting large-scale system problems by mining console logs[END_REF], since they use multi features and also look at the whole data set. But the results are promising, in [START_REF] Du | Deeplog: Anomaly detection and diagnosis from system logs through deep learning[END_REF], the best model, also got a F-score of 0.96 when it analysed the HDFS data set.

Conclusion and Future Work

We have created a novel method that turns the LSTM model into a classifier for anomaly detection in system logs, in this case, logs from a large scale system such as 5G RAN. Furthermore we have also shown how the AdaBoost algorithm can be adopted to use our LSTM classifier to create an even better anomaly detection ensemble, the BoostLog.

Evaluation of performance in two different datasets shows that it is more beneficial to use BoostLog in domains such as RAN, where the sequences are similar to each other and the anomalies are difficult to distinguish from the normal behaviour.

BoostLog also adapts well when a feedback loop was used to train extra on false positive sequences and maximized the precision for the RAN data set. We expect that deep ensemble based networks such as BoostLog, can be used to find more anomalies early in RAN development cycle and will require less domain expertise in order to find anomaly behaviour in the system logs.

In this paper we investigated how one feature could be used to predict the anomalies and it will be interesting to see how our future work can be improved by selecting more features and also take into account the time that events occur. In our previous work we have examined how anomalies and security issues can be detected using performance metrics [START_REF] Ibidunmoye | Adaptive anomaly detection in performance metric streams[END_REF], [START_REF] Bhuyan | Network Traffic Anomaly Detection and Prevention: Concepts, Techniques, and Tools[END_REF]. It will be interesting to see if the anomaly detection can be further improved using a combination of ensembles that analyze both functional and performance behaviour. We must also test our method on system logs from more different areas to see if the ensemble also is suitable to use in those domains.

Fig. 2 :

 2 Fig. 2: Using previous time steps to predict the next, case A, B and C

Fig. 3 :

 3 Fig. 3: 5G Radio Access Network (RAN)

Fig. 4 :

 4 Fig. 4: BoostLog architecture

Fig. 5 :

 5 Fig. 5: Probabilities for a state to occur in different sequences detection.

5 Fig. 6 :

 56 Fig. 6: Tuning of Single LSTM classifier same tendency is seen for the 1e -5 threshold and even if the accuracy for the training sequences increases the accuracy is not increased any more for the test sequences when the number of look backs are increased above 4, see Fig. 6b.

Fig. 7 :

 7 Fig. 7: BoostLog Section 3.3 the single LSTM model can only handle a certain amount of different sequences.When the same procedure was performed for the BoostLog the accuracy stayed the same for the anomaly sequences and increased from 84% to 100% for the sequences it had not seen before, see Fig.7a.Among the false negative cases, there were only two sequences that both the single LSTM and BoostLog classified as normal and both these cases missed one state in the sequence. These anomalies are difficult for the LSTM classifier to detect since the single missed state will not cause a very large drop in sequence error. If the LSTM predicts the probability for time step t+1 it will also know what state that is likely to occur in time step t+2. The probability for the state to occur in t+2 will be slightly higher than other states and if the state for time step t+1 is missing, the second highest probability is likely to be the state for t+2.

  .

  2.

Table 1 :

 1 Precision, recall, and F-score for RAN data set

	Model	Precision Recall F-score
	Single LSTM	0.64	0.88	0.74
	AdaBoost LSTM	0.77	0.92	0.84
	Feedback Single LSTM	0.59	1.0	0.74
	Feedback AdaBoost LSTM 1.00	0.92	0.96

Table 2 :

 2 Precision, recall, and F-score for HDFS data set

	Model	Precision Recall F-score
	Single LSTM	0.96	0.94	0.95
	AdaBoost LSTM	0.94	0.98	0.96
	Feedback Single LSTM	0.96	0.93	0.94
	Feedback AdaBoost LSTM 0.97	0.98	0.97

Acknowledgement

This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by Knut and Alice Wallenberg Foundation.


Petr Hajek
email: petr.hajek@upce.cz

Aliaksandr Barushka
email: aliaksandr.barushka@student.upce.cz

Michal Munk
email: mmunk@ukf.sk

Opinion Mining of Consumer Reviews Using Deep Neural Networks with Word-Sentiment Associations

Keywords: Opinion Mining, Consumer Review, Word Embedding, Lexicon, Sentiment, Deep Neural Network

Automated opinion mining of consumer reviews is becoming increasingly important due to the rising influence of reviews on online retail shopping. Existing approaches to automated opinion classification rely either on sentiment lexicons or supervised machine learning. Deep neural networks perform this classification task particularly well by utilizing dense document representation in terms of word embeddings. However, this representation model does not consider the sentiment polarity or sentiment intensity of the words. To overcome this problem, we propose a novel model of deep neural network with word-sentiment associations. This model produces richer document representation that incorporates both word context and word sentiment. Specifically, our model utilizes pretrained word embeddings and lexicon-based sentiment indicators to provide inputs to a deep feed-forward neural network. To verify the effectiveness of the proposed model, a benchmark dataset of Amazon reviews is used. Our results strongly support integrated document representation, which shows that the proposed model outperforms other existing machine learning approaches to opinion mining of consumer reviews.

Introduction

Opinion mining (sentiment analysis) of consumer reviews studies consumers' opinions on products and services [START_REF] Liu | Sentiment analysis: Mining opinions, sentiments, and emotions[END_REF]. The increasing number of users on online platforms produces a huge number of online product reviews. In the last two decades, opinion mining has become one of the most important text classification tasks because consumers' opinions affect the purchase decisions of other consumers. In addition, consumers' opinions in online reviews provide invaluable insights into consumer behavior and are thus central to companies. The large number of consumer reviews available across diverse online sources has led to the necessity of employing automated opinion mining systems. Numerous machine learning methods have been used for this task, including methods with supervised learning and methods exploiting sentiment lexicons [START_REF] Liu | Sentiment analysis: Mining opinions, sentiments, and emotions[END_REF]. Recently, deep neural networks have emerged as an effective tool. Multiple layers enable learning complex representations of features [START_REF] Zhang | Deep learning for sentiment analysis: A survey[END_REF]. Many deep neural networks in this domain use word embeddings as input features. Thus, words are transformed from a high-dimensional sparse space to lower-dimensional dense vectors, representing latent features and word context. Opinion mining has been investigated at three levels of granularity, namely the document, sentence and aspect levels. For example, product reviews can be represented as documents classified into positive or negative opinion categories. Note that in this task, it is assumed that the review concerns a single product entity. In sentence-level categorization, only opinionated sentences must be first selected. Aspect-level opinion mining requires the identification of a product's aspect (target). In other words, this approach comprises several subtasks such as aspect extraction and aspect opinion classification.

Concerning the features used for opinion mining of consumer reviews, the bag-ofwords model represents a traditional document representation in which word frequencies are calculated for each word (phrase) in the vocabulary [START_REF] Johnson | Effective use of word order for text categorization with convolutional neural networks[END_REF]. However, this approach results in high-dimensional sparse document representation. Moreover, this representation ignores word order. In the case of using n-grams instead of single words, a short context is considered. To overcome these problems, word embeddings were introduced to produce low-dimensional dense word representation [START_REF] Tang | Document modelling with gated recurrent neural network for sentiment classification[END_REF][START_REF] Tang | Learning semantic representations of users and products for document level sentiment classification[END_REF][START_REF] Chen | Neural sentiment classification with user and product attention[END_REF][START_REF] Xu | Cached long short-term memory neural networks for document-level sentiment classification[END_REF]. Compared with bag-ofwords, word embeddings are also more effective in modeling word context and word meaning. After the appropriate document representation is generated, various neural network models can be employed for opinion classification. Alternatively, neural networks can be used to produce word embeddings; then other machine learning methods, such as support vector machines, can be used for the classification task [START_REF] Do | Deep learning for aspect-based sentiment analysis: a comparative review[END_REF].

The core problem of word embedding representations in existing studies is that the sentiment polarity and intensity of the words are ignored. As a result, a word embedding may comprise words with opposite sentiment polarity. This study aims to overcome this problem by developing a deep neural network model integrating word embeddings with their sentiment associations obtained from a wide range of lexicons. To further improve the performance of the opinion classifier in domain-specific context of reviews on different products, bag-of-words features are incorporated into the model.

The rest of this paper is structured as follows. Section 2 briefly reviews the recent advances in deep learning for opinion mining of consumer reviews. Section 3 outlines the proposed model. In Section 4, the benchmark dataset is introduced. Section 5 presents the results of the experiments in comparison with existing approaches. Section 6 presents future research directions and concludes the paper.

Deep Learning for Opinion Mining of Consumer Reviews -A Literature Review

This section reviews existing deep neural network (DNN)-based approaches to opinion mining of consumer reviews. As demonstrated in earlier studies, NNs outperform other traditional machine learning methods such as support vector machine (SVM) and Naïve Bayes (NB) in this task, irrespective of the context of balanced/unbalanced datasets [START_REF] Moraes | Document-level sentiment classification: An empirical comparison between SVM and ANN[END_REF]. However, the initial efforts in this domain relied on a traditional bag-of-words model that produced high-dimensional and sparse datasets. It should be noted that shallow NNs are not effective in handling sparse datasets [START_REF] Barushka | Spam filtering using integrated distribution-based balancing approach and regularized deep neural networks[END_REF]. By contrast, DNNs have the capacity to overcome this problem by capturing more complex features from the data. A DNN unsupervised learning approach was developed in [START_REF] Glorot | Domain adaption for large-scale sentiment classification: A deep learning approach[END_REF] to show that word representation can be effectively learned by a stacked denoising autoencoder and that this representation can also be easily adapted to different review domains. To address the problem of scalability with the high-dimensional bag-of-words representation of the traditional autoencoders, a semisupervised autoencoder was developed for sentiment analysis in [START_REF] Zhai | Semisupervised autoencoder for sentiment analysis[END_REF]. Supervision is introduced into the model via the loss function obtained from a linear classifier. Convolutional NNs (CNNs) were also employed to use the bagof-words representation [START_REF] Johnson | Effective use of word order for text categorization with convolutional neural networks[END_REF], which was also one of the first attempts to effectively use word order for opinion classification.

To further improve the performance of DNNs in opinion classification, vector representation models such as Word2Vec [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF][START_REF] Le | Distributed representations of sentences and documents[END_REF] and Glove [START_REF] Pennington | GloVe: Global vectors for word representation[END_REF] were used to generate dense documents by reconstructing the linguistic context of the words. As a result, words that share a common context are located close to each other in the vector space, and the dimensionality of the space is reduced to several hundred word embeddings. CNNs and long short-term memory (LSTM) NNs were used to learn sentiment representation from word embeddings by [START_REF] Tang | Document modelling with gated recurrent neural network for sentiment classification[END_REF]. In the next step, document representation was learned using gated recurrent units (GRUs). Different approaches for generating word embeddings were combined in a CNN model that outperformed SVM and NB. Another proposed CNN model integrates word embeddings with the representation of user text, thus incorporating user preferences [START_REF] Tang | Learning semantic representations of users and products for document level sentiment classification[END_REF]. Similarly, user and product information were utilized in an LSTM model with word and sentence attention [START_REF] Chen | Neural sentiment classification with user and product attention[END_REF]. To overcome the problem of the memory unit with long texts, a cached LSTM model was developed to capture the overall semantic representation [START_REF] Xu | Cached long short-term memory neural networks for document-level sentiment classification[END_REF]. Cross-domain sentiment classification represents another challenge in related literature. To learn a document representation that can be shared across domains, an end-to-end adversarial memory network was introduced in [START_REF] Li | End-to-end adversarial memory network for cross-domain sentiment classification[END_REF].

Recently, a cross-modality consistent regression model was employed to utilize three different CNN models with attention mechanisms, namely semantic, lexicon and sentiment representations. It was shown that sentiment and lexicon representations overcome the disadvantages of semantic embeddings in Twitter sentiment analysis [START_REF] Zhang | Textual sentiment analysis via three different attention convolutional neural networks and cross-modality consistent regression[END_REF]. Indeed, word embeddings used in previous studies ignore the sentiment polarity and sentiment intensity of the words and, hence, often combine words with different sentiment polarity. This may lead to misrepresentation of the documents in the context of sentiment analysis. Moreover, the hybrid representation models combining word embeddings with the traditional bag-of-words representation may further improve the classification performance in related tasks due to highly domain-specific context [START_REF] Sun | Exploiting product related review features for fake review detection[END_REF][START_REF] Hajek | Fake consumer review detection using deep neural networks integrating word embeddings and emotion mining[END_REF]. Product reviews from different domains is exactly such a task. Inspired by these observations, the original contribution of this study is the proposal of a DNN model integrating word embeddings, bag-of-words and a wide range of sentiment polarity and sentiment intensity features to overcome the problems of the above approaches. Notably, word-sentiment associations enable to obtain both the meaning and sentiment intensity of the words in the review representation. Deep feed-forward neural network (DFFNN) was employed in this integrated model to effectively handle the high-dimensional sparse bag-of-words representation [START_REF] Barushka | Spam filtering using integrated distribution-based balancing approach and regularized deep neural networks[END_REF].

DNN Model with Word-Sentiment Associations

The architecture of the proposed DNN with word-sentiment associations (DNN-WSA) model for opinion mining of consumer reviews is presented in Fig. 1. The DFFNN with two dense hidden layers was used to process the variety in the input features, including both the word-sentiment representation and the n-gram representation. The word-sentiment representation is generated in two steps. First, word embeddings are trained using the Skip-Gram model because it is reportedly more effective than its competitors in exploiting the word context [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF]. Second, the vocabulary obtained from the corpus of reviews is compared with several lexicons to append lexicon-based sentiment polarity and intensity.

To calculate the embedding weight matrix, the adapted embedding function is applied to each word wt in the vocabulary. The embedding function is adapted for the sequence W = {w1, w2, … , wt, … , wT} of training words so that the following objective function is maximized

𝐸 = 1 𝑇 ∑ ∑ log𝑝(𝑤 𝑡+𝑗 |𝑤 𝑡 ) -𝑐≤𝑗≤𝑐 𝑇 𝑡=1 , ( 1
)
where c represents the context window radius (how many surrounding words are considered), and p(wt+1|wt) is the probability of the output word given the input words calculated using the hierarchical softmax algorithm

𝑝(𝑤 𝑂 |𝑤 𝐼 ) = ∏ 𝜎(⟦𝑛(𝑤, 𝑗 + 1) = ch(𝑛(𝑤, 𝑗)⟧𝑣´𝑛 (𝑤 𝑂 ,𝑗) 𝑇 𝑣 𝑤 𝐼 ) 𝐿(𝑤)-1 𝑗=1 , ( 2
)
where wI and wO are input and output words, respectively; vw and v'w denote the vector representations of the input and output words, respectively; n(w,j) is the j-th node in the binary tree; L(w) is the path length in the tree; ch(n) represents a child node; and σ(x) denotes a sigmoidal function, where if x is true, then ⟦𝑥⟧=1; otherwise ⟦𝑥⟧=-1. To obtain the document representation for the next layer in the DNN-WSA architecture, the mean values of the vectors from the embedding weight matrix were calculated.

To complement the word-sentiment representation with the sentiment polarity and intensity, we used several predefined sentiment lexicons. To obtain a reliable sentiment assessment, it is suggested not to rely on a single lexicon [START_REF] Bravo-Marquez | Determining word-emotion associations from tweets by multi-label classification[END_REF]. Moreover, the combination of lexicon-based sentiment indicators overcomes the problem of susceptibility to indirect opinions typically present in the machine learning models. To calculate sentiment polarity, we used two handcrafted lexicons of positive and negative words: Bing Liu's opinion lexicon [START_REF] Hu | Mining and summarizing customer reviews[END_REF] and OpinionFinder [START_REF] Bravo-Marquez | Determining word-emotion associations from tweets by multi-label classification[END_REF]. One shortcoming of these lexicons is that equal weight is assigned to all words regardless of their sentiment intensity. To address this issue, we incorporated the sentiment intensity indicators obtained from the following lexicons with pre-trained sentiment strengths [START_REF] Bravo-Marquez | Determining word-emotion associations from tweets by multi-label classification[END_REF][START_REF] Kiritchenko | Sentiment analysis of short informal texts[END_REF]: S140, NRC Hashtag, AFINN and SentiWordNet. Thus, the overall positive and negative scores can be calculated for each lexicon. In addition, the combination of several lexicons ensures higher lexical coverage [START_REF] Bravo-Marquez | Determining word-emotion associations from tweets by multi-label classification[END_REF].

To obtain the n-gram representation, the weight of each n-gram is calculated as follows

ωij = (1 + log(tfij)) × log(N/dfi), (3)
where ωij denotes the weight of the i-th n-gram in the j-th document (review); j = 1, 2, … , N; and tfij and dfi represent term and document frequency, respectively. Thus, review length is considered, and a relatively higher weight is assigned to rare n-grams.

For further processing, the n-grams are ranked according to their weights, and top ngrams are selected to enter the document representation layer in the DNN architecture. The next two hidden layers are used to process the complex relationship between the document representation and output sentiment positive/negative classes. To avoid overfitting and to make the training more effective, we used dropout regularization (dropout rate of 0.2 and 0.5 for the input and the two hidden layers, respectively) and ReLU (rectified linear units), respectively. The mini-batch gradient descent algorithm with b = 100 mini-batches, a learning rate of 0.1 and 1,000 iterations provided us with good and stable convergence behavior. Different numbers nh1 and nh2 of ReLU in the two hidden layers = {2 4 , 2 5 , 2 6 , 2 7 } were tested to obtain the optimal architecture. As presented below, the best results were obtained for nh1=2 5 and nh2=2 4 neurons. Note that we also experimented with one hidden layer but without improvement. The objective function was represented by cross-entropy loss. The overall complexity of the proposed model can be expressed as O(b×I×(m×nh1+nh1×nh2+nh2×nO)), where I is the number of iterations; m denotes the number of features in the document representation layer; and nh1, nh2 and nO represent the numbers of neurons in the first and second hidden layers and the output layer, respectively.

Data and Preprocessing

For the experiments, a large enough Amazon dataset that is openly accessible at Kaggle1 was used. The dataset, provided by Xiang Zhang, was originally used in [START_REF] Zhang | Text understanding from scratch[END_REF] to classify opinions in consumer reviews using temporal CNNs with character-level features. The dataset was collected from the Stanford Network Analysis Project since 1994 [START_REF] Mcauley | Hidden factors and hidden topics: Understanding rating dimensions with review text[END_REF], resulting in ~34 million reviews from ~6.6 million users on ~2.4 million products. The mean character length of the reviews was 764 (90.9 words). Extremely short and long reviews were discarded, and duplicates were removed. Users' rating scores were used to categorize the consumer reviews into positive and negative classes. More precisely, labels 1 and 2 were converted to negative opinion, and the scores of 4 and 5 were transformed to positive opinion. We used the testing data from the original dataset, represented by 130,000 samples from each score category. Overall, the dataset comprised 400,000 reviews evenly distributed into positive and negative opinion classes. Review title and review content were used in the dataset.

In the data pre-processing step, we performed tokenization (using the following delimiters: ".,;:'"()?!"), removal of stopwords (using the Rainbow list for noise reduction), and transformation to lowercase letters.

Experimental Results

The experiments were conducted on the Amazon dataset of 400,000 reviews. To learn word embeddings, we used the Skip-Gram model trained on the Amazon dataset. As shown in Fig. 2, we experimented with different settings of the model; the best performance was achieved with 200 word embeddings and context window radius c = 5. The Skip-Gram model was trained in the Deeplearning4j environment (distributed, opensource DNN library written for Java, compatible with Scala or Clojure and integrated with distributed computing frameworks Apache Spark and Hadoop). Regarding the bag-of-words representation, the top 1,000 n-grams (unigrams, bigrams and trigrams) were generated according to their tf.idf (term frequencyinverse document frequency) weights in agreement with the previous literature [START_REF] Kouloumpis | Twitter sentiment analysis: The good the bad and the omg![END_REF]. To obtain the word-sentiment associations, the AffectiveTweets package was employed.

In our experiments, three evaluation measures were considered: accuracy (Acc), area under receiver operating characteristic curve (AUC), and F-score. To evaluate the performance of the proposed model, stratified 5-fold cross-validation was performed. The mean values and standard deviations are presented.

In a further set of experiments, we examined the effects of the used word representations. Fig. 3 shows that the DNN model using lexicon-based sentiment features had the worst performance. More precisely, the DNNs with n-gram and Skip-Gram features increased accuracy by 2.7% and 3.0%, respectively, compared with DNN-LexSent. DNN-BoW and DNN-SkipGram performed similarly in terms of all the evaluation measures. The DNN-WSA model performed best with a 3.8% increase in accuracy compared with the DNN-SkipGram model. Overall, the combination of the three word representations performed significantly better than the baseline models at the 5% significance level using the Wilcoxon signed rank test. To comprehensively evaluate the effectiveness of the DNN-WSA model, we compared its performance against the following existing models:

• Improved Naïve Bayes (INB-1) [START_REF] Kang | Senti-lexicon and improved Naïve Bayes algorithms for sentiment analysis of restaurant reviews[END_REF] accommodates the sentiment word using the SentiWordNet lexicon in the feature extraction component. Following [START_REF] Kang | Senti-lexicon and improved Naïve Bayes algorithms for sentiment analysis of restaurant reviews[END_REF], we extracted the unigrams, bigrams and sentiment patterns. • Support vector machine with word sense disambiguation (SVM-WSD) [START_REF] Kausar | A sentiment polarity categorization technique for online product reviews[END_REF] uses adverbs scored using the SentiWordNet lexicon as input features. Thus, positive and negative scores were assigned to adverbs, and SVM was trained using the LibLINEAR library. L2-regularized L2-loss SVM type was employed with cost parameter C=1. • A multiple classifier model combining three baseline classifiers, namely NB, SVM and bagging (NB+SVM+Bagging) [START_REF] Catal | A sentiment classification model based on multiple classifiers[END_REF]. In agreement with the original study, we used unigrams as features and voting as the meta-classifier.

• LSTM [START_REF] Tang | Document modelling with gated recurrent neural network for sentiment classification[END_REF] and CNN [START_REF] Tang | Document modelling with gated recurrent neural network for sentiment classification[END_REF] were used to obtain the semantic sentence-level representation. Following [START_REF] Chen | Neural sentiment classification with user and product attention[END_REF], the dimension of hidden/cell states was set to 200, corresponding to the number of word embeddings. The CNN architecture comprised the convolutional layer with five filters of size 5 and a max pooling layer of size 4. For both models, the sentence representation was fixed and the number of words in the sentence corresponded to the review with maximum length. Document representation for both models was produced as the composition of sentence representation using GRUs. Stochastic gradient descent with Adam optimizer was used to train both models in the Deeplearning4j environment. Table 1 shows the results of DNN-WSA in comparison with the above models. Note that the proposed model not only performed best in terms of all the used evaluation measures, but its performance was also significantly better at the 5% significance level using the Wilcoxon signed rank test, which demonstrates the effectiveness of the proposed model. SVM-WSD also performed well in terms of accuracy, especially when considering the computational time.

In this study, we adopted the testing time criterion -as suggested in related studies [START_REF] Hajek | Fake consumer review detection using deep neural networks integrating word embeddings and emotion mining[END_REF] -to show the real-time capacity of consumer review classifiers. The proposed DNN-WSA model performed the worst regarding time efficiency, but it can still be considered time efficient with approximately 7,700 reviews classified per second. Recall that the key determinants of the overall complexity are the numbers of iterations and features in the DNN model. Therefore, better time efficiency can be expected with the decrease in the number of n-grams. Overall, the DNN-WSA model performed well for both opinion categories, as indicated by the high value of AUC. The other two DNN models, LSTM and CNN, also performed well regarding AUC. Additionally, the high value of the F-score for DNN-WSA indicates a balanced performance in terms of precision (0.896 on average) and recall (0.899).

Conclusion

In this study, we proposed an efficient DNN model integrating word-sentiment associations for the opinion mining of consumer reviews. We proved the model's performance improvement compared with baseline word representations by conducting extensive experiments on the Amazon dataset. We compared the proposed DNN-WSA model with several existing approaches, including both DNNs and other machine learning methods. Hence, the effectiveness of the proposed model was demonstrated. The results of the experiments suggest that word-sentiment associations might be more effective than word representation based on word embeddings only. Integrating the wordsentiment associations with n-gram representation provides further improvement. However, such a word representation model leads to a partly sparse dataset, which necessitates further requirements for the opinion mining machine learning methods. We showed that the proposed DNN model can handle such a word representation model.

In future research, a more thorough analysis can be performed by investigating the word-sentiment associations at the entity/aspect level. One of the limitations of the proposed model is that only local features were captured. Therefore, alternative DNN models with attention mechanisms could be considered to overcome this limitation. A crossdomain modification of the model is another problem that needs to be addressed. The n-gram feature extraction used in this study does not consider the semantic similarity or the discriminative ability of words. Therefore, enhanced n-gram representations [START_REF] Chen | A novel feature extraction methodology for sentiment analysis of product reviews[END_REF] are recommended to reduce the dimensionality and sparsity of the data. The application of an effective feature selection method may also lead to lower computational complexity and improved time efficiency [START_REF] Barushka | Spam detection on social networks using cost-sensitive feature selection and ensemble-based regularized deep neural networks[END_REF]. Alternative embedding-based schemes can also be utilized [START_REF] Onan | Deep learning based sentiment analysis on product reviews on Twitter[END_REF].

Fig. 1 .

 1 Fig. 1. The proposed DNN-WSA architecture for opinion mining of consumer reviews.

Fig. 2 .

 2 Fig. 2. The effect of the number of word embeddings on the performance of the DFFNN model with two hidden layers of nh1=2 5 and nh2=2 4 neurons.

Fig. 3 .

 3 Fig. 3. The performance of FFDNN models using a) bag-of-n-grams (DNN-BoW), b) Skip-Gram word embeddings (DNN-SkipGram), c) lexicon-based sentiment polarity and intensity (DNN-LexSent), and d) all the word representations together (DNN-WSA). All the models were trained using two hidden layers with nh1=2 5 and nh2=2 4 neurons.

Table 1 .

 1 Results of the experiments.

	Model	Acc [%]	AUC	F-score	Testing time [s]
	INB-1 [26]	78.84±0.70 0.833±0.008 0.796±0.005 5.862±0.363
	SVM-WSD [27]	85.61±2.67 0.856±0.027 0.862±0.022 0.070±0.011
	NB+SVM+Bagging [28] 83.10±0.68 0.906±0.006 0.833±0.006 0.105±0.011
	LSTM [4]	84.05±0.28 0.917±0.003 0.841±0.002 2.042±0.128
	CNN [4]	84.29±0.17 0.921±0.001 0.844±0.003 8.139±0.286
	DNN-WSA (this study)	89.70±0.92 0.959±0.011 0.897±0.009 10.398±0.226

Notes: The best results are in bold. The experiments were conducted using AMD Opteron 6180 SE 2.50 GHz with twelve cores/threads and 256 GB RAM.

https://www.kaggle.com/bittlingmayer/amazonreviews

Acknowledgments. This article was supported by the scientific research project of the Czech Sciences Foundation Grant No. 19-15498S and by the Student Grant Competition SGS_2020.


S Sarlis
email: stevensarlis@gmail.com

I Maglogiannis

On the reusability of sentiment analysis datasets in applications with dissimilar contexts

The main goal of this paper is to evaluate the usability of several algorithms on various sentiment-labeled datasets. The process of creating good semantic vector representations for textual data is considered a very demanding task for the research community. The first and most important step of a Natural Language Processing (NLP) system, is text preprocessing, which greatly affects the overall accuracy of the classification algorithms. In this work, two vector space models are created, and a study consisting of a variety of algorithms, is performed on them. The work is based on the IMDb dataset which contains movie reviews along with their associated labels (positive or negative). The goal is to obtain the model with the highest accuracy and the best generalization. To measure how well these models generalize in other domains, several datasets, which are further analyzed later, are used.

1.

Introduction

Sentiment analysis, also known as opinion mining, refers to the subjective analysis of a text [START_REF] Angiani | A Comparison between Preprocessing Techniques for Sentiment Analysis in Twitter[END_REF]. Its basic task is predicting the overall sentiment polarity of a given sentence. In more recent works [START_REF] Cambria | SenticNet 3: a common and common-sense knowledge base for cognition-driven sentiment analysis[END_REF], sentiment analysis expands this task on trying to identify the emotional status of a sentence (e.g., sadness, excitement, joy, anger). It mainly applies to reviews, social media posts and survey responses, helping businesses understand their customer's opinions about their products and services, improving their marketing strategies and decision policies [START_REF] Younis | Sentiment analysis and text mining for social media microblogs using open source tools: an empirical study[END_REF]. This practice of extracting insights from data is widely adopted by organizations and enterprises, affecting many aspects of human life.

Every day, many positive and negative comments are shared on the Internet. The impact of these comments on the economy is palpable and has led to the implementation of techniques that take advantage and extract knowledge from them. The sentiment of a movie review is usually associated with a distinct rating, which can be used for classification problems. From a user's perspective, it can be used as a recommendation tool for movie selection [START_REF] Lopez | IMDb Sentiment Analysis[END_REF]. The IMDb dataset contains 50,000 reviews and is evenly divided into 25,000 train reviews and 25,000 test reviews. Movie reviews are labeled as positive or negative, and the challenge is to predict the sentiment of an unseen review. In this work, the following classifiers are used: Logistic Regression, Bernoulli Naïve Bayes, Multinomial Naïve Bayes, Linear Support Vector Machine and two Neural Networks. Different techniques, including Count Vectorizer, TF-IDF (Term Frequency -Inverse Document Frequency) Vectorizer, lemmatization, stopwords, n-grams, minimum-maximum number of words and max features are implemented. The experiments conducted show that Logistic Regression, Linear Support Vector Machine and Neural Network classifiers perform noticeably better than Bernoulli Naïve Bayes and Multinomial Naïve Bayes. The study aims to find answers to the following three questions: 1. How various classification algorithms perform on the IMDb sentiment analysis classification problem? 2. How much pre-processing improves the performance of these classifiers? 3. Do these classifiers generalize well to other datasets and can they therefore be used in different real-life problems? The remainder of this paper is structured in four sections, as follows: Section 2 presents the background information and related research work, while Section 3 proposes the sentiment analysis system architecture. Section 4 describes the system in practice and reports the experiments conducted and the corresponding results. Finally, Section 5 concludes the paper and provides future work ideas.

Background and Related Work

Several approaches have been proposed for the extraction of features from text and the exploitation of them in forming appropriate classifying models. Initial attempts in order to represent a document used Bag of Words (BOW) representations, where the document was represented as the sum of one-hot vectors of its words. However, in the field of NLP, state-of-the-art Word Embeddings outperform BOW [START_REF]Neural Information Processing: 23rd International Conference, ICONIP 2016[END_REF], as BOW representations lose some of the semantic meaning associated with text [START_REF] Pasi | Advances in Information Retrieval: 40th European Conference on IR Research, ECIR 2018[END_REF]. Therefore, more recent works create vector representations for both words and documents, to retain as much information as possible. Furthermore, word embeddings can boost the generalization capabilities of neural systems [7][8]. However, in this work, BOW representations are used, as they require just a few lines of code to build the models and they work just fine in these specific datasets. The scikit-learn library, incorporated in Python [START_REF] Wagh | A Twitter sentiment analysis using NLTK and machine learning techniques[END_REF], provides different schemes that can be used, including word counts with Count Vectorizer and word frequencies with TF-IDF (Term Frequency -Inverse Document Frequency) Vectorizer, which are further analyzed in this work. Moreover, [START_REF] Martineau | Delta tfidf: An improved feature space for sentiment analysis[END_REF] presented the Delta TF-IDF technique, which is considered to be an improvement of the TF-IDF. The difference is that in regular TF-IDF, each word or combination of words, is associated with a TF value (word counts in the document) and an IDF value (word counts in all documents), while delta TF-IDF gives more weight to words that occur more often in that text, and are rarer in oppositely labeled documents. It is shown that the Delta TF-IDF produces noticeably better results than the regular TF-IDF. As mentioned earlier, text preprocessing is considered the most important step for NLP tasks. In an NLP pipeline, many preprocessing techniques are used including but not limited to lowercasing, lemmatization, character removal. These techniques have already been studied in [START_REF] Younis | Sentiment analysis and text mining for social media microblogs using open source tools: an empirical study[END_REF] and they are further analyzed later in this work.

Proposed Methodology for Sentiment Analysis

The process steps and the workflow diagram used for the sentiment analysis is shown in Figure 1. Firstly, the IMDb dataset is collected and imported. Subsequently, in order to remove noise and clean the data, pre-processing is performed. The next step is feature extraction and feature selection, where each document is represented as a documentterm matrix. In this matrix, rows refer to documents in the collection and columns correspond to terms. In the final step, the classification process is performed, and the used algorithms are evaluated.

Text Preprocessing

Before performing any classification algorithm, it is important to clean up the data. Finding and removing noise, greatly affects the accuracy of the classification algorithms. The following preprocessing techniques are applied: lowercasing, lemmatizing and removing noise characters and stop words. Lowercasing is probably the simplest and most effective preprocessing technique, where words map to the same lowercase form. This technique applies to all characters of the input text. Furthermore, the reviews are cleaned by removing HTML tags, non-word characters, punctuation and accents, which do not serve any purpose for detecting sentiment. Subsequently, a set of commonly used, low information words (i.e., stopwords), which do not carry any sentiment, is removed. During the stopwords process, the words included in the "NLTK list of English stopwords" are removed, except for the words "no" and "not" which are excluded, as they affect the value of the label. This initiative stems from the idea that, for example, the expression "not good" expresses a negative feeling, whereas if the word "not" was removed, the expression would become "good", which has a positive connotation, thus distorting the result. The process of lemmatization replaces the various forms a word can have, with its corresponding lemma (i.e., cutting out endings). This process is similar to stemming but it is preferred, as it doesn't just chop things off, but it actually transforms words to their actual root. Lemmatization is a standard preprocessing technique for linear text classification systems and has been shown to improve classification accuracy in sentiment analysis tasks [START_REF] Toman | Influence of word normalization on text classification[END_REF]. Finally, n-grams and more specifically bi-grams are used in some models. N-grams are a contiguous sequence of n items and consequently bi-grams are a sequence of 2 words [START_REF] Fotis | Sentiment analysis of social media content using N-Gram graphs[END_REF].

Feature Selection and Feature Extraction (Vectorization)

Common machine learning algorithms expect a two-dimensional table as an input, where rows are instances and columns are attributes. In this context, documents must be converted into vector representations. This process is called "feature extraction" or "vectorization" and is an important step when analyzing text. Words are coded as integers or as floating-point values, depending on which technique is implemented and are then used as an input to machine learning algorithms. In this work, two of the most well-known such techniques are used: The Count Vectorizer technique and the TF-IDF Vectorizer technique [START_REF] Avinash | A Study of Feature Extraction Techniques for Sentiment Analysis[END_REF].

Moreover, in order to apply machine learning algorithms on text segments, "feature selection" may be need, so as to reduce the input dimension, since the datasets may be very large and contain many words. Even after the preprocessing and the cleaning of the data, there may be a lot of different words in the dataset, which create a dictionary. Depending on the selected technique, Count Vectorizer or TF-IDF Vectorizer, these words are sorted based on their importance.

Count Vectorizer

Count Vectorizer is one of the simplest and relatively satisfactory techniques. It counts the number of times a word appears in a document and uses this value as its weight. The Count Vectorizer technique provides tokenization of the text documents and builds a vocabulary of words. Count Vectorizer counts words, so it returns integers, while TF-IDF Vectorizer assigns a score to words, so it returns floats.

Term Frequency and Inverse Document Frequency (TF-IDF) Vectorizer

In TF-IDF, the weight corresponding to each word not only depends on its frequency in the document, but also on how repetitive this word is in all documents. TF-IDF is a statistic that aims to reflect a word's importance in a document within a collection of documents [START_REF] Tarımer | Sentiment Analysis on IMDB Movie Comments and Twitter Data by Machine Learning and Vector Space Techniques[END_REF]. This technique is mainly used in information retrieval in order to rank how important a keyword is to a given document in a corpus and is a common approach for sentiment analysis [START_REF] Pelaez | Sentiment analysis of IMDb movie[END_REF]. Term Frequency (TF) is the frequency of the term within a document and indicates how many times a word is used within this document [START_REF] Tarımer | Sentiment Analysis on IMDB Movie Comments and Twitter Data by Machine Learning and Vector Space Techniques[END_REF]. TF describes how important a word is in the document, but it doesn't consider the possibility of that word occurring to other documents too, thus a balance term is needed. The Inverse Document Frequency (IDF) refers to the specificity of a term, which can be quantified as an inverse function of the number of documents in which it occurs. The more documents that have a specific word, the lower its IDF score is.

Classification and Comparison

To solve the classification problem, the following algorithms are used: Logistic Regression (LR), Bernoulli Naïve Bayes (BNB), Multinomial Naïve Bayes (MNB), Linear Support Vector Machine (Linear SVM) and two Neural Networks (NN1 & NN2).

All the parameters of the classifiers use the default values provided by the scikit-learn package, except for the Logistic Regression and the Linear SVM, where in the Count Vectorizer technique, the value of the max interactions parameter is set to 1,000 and 10,000 respectively. Furthermore, the Linear SVM classifier uses the hinge loss function, which is the standard SVM loss. Regarding the architecture of the Neural Networks, they are Sequential Neural Networks with four Dense layers. The first three Dense layers use the "relu" activation function and the last uses the "sigmoid". The Networks use two Dropout layers to avoid overfitting, "killing" random nodes each time, so that the network can recalibrate its weights. The optimizer is "adam" and the loss function is binary cross entropy. The batch size is set to 500 and the epochs used are only two, as the model overfits if trained longer. The two networks need to know what input shape they should expect. In this work, they have different input shapes. More specifically, the NN1 has an input shape equal to 57210 and the NN2 has an input shape equal to 50000. This results from the fact that in the NN1 the two vectorization techniques (Count and TF-IDF Vectorizer) are implemented with the first approach, which does not consider max features, while in the NN2 the two techniques are implemented with the second approach, which considers max features (50,000) and bigrams. Max features refer to the number of words or combinations of words required, to construct the feature vectors. The metric used to evaluate these classifiers is accuracy, as there is an equal number of samples belonging to each class [START_REF] Oswal | Predicting Rainfall using Machine Learning Techniques[END_REF]. Moreover, the models are trained on the IMDb dataset and tested on a variety of domains including food, clothing, tweet and hotel datasets. The IMDb is selected as the training dataset, as it contains more than 83 million registered users [START_REF] Younis | Sentiment analysis and text mining for social media microblogs using open source tools: an empirical study[END_REF], providing a rich and diverse source of human sentiments.

Experimentation and Results

This section reports the experiments conducted and the corresponding results. More specifically, in this section, the evaluation of the performance of these algorithms and their ability to generalize to other sentiment recognition problems is explored.

Description of Problems and Datasets

The IMDb dataset contains 50,000 reviews and is divided into 25,000 train and 25,000 test reviews. Movie reviews are labeled as positive or negative. The first problem is a classification problem, in which a variety of classification algorithms is being evaluated.

After preprocessing and cleaning the data, there are 57,210 different words in the train set (25,000 reviews). Since there is no limitation on the run-time of the algorithms and the memory size, a set of words was chosen, which leads to better accuracy rates. Therefore, for every classifier, the Count Vectorizer and the TF-IDF Vectorizer techniques are implemented. The main contribution of this work is to examine the generalization of the trained with the IMDb dataset classifiers on other sentiment recognition problems. A large set of words is chosen, in order to see if the classification models can generalize well to other datasets, since with larger sets of words (i.e., dictionaries) it is more likely to achieve better generalization results. More specifically, two different approaches are implemented. The first, takes all the 57,210 words for constructing the feature vectors. The second one, takes the top 50,000 words or combinations of words (i.e., bi-grams) for constructing the feature vectors. The datasets used to measure how well these models generalize include Women's E-Commerce Clothing Reviews Dataset [START_REF] Nicapotato | Women's E-Commerce Clothing Reviews[END_REF], Amazon Fine Food Reviews Dataset [START_REF]Amazon Fine Food Reviews[END_REF], Datafiniti's Business Hotel Reviews Dataset [START_REF] Datafiniti | Hotel Reviews[END_REF], Consumer Reviews of Amazon Products Dataset [START_REF] Datafiniti | Consumer Reviews of Amazon Products[END_REF] and two Twitter Datasets [START_REF] Youben | Twitter Sentiment Analysis[END_REF][START_REF]Twitter Sentiment Analysis Basic[END_REF]. The first four datasets have a score column with values between 1 to 5, therefore in order to determine if a sentiment is positive or negative, reviews with score ≥4 are mapped to label 1 (positive review) and reviews with score ≤2 are mapped to label 0 (negative review). The Twitter datasets are ready to be used directly in binary classification models (0 for negative and 1 for positive sentiment).

Results

As already mentioned, the IMDb dataset contains 50,000 reviews and the 25,000 of them are used to train the classification models. The other four datasets consisted of more test samples but since their label was mapped to 1 and 0 if they had score ≥4 or score ≤2 respectively, records with score equal to 3 were ignored, as they did not serve any purpose. The first Twitter dataset was fully used and from the second Twitter dataset 1/16 of the data was randomly selected. The results are shown in Tables 2 and3 and they are derived from the following classification algorithms: Logistic Regression (LR), Bernoulli Naïve Bayes (BNB), Multinomial Naïve Bayes (MNB), Linear Support Vector Machine (Linear SVM), Neural Network (NN1) and the following vectorization techniques: Count Vectorizer, TF-IDF Vectorizer. These two vectorization techniques are also implemented with the second approach, which considers max features (50,000) and bi-grams and is tested on the second Neural Network (NN2). The results on the IMDb Dataset concern the 25,000 reviews used for testing.

Discussion on the Results

From the experimental results listed above, the following conclusions arise:

1. In each model, TF-IDF technique provides higher accuracy rates, except for Multinomial Naïve Bayes and Neural Network I, where Count Vectorizer technique gives higher percentages. More specifically, in these two models, TF-IDF is better on the IMDb test set, resulting from the dataset in which the models have been trained and worse in other datasets. This means that it does not generalize as good as the other technique. a. The Multinomial classifier is used when features have discrete values. In practice, fractional counts such as TF-IDF also work [START_REF] Kibriya | Multinomial naive bayes for text categorization revisited[END_REF], however they lose some accuracy, as shown in the results. due to the fact that when analyzing tweets, different preprocessing techniques should be used. In particular, punctuation marks and emoticons may need to be preserved. Tweets preprocessing may involve converting many types of emoticons into tags that express their sentiment (i.e. :( → unhappy, sad) [START_REF] Thongtan | Sentiment Classification using Document Embeddings trained with Cosine Similarity[END_REF]. Therefore, emoticons play an essential role in tweets analysis as they serve the purpose of detecting emotion. Moreover, tweets are often too short in length, which makes the sentiment analysis process difficult.

Generally, as far as the Count Vectorizer technique is concerned, the models with the highest accuracy are the two Neural Networks and the Logistic Regression classifier.

Regarding the TF-IDF Vectorizer technique, the models with the highest accuracy are the Neural Network II and the Logistic Regression classifier. The Neural Network I and the Linear SVM only performed good accuracy rates in some datasets.

Finally, comparing all the mentioned models, those with the best accuracy rates are the Neural Network II and the Logistic Regression classifiers using the TF-IDF Vectorizer technique. These models achieve accuracies between 0,78 to 0,88, excluding the two Twitter datasets.

Conclusion

In this work, several classification algorithms were implemented and evaluated on sentiment-labeled datasets. The current state-of-the-art accuracy on IMDb [START_REF] Thongtan | Sentiment Classification using Document Embeddings trained with Cosine Similarity[END_REF], scores 97.42%, using document embeddings trained with cosine similarity. The scope of this work is different. The goal is not only to achieve good accuracy rates but also to get good generalization results on various datasets, by using traditional machine learning techniques. Logistic Regression and the second implementation of the Neural Network worked best on the IMDb dataset. More specifically, an accuracy score of 0,8888 was achieved on this dataset. Furthermore, the impact of constructing the feature vectors, on the performance of sentiment analysis classifiers, was analyzed. The Neural Network II with the second approach which takes the top 50,000 words or bi-grams to construct the feature vectors, generalizes better that the Neural Network I on every dataset. Common preprocessing techniques were used such as lowercasing, lemmatization, stopwords and non-word removal. All techniques provided significant improvements to the classifiers performances. Some of the techniques simply removed noise from the data, while others increased the importance of specific words or combinations of words. None of the classifiers performed well in the Twitter datasets. These moderate percentages may also be due to the misspellings occurring in data [START_REF] Agarwal | Sentiment analysis of twitter data[END_REF]. It is obvious that analyzing social media data is hard, as many of them are composed of misspelled words and special words or expressions. These are difficult for researchers to understand, as they are not used in everyday speech. However, sentiment analysis is not only useful for classification problems. An important perspective for future work would be to use sentiment analysis as a recommendation tool. Finally, datasets with discrete rating scores of each review (e.g. from 1 to 5) could potentially lead to better performance of the models used.

Figure 1 :

 1 Figure 1: Workflow Diagram

Table 1 .

 1 Datasets utilized in this work and corresponding samples.

	Datasets	Test Samples
	IMDb Large Movie Review Dataset v1.0	25,000
	Women's E-Commerce Clothing Reviews	20,615
	Amazon Fine Food Reviews	36,813
	Datafiniti's Business Hotel Reviews	8,438
	Consumer Reviews of Amazon Products	33,128
	Tweets dataset -Ayoub Benaissa	99,989
	Sentiment140 dataset with 1.6 million tweets	100,000

Table 2 .

 2 The accuracy of the models using the Count Vectorizer technique.

					Experiments
	Classifier	Trained	IMDb trained classifiers evaluated on various Datasets
		on this
		set
		(50%
		split)
		IMDb	Clothing	Food	Hotel	Product	1 st	2 nd
		Dataset	Dataset	Dataset	Dataset	Dataset	Twitter	Twitter
							Dataset	Dataset
	Logistic	0.85824 0.80392 0
	Regression

.75470 0.77281 0.84876 0.59829 0.58368

	Bernoulli	0.8196	0.45651 0.40895 0.57703 0.29890 0.45524 0.51936
	Naïve Bayes
	Multinomial	0.82076 0.70972	0.5827 0.77447 0.75066 0.57522 0.58787
	Naïve Bayes
	Linear SVM 0.8318	0.69876 0.70211 0.71284 0.80919 0.57536 0.56601
	Neural	0.8769	0.8385	0.7287	0.7817	0.8405	0.6043	0.5979
	Network I
	Neural	0.8865	0.8432	0.7659	0.7687	0.8156	0.5774	0.5907
	Network II

Table 3 .

 3 The accuracy of the models using the TF-IDF Vectorizer technique.

  The Bernoulli Naïve Bayes classifier gives the same accuracy rates for both techniques, in each dataset. Bernoulli Naïve Bayes is used for features with binary or boolean values. Therefore, TF-IDF values are converted to 0 or 1 (e.g., float values equal to zero are mapped to zero and float values greater than zero are mapped to one), hence, to Count Vectorizer values. The comparison of Bernoulli and Multinomial models was conducted, since training Naïve Bayes models is fairly cheap. 3. Lastly, neither of the classifiers performs good on the two Twitter datasets. This is

		b. As far as the Neural Network I is concerned, which approaches the task by
			taking all the 57,210 words to construct the feature vectors, Count
			Vectorizer outperforms TF-IDF Vectorizer. This approach is followed by Experiments all the models except for the Neural Network II, which takes the max
	2.	Classifier	Trained on this features (50,000) and bi-grams. IMDb trained classifiers evaluated on various Datasets
			set
			(50%
			split)
			IMDb	Clothing	Food	Hotel	Product	1 st	2 nd
			Dataset	Dataset	Dataset	Dataset	Dataset	Twitter	Twitter
								Dataset	Dataset
		Logistic	0.87996 0.84826 0.78792 0.82223 0.84964 0.61008	0.6053
		Regression
		Bernoulli	0.8196	0.45651 0.40895 0.57703 0.29890 0.45524 0.51936
	Naïve Bayes
	Multinomial	0.82744 0.69721 0.57618 0.73631 0.73025 0.56942 0.58517
	Naïve Bayes
	Linear SVM 0.87516 0.83371 0.74568 0.78916 0.81010 0.59899 0.59454
		Neural	0.8786	0.7392	0.6747	0.7234	0.7479	0.5800	0.5952
		Network I
		Neural	0.8888	0.8488	0.7776	0.8072	0.8310	0.5997	0.5944
		Network II