Keywords: Fouille de Données, Cloud Computing, Détection d'Anomalies, Apprentissage Profond, Flux de Données, Séries temporelles, Systèmes Distribués Data Mining, Anomaly Detection, Cloud Computing, Deep Learning, Data Streams, Distributed System, Time Series . . . . . . .

reforcées par notre utilisation des récentes avancées dans le domaine de l'apprentissage automatique. Il génère également des rapports détaillés mettant en évidence les composants impliqués et les applications associées à une anomalie.

Nous avons implémenté une instance de Monilog à l'échelle d'une plateforme cloud et mené des analyses expérimentales pour évaluer sa capacité à prévoir des événements anormaux, tels que des pannes de serveur résultant de problèmes de virtualisation. Les résultats obtenus soutiennent fortement notre hypothèse concernant l'utilité des logs pour la détection et la prévision d'événements anormaux. Notre implémentation de Monilog a identifié avec succès des périodes anormales et fournie des informations précieuses sur les applications concernées.

Avec Monilog, nous démontrons la valeur des logs pour la prédiction des anomalies dans de tels environnements et proposons une architecture flexible pour les études futures. Notre travail dans le domaine de la structuration des logs avec la proposition d'USTEP et d'USTEP-UP nous fournit non seulement des informations supplémentaires pour la construction de modèles de détection des anomalies, mais présente également des avantages potentiels pour d'autres applications d'exploration des logs.

Résumé

Les plateformes de Cloud Computing mettent à disposition de leurs clients différentes ressources informatiques à la demande. Cette externalisation rend les fournisseurs garants de la haute disponibilité et de la qualité de leurs services. La gestion d'un parc de ressources mutualisées en croissance constante demande de minimiser l'intervention humaine afin de suivre le changement d'échelle des infrastructures et d'éviter les erreurs. Dans cette thèse, réalisée en collaboration avec 3DS OUTSCALE, un fournisseur français de cloud public, nous explorons le potentiel des logs informatiques pour la détection automatique d'anomalies au sein des plateformes de cloud computing.

Les journaux de logs sont écrits pendant l'exécution et fournissent des informations sur l'état actuel d'un système. Ils sont déjà largement utilisés à des fins diverses, telles que la surveillance, le diagnostic, l'évaluation des performances ou la maintenance. Cependant, l'utilisation des logs pour la détection automatique et en temps réel d'anomalies reste compliquée. La nature complexe des plateformes de cloud computing doit être dûment prise en compte. L'extraction d'informations pertinentes à partir d'une multitude de sources de logs et les évolutions fréquentes de la base de code posent des défis et introduisent des risques d'erreurs. De plus, établir des relations entre les logs au sein de tels systèmes est souvent une tâche impossible.

Les solutions de structuration visent à retrouver les variables injectées dans les messages des logs. Notre première contribution implique une étude approfondie de deux de ces méthodes en examinant l'impact de l'optimisation des hyperparamètres et du prétraitement sur leur précision. Étant donné la nature laborieuse de l'étiquetage des logs dans le contexte des plateformes de cloud computing, nous avons cherché à identifier des valeurs génériques potentielles permettant une analyse précise dans divers scénarios. Cependant, nos recherches révèlent l'impossibilité de trouver de telles valeurs, soulignant ainsi la nécessité d'approches de structuration des logs plus robustes.

Notre deuxième contribution présente USTEP, une approche innovante de structuration des logs en ligne qui surpasse les méthodes existantes en termes de précision, d'efficacité et de robustesse. USTEP atteint une complexité temporelle d'analyse constante dans le pire des cas, le distinguant ainsi de ses prédécesseurs pour qui le nombre de patrons déjà découverts ralentit la vitesse de structuration. À travers une analyse comparative de cinq méthodes de structuration en ligne des logs utilisant 13 ensembles de données open source et un ensemble de données dérivé des systèmes de 3DS OUTSCALE, nous démontrons les performances supérieures d'USTEP. De plus, nous proposons USTEP-UP, une architecture qui permet l'exécution distribuée de plusieurs instances d'USTEP.

Notre troisième contribution présente Monilog, une architecture système conçue pour la détection automatique des anomalies à partir de journaux de logs. Monilog exploite des paires modèle/métrique pour prédire l'activité logs au sein d'un système et détecter les anomalies en identifiant des changements de comportement. Les

Context

Cloud computing services offer users on-demand self-service access to a shared pool of IT resources. This model provides clients with significant advantages [2; 3], including elasticity, enabling flexible scaling of computing resources according to their needs. Additionally, it offers broad network access, allowing users to access cloud services from anywhere at any time.

Moreover, the pay-by-use model eliminates the requirement for substantial upfront investments in computing resources, physical infrastructure, electricity consumption, and staff for system administration, network management, and database maintenance. These costs are transferred to the service provider.

The cost-effectiveness and flexibility offered by cloud computing have resulted in a widespread adoption among businesses and organizations. However, this shift towards outsourcing also places the responsibility on cloud service providers to ensure high availability and service quality.

From the provider's perspective, this exponential growth (with a projected 29.8% increase in the public sector by 2023 [4]) is accompanied by an expanding deployment of infrastructure to manage. To cope with the constant expansion of resources, it becomes crucial to minimize human intervention and adapt to changes in infrastructure scale while mitigating the risk of errors. Consequently, the development of automated monitoring and alerting tools becomes imperative to support administrators [5; 6; 7].

In large-scale online systems such as cloud computing platforms, even a single incident can have a profound impact on millions of users [8; 9; 10]. This impact extends beyond the immediate inconvenience and leads to significant financial consequences [START_REF] Das | Desh: deep learning for system health prediction of lead times to failure in hpc[END_REF] as well as performance losses.

For example, a four-hour downtime in Amazon Web Services resulted in a staggering loss of $150 million 1 . Consequently, anomaly detection plays a crucial role in the development of secure and reliable platforms. By enabling timely and accurate identification of anomalous events, operational teams can swiftly respond and take measures to mitigate potential losses [12].

The Landscape of Cloud Monitoring

Cloud computing platforms are intricate systems comprised of numerous physical components, distributed across multiple data centers. It is not uncommon for a large data center to contain over 100,000 components, such as servers, network equipment, and storage platforms. Monitoring the health of such infrastructure poses a significant challenge for operational teams [13].

Site Reliability Engineers (SREs) often rely on metrics like network latency or Key Performance Indicators (KPIs), such as the number of active virtual machines per region, to carry out their tasks [START_REF] Ren | Time-series anomaly detection service at microsoft[END_REF]. These metrics enable passive monitoring, where alerts are triggered when values exceed preset thresholds (e.g., when the memory usage of a machine approaches its limit). Additionally, SREs may employ manually defined rules to detect specific undesirable scenarios, such as denial-of-service and distributed denial-of-service attacks, which pose a serious threat to the availability of cloud computing environments [START_REF] Yan | Distributed denial of service attacks in software-defined networking with cloud computing[END_REF]. Studies have shown that rule-based approaches can effectively identify the machines responsible for such attacks and ban their IP addresses [16; 17].

The rule-based approaches lean on expert knowledge to identify and encode new undesirable scenarios. For example, they may monitor updates to lists of common vulnerability exposures.

However, the inability to detect or adapt to novel attacks or failures becomes increasingly problematic as the system expands [18]. In the context of cloud computing, platform updates and the diversity of behaviors among components amplify the potential attack vectors and the occurrence of false alarms within the system.

An Introduction to Logging

Capturing runtime information is a common practice within software systems [19; 20]. The produced logs describe a vast range of events as well as variations in the monitored system states [START_REF] Chen | A survey of software log instrumentation[END_REF]. Figure 1.1 is a sample of 20 log lines from a High Performance Computing cluster (HPC) at the Los Alamos National Laboratories log file. Each line within a log file is called a log event and is generated by a log statement inside the source of a program. Logs are

Research Objectives and Axis Covered by This Manuscript

In this thesis, we explore the automation of the exploitation of logs to detect anomalies and predict incidents within a cloud computing environment. The goal was to design and implement an innovative autonomous log-based Anomaly Detection (AD) system, capable of detecting anomalies on a cloud computing platform with high accuracy and minimal false positives to reduce false alarms; characterize the anomalies according to properly target the experts able to manage the alerts and provide them with all relevant information; process a large volume of log files online with a short detection time. This research project is supported by 3DS OUTSCALE, a French provider of multi-sovereign cloud services. The company extensively employs logs to monitor various systems comprising their cloud computing platform, as well as to identify undesirable outcomes like virtual machine crashes and security breaches. Being able to conduct this study within an industrial context has granted us opportunities to gather insights from field experts and validate our proposals using a large-scale, real-world cloud platform.

Elements of Vocabulary

Within this manuscript, we will regularly use the terms: autonomous, robust, efficient, and precise. We, therefore, think it is important to precisely define here the meaning of these terms within the context of this manuscript:

• Autonomous: A system will be considered autonomous if it can perform a targeted task for an extended period (months), without the need for human assistance.

• Robust: A robust system retains its operational capability despite changes occurring within its environment. Such environmental changes can for instance be the appearance of new log statements inside the source code of a monitored system and with it the appearance of new log messages.

• Efficient: The efficiency of a system or an algorithm reflects its ability to process a large volume of events in a short time. In our context, this means being able of processing millions of log lines per minute, this to be able to operate in real-time.

• Precise: The Precision of an alerting system is evaluated using Equation 1.1. With True Positive the number of events correctly classified as abnormal and False Positive, the number of irrelevant alerts. Note, a system can be perfectly precise but still miss shows that after 8 versions, the proportion of changed log statements reaches 30.3% [START_REF] Zhang | Robust log-based anomaly detection on unstable log data[END_REF].

The challenging aspects of cloud computing platform architectures, a large number of connected devices of various types (thousands inside 3DS OUTSCALE European platform), frequent changes in the code base, and an important logging volumetry (millions of log lines per minute generated by 3DS OUTSCALE European cloud platform) served as the catalyst for the research work presented herein. The main identified challenges revolve around the automated and efficient exploitation of log messages (log parsing), the precise detection of anomalies in highly distributed infrastructure, and the contextualization of the identified anomalous events. We organized our research work around the three aforementioned axes. Programs generate log events during runtime, which are then stored in log files. Log events constitute semi-structured data, featuring a header with a consistent format, while the message content is dictated by the developers' discretion. Log messages are generated by injecting variables linked to the real-time state of a system inside a code-embedded message template. As discussed, logs are created according to a predefined format (e.g., rsyslog RFC5424 [START_REF] Gerhards | The Syslog Protocol[END_REF])

and can be divided into two parts:

• A header, composed of different predefined fields. In the Figure 1.3 example, the header is composed of 4 distinct fields (TIMESTAMP, HOST, APPLICATION, and SEVERITY ).

As the header formatting is defined as software level, all the log headers will follow the same format.

• A message, which is a text field without format constraints. Due to its flexibility, it is common to embed information describing the current state of the system inside the message.

Mining log-embedded information in contexts where you don't have access to the source code is a common issue for automated log-based applications. Log parsing is an active research topic [START_REF] Zhu | Tools and benchmarks for automated log parsing[END_REF] and, a large panel of approaches have been proposed during the last two decades. To deal with the unstructured format of log messages and provide them with structured inputs, existing log-based anomaly detection methods rely on log parsing algorithms to mine logs underlying templates. As the quality of the parsing process has a direct influence on the precision, and the efficiency of the downstream log-based applications, and to be relevant in a cloud environment, a parsing solution must be able to work online, efficiently, and be robust to the evolution of the log statements.

Alongside this data mining axis, we explore state-of-the-art log parsing solutions and their relevance for a cloud logging environment, as well as ways of exploiting log-embedded information. Particular attention was given to the robustness of the existing solutions and of our 1.2 Research Objectives and Axis Covered by This Manuscript proposals, as well as their ability to process large volumes of logs efficiently.

Axis 2: Anomaly Detection in Highly Distributed IT Infrastructures

Due to their abundance and the information they contain, the logs are one of the most valuable data sources for anomaly detection [26; 27]. Different log-based anomaly approaches have been proposed within the previous years based on traditional machine learning techniques and more recently on neural approaches [28] such as LSTM [24; 29] or Transformer structures [START_REF] Nedelkoski | Self-supervised log parsing[END_REF]. Deep learning techniques employ trained models to anticipate the subsequent element in a log template sequence, flagging an anomaly if the predicted probability falls below a set threshold. Those methods have strong claims regarding their performances, however, in practice, we found that knowing or inferring log relationships to identify log sequences is a challenging and often not possible task within a cloud environment. This is due to logs most of the time not having a unique operation identifier. Also, inferring related events is hard due to the multiplicity of logging devices, the involved volumetry and the temporal proximity of operations. Such constraints limit the application of previously cited anomaly detection methods to the monitoring of specific cloud components (e.g., the API level where you are likely to find request-id).

Alongside this axis, we explore automated anomaly detection systems that meet the requirements of volume, variability and evolution of a cloud platform logs such as 3DS OUTSCALE one.

Axis 3: Contextualization of Anomalies for Decision Support

When an anomaly is detected, the information sent back to a human team must indicate as precisely as possible the cause and the context of the problem. To ensure effective resolution, identifying the context is as important as the anomaly itself. The classification of an anomaly serves, first of all, to target the most qualified experts to manage it. Work-related to this axis includes a literature review of the existing classification systems for log-based anomalies and their relevance in a cloud computing context.

An Overview of Our Contributions

Due to the constant evolution affecting the logging behavior of the underlying components of a cloud platform, classifying anomalies is a tedious and not relevant in the long-term task. Also, professionals may have different ways of handling the same anomalies depending on their context.

Those concerns drove us to propose and evaluate a context-aware anomaly detection system that outputs anomalies alongside their context. This context includes the concerned equipment and applications as well as ways of evaluating its criticity. This output allows practitioners to build their own business rules on top of it instead of forcing them inside a classifier.

An Overview of Our Contributions

To be efficient in a cloud computing environment, a parsing method should be accurate, efficient, and able to work in an online fashion. When working with existing log-based anomaly detection methods, we noticed that poor parsing quality drastically reduces the efficiency and precision of the considered approaches. Our experiments with state-of-the-art online parsing methods reveal that hyperparameter tuning and the use of preprocessing regular expressions have a significant impact on the parsing accuracy. This lack of robustness is a major concern in our cloud computing context as it is impracticable to label logs to correctly tune them. In response to this, we introduced USTEP, a log parsing algorithm. This evolving tree-structured algorithm can discover and encode new parsing rules while processing logs online. USTEP is, to the best of our knowledge, the only parsing method that achieves constant parsing time where other methods slow down with new template discovery. USTEP is robust and does not require any prior knowledge regarding the logging environment. We evaluate our proposal against four state-of-the-art online methods over 14 datasets coming from real-world applications. USTEP demonstrated superior performance in both effectiveness and robustness. We also introduced USTEP-UP, a way of running decentralized USTEP instances to ensure the scalability of the parsing step.

Those contributions to the log parsing field led to four publications: To facilitate its use by practitioners and researchers, we have made the source code of USTEP1 publicly available.

When reviewing log-based anomaly detection state-of-the-art, we noticed that existing work was focused on detecting anomalous log sequences. In practice, the reconstruction of log sequences in a cloud computing context is often impossible, due to a lack of unique identifiers as well as the technical impracticability to link logs from two different sources related to the same event. As we could not apply existing log-based anomaly detection methods due to prerequisite conflicts, we chose to explore new log-based ways of detecting abnormal events within large composite platforms.

The design of Monilog, our automated log-based anomaly detection system, is inspired by the field of multivariate time series. Monilog uses models to forecast the log traffic and the resulting error time series is computed using this forecast and the real traffic to detect anomalous patterns to detect abnormal behavior of equipment. Monilog has the capacity to generate detailed reports, regarding anomaly candidates as well as the implicated system and applications. Monilog doesn't

Organization of This Manuscript

require any prior knowledge regarding the relationship between logs, and can work in a streaming fashion allowing close to real-time alerting. We implemented and tested our proposal at a cloud scale using all the syslog messages [START_REF] Gerhards | The Syslog Protocol[END_REF] generated by the Kernel Based Virtual Machine of 3DS OUTSCALE European cloud region for 11 consecutive days. Our system successfully predicted all three reported server crashes at least 80 minutes prior to their occurrence. Our precision evaluation also displays the relevance of the other reported abnormal events.

Monilog is the corner stone of this thesis, related work led to two publications:

• Vervaet,

Organization of This Manuscript

The remainder of this manuscript is organized as follows. In Chapter 2, we present existing usages and applications based on logs as well as the state-of-the-art on automated log-based anomaly detection and log parsing. In Chapter 3, we study the impact of parsing accuracy on AD method's accuracy and the impact of tuning on existing methods. Chapter 4 is dedicated to USTEP, our proposal log parsing solution workflow and memory structure, the USTEP-UP architecture, and our perspectives regarding the log-parsing problem. Chapter 5 is dedicated to Monilog, our automated log-based anomaly detection system. We discuss here its architecture and state-of-the-art limitations. In Chapter In this Chapter, we present a state-of-the-art of automated log-based anomaly detection (Section 2.3), and we discuss identified limitations of the existing methods in particular for their application to the cloud-computing platform field (Section 2.6). This chapter also includes a state-of-the-art of log parsing domain as well as an overview of some log-mining applications.

Log Mining Applications Within the Literature

Logs are widely available data sources containing real-time information regarding the states of a system, and they have already proved their utility for a wide range of mining tasks such as user, or resource profiling. With the increasing amount of generated log lines, multiple automated log-based approaches have been proposed [35; 36] to exploit the diversity of information they contain. In this section, we present examples of possible exploitation of the logs that we found in the literature. We do not intend to present an exhaustive list of existing applications, what follows should in no way be considered as a survey but rather as an inspiration and an opening on the existing usages of logs.

Root Cause Analysis

Root cause analysis is one of the most common and traditional uses of logs [START_REF] Oliner | Advances and challenges in log analysis[END_REF]. By examining logs preceding a system failure or other undesirable state, one can reconstruct the sequence of events leading to the system's failure. One challenge is that log files are typically designed to represent a single stream of events. Messages from multiple sources, however, may be interleaved both at runtime (from multiple threads or processes) and statically (from different modules of a program). For static interleaving, header-embedded information can help filter out logs and focus on the one related to the considered context. However, for runtime interleaving, a thread ID does not solve the problem because a thread can be reused for independent tasks. This is a major challenge when performing root cause analysis on a supercomputer, leading to the proposal of tools like LogAider [START_REF] Di | Logaider: A tool for mining potential correlations of hpc log events[END_REF], to mine potential correlations of HPC log events.

Dissection of Performance Issues

Log analysis can help optimize or debug system performance. Understanding a system's performance is often related to understanding how the resources in that system are used. Distributed systems are notoriously difficult to get right, and we came across several research papers proposing log-mining approaches to dissect and improve the performances of such systems. Wang et al. [START_REF] Wang | A zoom-in analysis of i/o logs to detect root causes of i/o performance bottlenecks[END_REF] proposed a log-based approach to detect the root causes of writing data on parallel file performance bottlenecks. By analyzing logs collected on Cray XC40 supercomputer system used for scientific purposes, authors produced guidelines to help developers optimize the I/O behavior of their applications. Tan et al. [START_REF] Tan | Visual, log-based causal tracing for performance debugging of mapreduce systems[END_REF] explored a log-based approach for performance debugging of MapReduce, a programming paradigm and framework [START_REF] Dean | Mapreduce: simplified data processing on large clusters[END_REF] for parallel distributed computations on commodity clusters. Lu et al. [START_REF] Lu | Cloudraid: Detecting distributed concurrency bugs via log-mining and enhancement[END_REF] focused their work on detecting concurrency bugs in distributed systems by the generated logs. Concurrency bugs are triggered by complex interleaving of messages and are difficult for programmers to correctly reason about and handle concurrent executions on multiple machines. The use of the proposed log mining approach alleviates the pain of hands-on root cause analysis in such scenarios.

Detection of Intrusions and Attacks

Logs are commonly used for security purposes. Searching log files can help detect intrusion traces or identify malicious attack patterns. When searching for attacks, pattern matching is an efficient way of looking for traces of classical malicious behaviors. With this approach, only known patterns can be recognized, yet new types of attack may appear with only small changes made to existing patterns. This explains why more recent approaches are focused on machine learning techniques to detect anomalous behaviors to be more robust to attacker's small changes.

As logs are a broadly available data source in many systems, they are used by a wide range of security applications. We came across methods using logs to detect SQL injections, a classic

Anomaly Detection

attack to mine or alter the content of a database [START_REF] Moh | Detecting web attacks using multistage log analysis[END_REF]; or detect malicious behaviors using proxy logs [START_REF] Mimura | Adjusting lexical features of actual proxy logs for intrusion detection[END_REF]. There are also examples of log-mining applications that reconstruct behavior graphs to detect intrusions or other types of unusual behaviors [45; 46; 47].

Failure Prediction

Failure prediction also forms a dynamic focus area within the log mining field. Logs have been used by Gao et al. [START_REF] Gao | Incremental prediction model of disk failures based on the density metric of edge samples[END_REF] and Chaves et al. [START_REF] Chaves | Hard disk drive failure prediction method based on a bayesian network[END_REF] to predict hard disk drive failures. Considering the varied brands and models of drives with different input/output workload patterns present in data centers, their approach becomes particularly pertinent [START_REF] Shen | Random-forest-based failure prediction for hard disk drives[END_REF]. Log-based failure prediction has also been successfully applied for detecting failures in HPC nodes [START_REF] Das | Aarohi: Making real-time node failure prediction feasible[END_REF] and Oracle databases [START_REF] Karakurt | A machine learning approach to database failure prediction[END_REF].

Failure prediction is close to anomaly detection; however, failure prediction methods are designed for one system, or one class of system and usually exploit context-specific information to perform their task, whereas anomaly detection methods aim to be system agnostic. The remainder of this manuscript is focused on log-based anomaly detection methods.

Anomaly Detection

Example of Applied Research Work for Different Fields

Due to its practical utility, anomaly detection has emerged as a significant topic area across various fields of application [START_REF] Chalapathy | Deep learning for anomaly detection: A survey[END_REF]. In the medical domain, there are several recent examples [START_REF] Tschuchnig | Anomaly detection in medical imaging-a mini REFERENCES review[END_REF]

Within the Cloud Computing Domain

The development of large-scale services and their underlying systems typically involves the collaboration of hundreds of contributors, divided across different teams and sometimes, distinct organizational structures. Developers and operators usually have incomplete information regarding the overall system and tend to determine anomalous events from a local perspective, which is error-prone. Besides, manually detecting anomalous log sequences is not efficient nor scalable because of the continuous increase in the volume of logs generated by modern systems.

The use of keyword-matching techniques or regular expressions helps to detect simple and well-known anomalous events by seeking characteristic patterns. Although such pattern-matching approaches are unable to identify a large portion of the anomalies, as many of them are sequences of "non-anomalous" logs leading to an undesired outcome. Moreover, rule-based approaches are sensitive to changes in the code base. This need for automation and robustness are reflected in numerous research works on automated log-based anomaly detection methods [START_REF] He | Experience report: System log analysis for anomaly detection[END_REF]. According to Bhanage et al. [START_REF] Bhanage | It infrastructure anomaly detection and failure handling: A systematic literature review focusing on datasets, log preprocessing, machine & deep learning approaches and automated tool[END_REF], over 87 different publications related to log-based anomaly detection have been issued between 2016 and 2021. In this section, we cover only the most representative methods and their specificities. Table 2.1 presents a summary by years of the selected methods alongside the original papers and the used techniques.

Automated Log-Based Anomaly Detection

Traditional Methods

To the best of our knowledge, the first reported log-based anomaly detection method was brought up by Liang et al. in 2007 [64]. The proposed system is based on the traditional Support Vector Machines (SVM) set of classifiers and aims to detect failures affecting IBM BlueGene/L supercomputer.

In 2009, Xu et al. proposed a method based on the Principal Component Analysis (PCA) algorithm [START_REF] Xu | Largescale system problem detection by mining console logs[END_REF] to detect system runtime problems. For their evaluation, they used log traces generated by Hadoop1 , an open-source software dedicated to the management of big data files.

They reconstruct the log sequences using code-embedded log statements for perfect parsing and display the relevance of PCA for detecting anomalous log sequences in an online fashion.

Deep-Learning Based Methods

In 2017, Du et al. proposed Deeplog [START_REF] Du | Deeplog: Anomaly detection and diagnosis from system logs through deep learning[END_REF], a log-based anomaly detection method based on an LSTM neural network. Deeplog model logs as natural language sequences and can process them in a streaming manner. According to an experimental evaluation conducted by Deeplog authors, it achieves higher accuracy than the previously presented non-traditional methods. Deeplog is the first identified deep-learning based method and it led the way to multiple approaches driven by the progress in this domain.

Deeplog approach was directly extended by LogRobust [START_REF] Zhang | Robust log-based anomaly detection on unstable log data[END_REF], LogAnomaly [START_REF] Meng | Loganomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs[END_REF] and Swiss-Log [START_REF] Li | Swisslog: Robust and unified deep learning based log anomaly detection for diverse faults[END_REF], three methods based on bidirectional LSTM neural networks aim to detect sequential log anomalies. Authors of each method identified log statement instability as a significant issue, proposing innovative embedding mechanisms to mitigate the effects of log concept drift on their models. To regroup new templates with an existing one to keep unchanged the embedding vector size, LogRobust uses semantic information of the log event, LogAnomaly generates an intermediate template using Ft-tree [START_REF] Zhang | Syslog processing for switch failure diagnosis and prediction in datacenter networks[END_REF], and SwissLog uses a BERT [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF] encoder to encode log templates.

Convolutional Neural Network (CNN) has been proposed to capture local semantic information instead of global information and defeat the overfitting issues in regular neural networks.

It has become one of the most representative neural networks in the field of deep learning [START_REF] Li | A survey of convolutional neural networks: analysis, applications, and prospects[END_REF].

This structure has allowed significant advances in the field of computer vision, but it has also been successfully applied to time series prediction and signal identification problems. Closer to our context, CNN has been used by Ren et al. to assign a category (e.g., Network, Memory, Security, DataBase ...) to log events [START_REF] Ren | Deep convolutional neural networks for log event classification on distributed cluster systems[END_REF] based on their message. In 2018, Lu et al. [START_REF] Lu | Detecting anomaly in big data system logs using convolutional neural network[END_REF] proposed a CNN-based approach able to learn event relationships in system logs and detect anomalies based on this information. More recently, Yang et al. introduced PleLog [72], a logbased anomaly detection approach that combines a clustering step and a Convolutional Neural Network. The clustering step is used to assign a label to log events and allows the system to train more efficiently within unsupervised contexts.

An Introduction to Log Parsing

The great success of the Bidirectional Encoder Representations from Transformers (BERT) in modeling sequential data [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF] inspired multiple Transformer-based approaches. In our context, we have identified two different types of BERT usages: 1/ Nedelkoski et al. [START_REF] Nedelkoski | Self-attentive classification-based anomaly detection in unstructured logs[END_REF], and Guo et al. [START_REF] Guo | Logbert: Log anomaly detection via bert[END_REF] train such model to capture patterns of normal log sequences and used it later on to detect anomalous ones; 2/ Previously depicted method SwissLog uses only the encoder part of Bert to enhance the robustness of the addressed approach to new logging patterns.

Finally, we want to highlight here four log-based anomaly detection methods based on different techniques that we found interesting: 1/ Farzad et al. [START_REF] Farzad | Unsupervised log message anomaly detection[END_REF] used two deep autoencoder networks to extract features from logs and perform anomaly detection using the Isolation Forest (IF) algorithm [START_REF] Liu | Isolation forest[END_REF]; 2/ LogGAN, proposed by Xia et al. [START_REF] Xia | Loggan: a log-level generative adversarial network for anomaly detection using permutation event modeling[END_REF] an LSTM-based generative adversarial network for anomaly detection; 3/ QLLog is another adversarial method based on the Q-learning algorithm, a reinforcement learning method based on value function; 4/ LightLog, a light anomaly detection method to handle large-scale logs on edge devices with limited computed power. Lightlog uses a modified Temporal Convolutional Network (TCN) to detect anomalies.

An Introduction to Log Parsing

Log parsing is a classic first step for log-based anomaly detection and log-based applications in general. Log parsers aim to extract the underlying information of a log event without assessing the associated log statement. In practice parsing the header is straightforward as it follows a given format, however, separating the injected variables from the message template is a challenging task. The application of regular expressions is a straightforward method to parse raw log messages; thus, this requires listing all the logging statements and generating the appropriate expressions. In practice, the source code is not always accessible, and keeping track of all the changes is a tedious task [START_REF] Kabinna | Examining the stability of logging statements[END_REF]. Log-parsing methods aim to tackle this issue by providing an automated way of mining log templates from log messages. Previously presented Figure 1.3 illustrates the expected parsed version of a sample log line.

Problem Formulation

In this manuscript, we will adopt the formalism introduced by Nedelkoski et al. [START_REF] Nedelkoski | Self-supervised log parsing[END_REF] regarding the log-parsing problem. Logs are defined as sequences of temporally ordered unstructured text messages L = (l i : 1, 2, ...), with i the positional index of a log message l i within the sequence.

Tokens are the smallest inseparable singleton objects within a log message. Each log message is constituted of a finite sequence of tokens (words) separated by spaces

t i = (t i j : t ∈ T, j = 1, 2, ..., |t i |).
With T the set of all tokens, j the positional index of a token within a log message l i , and |t i | the total number of tokens inside l i . Tokenization is defined as a transformation function M : l i → t i , ∀i.

Log parsing aims to structure log messages by separating their constant (template) part, from the variable ones. A log parsing method is defined as a function f : t i → (e i , v i ), with e i a template, and v i a list of variables.

Log Parsing State of the Art

During the last two decades, different approaches to the log parsing problem have been proposed [1; 35; 90]. According to the usage workflow, log parsers can be classified in two approaches: Offline algorithms process logs in a batch manner updating the parser from time to time, this updated parser is used in production to structure incoming logs; Online algorithms process logs one by one in a streaming fashion and update the parser on the fly. In the following, we discuss the method, the opportunities, and the limitations of state-of-the-art parsing solutions.

Offline parsing

In surveys, we find algorithms based on different techniques such as clustering (LKE [START_REF] Fu | Execution anomaly detection in distributed systems through unstructured log analysis[END_REF],

LogSig [START_REF] Tang | Logsig: Generating system events from raw textual logs[END_REF] and LogMine [START_REF] Hamooni | Logmine: Fast pattern recognition for log analytics[END_REF]), iterative partitioning (IPLoM [START_REF] Makanju | Clustering event logs using iterative partitioning[END_REF]), frequent pattern mining (SLCT [START_REF] Vaarandi | A data clustering algorithm for mining patterns from event logs[END_REF], LFA [START_REF] Nagappan | Abstracting log lines to log event types for mining software system logs[END_REF] and LogCluster [START_REF] Vaarandi | Logcluster-a data clustering and pattern mining algorithm for event logs[END_REF]), or heuristics (AEL [START_REF] Jiang | Abstracting execution logs to execution events for enterprise applications (short paper)[END_REF] and Logram [START_REF] Dai | Logram: Efficient log parsing using n-gram dictionaries[END_REF]). All those log parsing approaches perform multiple passes on a batch of log messages making this process is costly both in terms of memory and compute utilization.

Recently, in a context closely related to ours, Nedelkoski et al. [START_REF] Nedelkoski | Self-supervised log parsing[END_REF] proposed NuLog, a log parser based on the transformer architecture [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF]. Their method displays strong parsing accuracy on datasets coming from different applications. They also provide two case studies illustrating the relevance of log parsing to feed anomaly detection models. However, training their model requires a labeled dataset representative of the system logs statements. In practice, such a dataset is hard to obtain, and as observed by Zhang et al., log statements evolve. To remain relevant a Nulog-based log parsing system would need to be trained regularly with newly labeled data.

Online parsing

Online parsing methods handle logs in a streaming manner, enabling the processing of large datasets by reducing memory bottlenecks. Even if it can help them tune their parameters, such approaches do not require labeled data to train. To the best of our knowledge, 4 online log parsers have been proposed within the literature: SHISO [START_REF] Mizutani | Incremental mining of system log format[END_REF] and LenMa [101] which are based on clustering techniques; Spell [START_REF] Du | Spell: Streaming parsing of system event logs[END_REF] which uses the longest common subsequence algorithm to match templates; and Drain [103; 104] a log parsing approach based on a fixed depth research tree. Those methods can detect new templates on the run, and adapt their parsing to new log statements. Among them, Drain is often described as the most effective method and is cited as the recommended log parser by different studies on log-based anomaly detection [1; 29; 105].

Online parsing methods all use a memory structure to remember the intrinsic characteristics of the previously parsed logs: Spell creates and maintains an LCSmap with the different templates it found; Drain encodes new parsing rules inside a research tree; SHISO and LenMa maintain clusters of log templates. This memory structure plays an important role in the space and time complexity, and therefore the efficiency of each method (Table 4.6).

Distributed parsing

Parsing time can be a bottleneck for downstream applications. Batch parsing methods are easily distributable (e.g., Logram [START_REF] Dai | Logram: Efficient log parsing using n-gram dictionaries[END_REF]), as their knowledge base doesn't evolve after the training phase. However, they can't adapt to new log statements, and with the exception of NuLog are less effective than the best online methods according to Zhu & al. benchmark [START_REF] Zhu | Tools and benchmarks for automated log parsing[END_REF]. Solutions were built on the top of spark for quick parallelization. He et al. [START_REF] He | Towards automated log parsing for largescale log data analysis[END_REF] proposed a parallel log parsing method (POP) using some heuristic rules and hierarchical clustering. Zhu et al. [START_REF] Zhu | Tools and benchmarks for automated log parsing[END_REF] ran an extension of Drain in production for more than one year with a 90% accuracy. In [START_REF] Agrawal | Logan: A distributed online log parser[END_REF],

authors propose Logan based on the longest common subsequence approach that parallelizes the training phase.

Parsing Accuracy Metric

Parsing Accuracy (PA) is a commonly used metric to quantify the effectiveness of automated log parsing solutions. It is defined by [START_REF] Du | Spell: Streaming parsing of system event logs[END_REF], and [START_REF] Zhu | Tools and benchmarks for automated log parsing[END_REF] as the ratio of correctly parsed log messages over the total number of log messages. Parsing operations associates each log message with a template and a set of variables. A log message is considered correctly parsed if and only if its event template corresponds to the same group of log messages as the ground truth does. For example, if a log sequence [E1, E2, E2] is parsed to [E1, E2, E3], we get PA=1/3, since the 2nd and 3rd messages are not grouped. In contrast to standard evaluation metrics that are used in previous studies, such as precision, recall, and F1-measure [94; 108; 109], PA is a more rigorous metric, as with PA, partially matched events are considered incorrect.

Discussion

As outlined in the previous sections, log-based anomaly detection is an active research topic.

Recent methods are mainly based on deep learning techniques and follow the trend of advances in this field. Despite the variety of proposed approaches, all the presented methods follow a common workflow. First, they use a log parsing algorithm or an encoder structure to mine logs underlying information. Then they use a previously trained model to predict the next element of a given log template sequence. An anomaly is flagged if the predicted probability of the verified next log template of a given sequence falls below a specified threshold.

Reconstructing Log Sequences

When trying to work with existing methods in the 3DS OUTSCALE cloud computing context, we run into several shortcomings. First, we found that reconstructing log sequences is not always possible in a cloud computing environment due to the volumetry and the different layers of components.

A 1

Disk Shelf API Layer

Network layer

A 2 A 3 A 4 B 1 C 1 C 2 B 2 Known Unknown Figure 2.1: Multi-Layer Logging Architecture Figure 2.
1 describes a logging environment with 3 layers: an API layer where we can correlate logs using an identifier (e.g., request id); a network layer where we cannot have any relationship information as it can englobe multiple routers and no tracking identifier is logged; a disk shelf layer where we can link related logs using an identifier. Gray's arrows represent effective but not discoverable relationships from a logging point of view. In this context, we are not able to retrieve and order the full log sequence related to a given event such as a user asking for a specific virtual machine to be deployed.

This limitation was our main motivation for the design of Monilog, a log-based anomaly detection system capable of functioning within complex environments such as a cloud-computing platform. We wanted Monilog to operate without any knowledge about the relationship between logs making it a suitable solution for multi-source logging environments. Thanks to our access to 3DS OUTSCALE, we were able to apply our proposal to the monitoring of a real-life cloud computing platform. More details regarding this system are to be found under Chapter 5 that presents its architecture and Chapter 6 that reports experimental results regarding its ability to forecast anomalous events within components of the 3DS OUTSCALE cloud platform.

A Limited Number of Open Datasets

Except some internal systems from Microsoft, methods in the literature are evaluated on a limited pool of log datasets. We think this is due to the need for reproducibility and easy comparison with the existing methods but also to the limited number of datasets available open-source. More specifically, four datasets containing labeled anomalies are frequently used to compare approaches [START_REF] He | Loghub: a large collection of system log datasets towards automated log analytics[END_REF]:

• BGL [START_REF] Oliner | What supercomputers say: A study of five system logs[END_REF], a collection of logs issued from a BlueGene/L supercomputer system.

• Hadoop Distributed File System (HDFS) 1 , is a distributed file system that aims to provide high-throughput access to application data. This log set [START_REF] Xu | Detecting large-scale system problems by mining console logs[END_REF] was generated in a private cloud environment using benchmark workloads and manually labeled through handcrafted rules to identify the anomalies.

• Hadoop2 is a big data processing framework that allows for the distributed processing of large datasets across clusters of computers using simple programming models. The associated dataset [START_REF] Lin | Log clustering based problem iden-REFERENCES tification for online service systems[END_REF] was collected from a Hadoop cluster with 46 cores across five machines running two different test applications. Failures were manually injected into parts of the dataset to simulate machine down, network disconnection or disk full issues.

• OpenStack3 is a cloud operating system that controls large pools of computing, storage, and networking resources throughout a data center. This dataset [START_REF] Du | Deeplog: Anomaly detection and diagnosis from system logs through deep learning[END_REF] was generated on CloudLab, a flexible, scientific infrastructure for research on cloud computing.

The sensitivity of information contained in logs from real environments often makes their publication infeasible, moreover, labeling anomalies in such datasets is a tedious and timeconsuming task. We believe this explains why such a small pool of open-source log traces has been studied through the years. One of the drawbacks of this phenomenon is that it becomes complicated to evaluate the relevance of the proposed methods in more complex log environments. Most of the previously presented methods achieve a 0,94+ precision on each of the considered datasets, therefore it can be hard to quantify the contribution of a method compared to the existing ones.

Anomalies affecting cloud computing platforms can be way more complex than the one listed in the open-source datasets. They can affect a vast range of interconnected components and therefore may require a multi-log traces approach to be detected [START_REF] Vervaet | Monilog: An automated log-based anomaly detection system for cloud computing infrastructures[END_REF]. Making a log dataset available for study is a challenging task as log data provides all the details about the execution of the infrastructure components, and the misuse of this data may cause serious problems. Due to confidentiality issues and strict security policies, it is not possible to open-source logs coming from 3DS OUTSCALE internal services. However, we paid attention to providing the maximum of information regarding the complexity of its logging environment and the lesson learned. We hope our results will help to understand the challenges raised by the monitoring of large online systems such as the cloud computing platforms and the complexity of their logging environment.

CHAPTER 3

Online Log Parsing Methods and Hyperparameters Tuning

Log Parsing for Log-Based AD Methods

Log parsing plays a crucial role in extracting valuable information embedded within logs and is an essential step for enabling most existing log-based AD methods. Over the years, there have been numerous methods proposed in this active research area, as documented by Zhu et al. [START_REF] Zhu | Tools and benchmarks for automated log parsing[END_REF] in their benchmark. They conducted an evaluation of 13 log parsers across 16 datasets, and Drain emerged with the highest overall performance, reaching an average PA of 0.865. However, it is important to note that the performance of parsing methods can vary greatly across different datasets. For example, Drain achieved a PA of 1.0 on the HDFS dataset, but a significantly lower PA of 0.527 on the Proxifier dataset.

In this section, we focus on investigating the impact of PA on log-based AD methods. We also explore the influence of hyperparameters and preprocessing techniques on PA.

Impact of the Parsing Accuracy on Anomaly Detection Methods

In order to assess the impact of PA on the accuracy of downstream AD methods, we conducted an evaluation using an instance of Deeplog [START_REF] Du | Deeplog: Anomaly detection and diagnosis from system logs through deep learning[END_REF]. The evaluation involved utilizing the same log dataset, but parsed with three different PAs. For this purpose, we employed a Drain instance as the parsing solution and deliberately varied the PA by adjusting hyperparameter values.

The dataset selected for this evaluation consists of HDFS system traces [START_REF] He | Loghub: a large collection of system log datasets towards automated log analytics[END_REF]. This particular dataset was chosen because it is commonly used by both the authors of Drain [START_REF] He | Drain: An online log parsing approach with fixed depth tree[END_REF] and Deeplog in their respective research papers. By applying different PA levels to the parsed logs and subsequently evaluating the performance of Deeplog, we aimed to examine how variations in parsing accuracy may impact the overall accuracy of the downstream AD method.

To investigate the relationship between parsing quality levels and the accuracy of downstream anomaly detection methods, we conducted training on 10 separate instances of Deeplog for each parsing quality level (1.0, 0.8, 0.65). In a study conducted by Zhu et al., it was found that the Drain parsing accuracy fell below 0.8 for 6 out of the 16 considered datasets. Furthermore, other online parsers examined in the study, such as Spell (9 out of 16) and LenMa (7 out of 16 with a PA as low as 0.174 on the HealthApp Dataset), also demonstrated lower PA values for a significant proportion of the datasets. Additionally, a more recent empirical study by Fu et al. [START_REF] Fu | An empirical study of the impact of log parsers on the performance of log-based anomaly detection[END_REF] further confirms the impact of log parsing on the performance of log-based anomaly detection methods. They highlight that log-based AD methods tend to exhibit poorer performance on datasets with higher numbers of underlying log templates, indicating the challenge posed by diverse datasets with a large number of underlying templates.

Parsing Errors

The results obtained from our experiments highlight the significant influence of log parser precision and accuracy on the overall performance of the parser/anomaly detection model system.

The presence of parsing errors introduces new log templates, which in turn artificially increases the input vector of deep learning models, resulting in reduced learning quality. Furthermore, we observed that the time complexities of existing online log parsing approaches are directly affected by the number of templates discovered. In systems with diverse logging structures, the parser's processing speed can be noticeably slower. This aspect becomes particularly critical for systems that require real-time processing, such as AD systems operating in complex logging environments.

In our specific case, where efficiency is a crucial factor in achieving real-time processing, log parsing efficiency becomes a sensitive issue. The ability to process logs efficiently and in a timely manner is of utmost importance for AD systems operating in complex logging environments. Table 3.1: Online Log Parsers Presented in [START_REF] Zhu | Tools and benchmarks for automated log parsing[END_REF].

Online Log Parsers and Hyperparameters Tuning

Two notable studies conducted by He et al. [START_REF] He | An evaluation study on log parsing and its use in log mining[END_REF] and Zhu et al. [START_REF] Zhu | Tools and benchmarks for automated log parsing[END_REF] have focused on evaluating the accuracy and robustness of log parsing solutions using log traces from various applications.

These studies provide valuable insights into the potential applications of existing methods for online log processing. The evaluated methods commonly employ a preprocessing step to filter out certain tokens and can be fine-tuned using various hyperparameters (as depicted in Table 3.1).

In this section, our focus shifts towards examining the influence of these two factors, the preprocessing step and the selection of hyperparameters-on the PA of two state-of-the-art log parsing solutions: Spell and Drain.

Preprocessing

Preprocessing is a well-established initial step in log parsing processes. Its purpose is to filter out specific tokens based on user-defined regular expressions (regexes) that represent commonly encountered variables, such as IP addresses, file paths, or URLs.

During our analysis of logs from a 3DS OUTSCALE internal service, we made an interesting observation. A pproximately 60% of the tokens within log messages originated from JSON or XML-formatted data. In API-like services, it is common practice to append such formatted data at the end of a log line, as it provides valuable context to understand the log entry (e.g., "Send 42 bytes to 121.13.4.26 {user id=125, service name=dart vader}"). Given that JSON and XML are already structured formats, it becomes advantageous to exclude these parts from the parsing process. By doing so, we can reduce the average length of log messages and enhance parsing efficiency and accuracy.

Experimental Motivations

Given the potential impact of preprocessing and parameter choices on the precision and efficiency of log parsing approaches, the selection of appropriate values becomes crucial. However, in many industrial contexts, evaluating the parsing accuracy can be challenging due to difficulties in labeling the data. Factors such as the evolution of log statements and limited access to source code make it complex to determine the optimal parameter values. In our study, we specifically address the following research questions:

• RQ1. What is the impact of preprocessing and hyperparameter tuning on the precision and accuracy of log parsing?

• RQ2. Is it possible to identify generic hyperparameter values that ensure a good parsing accuracy across different log datasets?

By addressing these research questions, we strive to provide insights and guidance for practitioners in selecting optimal preprocessing techniques and hyperparameter values, ultimately 3.2 Online Log Parsers and Hyperparameters Tuning enhancing the precision and efficiency of log parsing approaches. For our study, we have chosen to evaluate the performance of Drain and Spell, which are considered the top-performing online log parsing solutions according to the benchmarks conducted by Zhu et al. [START_REF] Zhu | Tools and benchmarks for automated log parsing[END_REF]. and He et al. [START_REF] He | Experience report: System log analysis for anomaly detection[END_REF]. To assess their performance, we have utilized a dataset consisting of 2 000 labeled log events from three different systems: Android, HDFS, and OpenStack (as shown in Table 3.

Experimental Context

2).

OpenStack is a cloud solution that enables the deployment of Infrastructure as a Service (IaaS) platforms. The Android dataset represents logs from the popular mobile operating system developed by Google, encompassing various patterns and logs related to resource allocation, process management, network management, and more. The Hadoop File System (HDFS) dataset, widely used in the literature, serves as a practical baseline due to its simplicity, featuring a lower number of patterns and variable parts. For further details regarding these open-source datasets and their collection methods, we refer to the paper by He et al. [START_REF] He | Loghub: a large collection of system log datasets towards automated log analytics[END_REF], which provides comprehensive information. By evaluating Drain and Spell on these diverse datasets from different logging environments, we aim to gain insights into the performance and effectiveness of these log parsing solutions in real-world scenarios.

Impact of the Hyperparameters on the Parsing Accuracy

Here, we explore the impact of the values of the hyperparameters on the Parsing Accuracy (PA)

of Spell and Drain to know if generic values are possible (RQ1.). For Spell, there is a single hyperparameter, τ ∈ [0, 1], which serves as a threshold to determine whether a log belongs to an existing known pattern. Drain, on the other hand, has two hyperparameters: the depth of its search tree (depth ∈ N * ) and a threshold (ST ∈ [0, 1]), which determines if a log belongs to an existing group.

The impact of hyperparameter values on the parsing accuracy is illustrated in Figure 3.2.

The optimal value of τ that maximizes Spell's accuracy varies across datasets: 0. in OpenStack and Android. In the case of Android, the maximum precision achieved is 0.75, while for the other two datasets it reaches 0.9. The optimal depth value varies: 5 for Android, 6 for HDFS, and 7 for OpenStack. It is evident that the accuracy of both solutions is influenced by the choice of hyperparameter values, and finding an appropriate combination of values for a given context is a non-trivial task.

Impact of the Preprocessing on the Parsing Accuracy

Table 3.4 presents the results obtained for each dataset and method, with and without preprocessing. We evaluated the parsing accuracy and also considered the number of patterns to detect any potential overclassification of logs. The hyperparameter values used were the ones that resulted in the highest PA with preprocessing. The regular expressions used for this experiment were those proposed by Zhu et al. in their benchmark [START_REF] Zhu | Tools and benchmarks for automated log parsing[END_REF].

For Spell, the PA is significantly increased by a factor of at least 1.5 when preprocessing is applied. The improvement is even more pronounced for the HDFS dataset, with a factor of 3.6. The number of templates also benefits from the preprocessing step, bringing it closer to the actual number: 166 for the Android dataset, 14 for the HDFS dataset, and 43 for the OpenStack dataset.

For Drain, the results vary depending on the dataset. While there is an improvement in PA for the Android dataset, the PA and the number of patterns worsen for the OpenStack dataset when preprocessing is applied. This drop in PA can be attributed to the presence of tokens in tokens affect one level of the partitioning tree. However, with preprocessing, these tokens are split into several parts, affecting multiple levels of the tree and resulting in a PA of 0 for any log containing this type of word. Spell is unaffected because its accuracy on words following this pattern is already zero.

Discussion

Based on our study, we can conclude that the selection of hyperparameter values and the implementation of preprocessing regexes have a significant impact on the Parsing Accuracy of log parsers. We have also highlighted the influence of PA on the Accuracy of AD methods, as a low PA can significantly degrade the AD potential of downstream models. It is crucial to find optimal hyperparameter values and appropriate regexes to improve the performance of log parsing.

However, finding the best hyperparameter values and regexes can be challenging, especially in complex logging environments with a high volume of data and multiple applications involved.

In the case of the 3DS OUTSCALE logging infrastructure, it is not feasible to label logs with their templates manually, which adds to the difficulty of fine-tuning existing parsing solutions.

Based on the obtained results, we are motivated to explore and propose a more robust log parsing solution in the next chapter. This solution aims to address the challenges faced in enabling log-based anomaly detection methods in a cloud computing context.

USTEP: Unfixed Search Tree for Efficient Parsing

Motivations

This section provides a comprehensive overview of our online log parsing method called USTEP (Unfixed Search Tree for Efficient Parsing) [31; 33]. USTEP was specifically designed for log parsing in large-scale traces generated by cloud infrastructure, with a focus on low latency and high robustness. It is worth noting that USTEP is currently the only parsing method known to achieve constant time complexity.

In our comparative evaluation (detailed in Section 4.2), we compared USTEP with four stateof-the-art online parsing methods using 10 datasets derived from real-world applications. The results of our evaluation demonstrate that USTEP outperforms the other methods by improving parsing accuracy by 3% and enhancing robustness by reducing the quartile inter-distance by half when compared to Drain.

The development of USTEP was motivated by the identified shortcomings in terms of robustness and efficiency of existing parsing methods, as discussed in the previous chapter. By addressing these limitations, USTEP aims to provide a more reliable and efficient log parsing solution for handling the complexities of cloud infrastructure log traces.

Detailed Workflow

USTEP employs an evolving tree structure to discover and encode parsing rules while performing online log parsing. A USTEP instance is defined as U = {P, σ, ϕ}, where P = {V, E, E} represents the parsing tree, σ ∈ [0; 1] is the similarity threshold that determines when a log is considered to match a template, and ϕ ∈ N * is the maximum number of templates that can be associated with a leaf node. V = {υ k } N k=1 is the set of nodes, E the set of discovered templates, and E ⊂ V × E the set of node-template links.

Evolving Research Tree Structure

The USTEP parsing tree is constructed using four types of elements, as depicted in In the USTEP research tree, each leaf node stores a set of discovered templates while root and internal nodes encode parsing rules. The parsing rules in USTEP differ based on the type of node. Root and internal nodes encode parsing rules, while leaf nodes store a set of discovered templates. These rules are utilized during the traversal of the tree to locate or identify a template that matches the given log message.

Each node υ j , which has K j children, is associated with a key function υ j .key(t i ) and a list of results r 1 , . . . , r K j , where each result r k corresponds to the k-th child of the node. In the case of the root node υ 1 , the key function is defined as υ 1 .key(t i ) = |t i |, which represents the number of tokens in the log message. On the other hand, for inner nodes, the key function is υ j .key(t i ) = t pivot j , where pivot indicates the specific position used for splitting.

(leaf i , (e i , v i )) ← SelectClosestLeaf&Parse(t i ) ▷ Step 1 3: UpdateTree(leaf i ) ▷ Step 2 4:
return (e i , v i ) 5: end function Parsing a tokenized log message t i using USTEP is a two-step process (Algorithm 1): 1/ Identify the template e i and the variables v i matching t i using descent rules; 2/ Refine the tree structure by dividing saturated leaves.

v ← (t i j : j = 1 . . . |t i | if e j = *) return v 14: end function
In Methods 2, we describe three utility functions used to work with log templates. Initial-izeTemplate to create a template based on an incoming log, UpdateTemplate to refine a template by filtering out position-wise non-identical tokens between the incoming log and an existing template, and getVariables to extract the variables list from a log once the matching template is known.

simF (t i , e) = |t i | j=1 equ(t i j , e j ) |t i | (4.1) equ(t, t) =        1 if t = t 0 otherwise (4.2)
Methods 3 Searching methods function SelectClosestLeaf&Parse(t i )

2:

leaf i ← TreeDescent(υ 1 , t i ) (e i , v i ) ← SelectTemplate(leaf i , t i ) 4:
return

(leaf i , (e i , v i )) end function 6: function TreeDescent(υ j , t i ) 8:
while not υ j .isLeaf () do if ∃k : r k = υ j .key(t i ) for k = 1, . . . , K j then 10: return (e i , v i ) end function

υ j ←
The main search function is SelectClosestLeaf&Parse (Methods 3). As templates are attached to leaves, to assign a template to a log t i , the first step is descending the tree using embedded rules. Once a leaf is reached, the list of associated templates is scanned to pick the one matching t i , if none apply, a new template is created based on t i and added to the tree structure (Lines 12 and 22). The similarity scores between a log and the templates attached to the selected leaf are computed as described in 4.1. simF (t i , e) represents the proportion of positions-wise identical tokens between t i and a template e. The template with the highest similarity score above σ is picked to represent the log. If no score above σ is found a new template is created using the current log. and is therefore saturated. The position with the highest number of unique tokens is p = 3 with 3 different ones (bytes, packages, packets). Using this pivot, the leaf is transformed into an internal node with key = t i 3 , and three new leaves are created, one for each unique word at p. The existing templates are dispatched to the new leaves according to their token value at position 3.

A Comparative Evaluation of Online Log Parsers

To evaluate the relevance of USTEP regarding the existing online parsing algorithms, we conducted a theoretical and an experimental evaluation of its accuracy, robustness and efficiency [START_REF] Vervaet | Ustep: Unfixed search tree for efficient log parsing[END_REF]. We selected 14 datasets coming from a wide panel of real-world applications: 12 comes from openly accessible online sources [START_REF] He | Loghub: a large collection of system log datasets towards automated log analytics[END_REF], 1 from 3DS OUTSCALE infrastructure (Internal-1 ), and 1 was crafted by combining multiple log traces (Big-comp). 3DS OUTSCALE dataset represents 30 minutes of consecutive logs issued only from sources related to TINA OS the 3DS OUTSCALE Cloud orchestrator. Big-comp was crafted using Loghub [START_REF] He | Loghub: a large collection of system log datasets towards automated log analytics[END_REF] open source datasets and represents a multi-source logging environment which is to be expected for a cloud computing platform. The 12 other datasets are extracted from the Loghub platform [START_REF] He | Loghub: a large collection of system log datasets towards automated log analytics[END_REF], a bank of openly accessible log traces. We summarize the datasets characteristics in Table 4.1. In some cases, the ground truth is unknown (HDSF-extended, OpenStack-extended, 3DS OUTSCALE ), so, some metrics cannot be calculated. To prove the effectiveness of USTEP, in this section, we compare its performance with four state-of-the-art online log parsers in terms of accuracy and efficiency.

Experimental Setup

• Drain [START_REF] He | Drain: An online log parsing approach with fixed depth tree[END_REF] is an online log parser based on a fixed-depth research tree structure. It is used by many log-based applications [24; 29] and is presented as the best parsing solution within Zu & al benchmark [START_REF] Zhu | Tools and benchmarks for automated log parsing[END_REF].

• LenMa [101] uses the distribution patterns within token length to group logs in an online fashion.

• SHISO [START_REF] Mizutani | Incremental mining of system log format[END_REF] uses a tree with a predefined number of children per node to guide its log grouping process.

• Spell [START_REF] Du | Spell: Streaming parsing of system event logs[END_REF] The experiments we present here were carried out on a CentOS Linux 7.8 Cloud Hosted virtual machine with 32 hearts, and 62 GB of RAM. We have made the implementation of each log parser used publicly available and linked to the datasets. However, due to security and confidentiality concerns, we have kept Internal-1 private.

The considered algorithms process logs online, and for fairness of comparison, we applied the same processing rules to each log parser. Used values of σ and ϕ for each dataset are summed up in Table 4.3. For space reasons, we didn't detail here the settings for all the other methods but they are available with the source code of USTEP 1 .

Effectiveness of the Log Parsers

Traditionally log parsing is evaluated using the Parsing Accuracy (PA) [ truth. Given the potential impact of errors on the performance of downstream applications, accuracy is a crucial aspect. For this study, we employed our ten labeled datasets. The parameters of all the log parsers are fine-tuned through over 100 runs to avoid bias from randomization, and we reported the best result for each dataset in Table 4.4.

Our proposal method, USTEP exhibits great performance, achieving the highest PA on eight out of ten datasets and closely approaching the best result on the remaining two. Overall, USTEP has the best average PA at 0.937, 3.4% higher than the second best (Drain with 0.865).

USTEP's better results are due to several intrinsic characteristics. First, the leaf division process allows it to choose the best token as a descend rule. Where Drain always selects the n-th token with the risk of selecting a yet-undiscovered variable leading to a branch explosion, splitting logs from the same log statement over different leaves. Besides, the τ parameters allow fine-tuning over different types of datasets.

USTEP and Drain appear to be the on average two most accurate methods on the considered datasets. Production environments are often more complex and englobe a large range of logging sources. Big-comp dataset was crafted using parts of each of the other 10 datasets to represent such an environment. USTEP achieves a 0.816 parsing accuracy over Big-comp dataset, this is 0.336 more than Drain (0.480). We believe this significant difference to be due to the way both algorithms manage their tree structures. Drain uses the depth-first tokens as descent rules, if a non-numeric variable is part of those, it will create a new leaf for each unique value it takes.

On the other side, USTEP descent rules are crafted thanks to the accumulation of templates allowing it to limit the case where a variable sensitive position is selected for a descent rule allowing it to handle more diverse datasets. We believe that a general parsing method must deliver robust performance on diverse datasets for it to be utilized in production environments. Our goal here is to illustrate USTEP's capacity to support a broad range of log data types. In Figure 4.4, we plot the accuracy distribution of each method. The log parsers are arranged in ascending order of the median PA. SHISO is on the lowest side and USTEP is on the highest one. Achieving consistently high PA values across a diverse range of datasets is critical for the general use of a log parser. Regarding this, despite their great average PA (Table 4.4), the considered log parsers have a large variance when applied to logs coming from different types of logging environments. The 1st to 3rd quartile distance for SHISO is 0.23, 0.146 for Drain, 0.079 for Spell, and 0.073 for USTEP which also has the highest median at 0.959 PA. The robustness of USTEP within diverse environments is supported by its achieved PA on Big-comp dataset. and USTEP were the only methods able to process the 3DS OUTSCALE, and Big-comp datasets in less than a day. The effectiveness of each method varies with the dataset; however, both present a linear cumulated parsing time on all the datasets. 4.5, we have consigned the memory used in MB by each parsing method after parsing a dataset. The results obtained show no significant deviation in the memory used by the methods. Used space seems to be most impacted by the number of underlying templates as well as the number of tokens in the logs of the dataset rather than by the parser memory structure.

Robustness of the

Memory Usage of Log Parsers

Complexity Analysis

We analyze the worst-case time and space complexity to process a single log for Drain, Spell, LenMa, SHISO and USTEP.

Theorem 1. Let's denote by t i an incoming tokenized log, |t i | the number of tokens constituting t i , |t| the highest number of tokens for a single discovered template, T the number of unique templates already discovered by the parser using a structure with N nodes. We have the time and space complexities in the worst-case scenarios summarized in Table 4.6.

O(|t i |) O(T × N ) Drain O(T × |t i |) O(T × N ) Spell O(T × |t i | × |t|) O(T ) LenMa O(T × |t i |) O(T ) SHISO O(T × |t i |) O(T × N )
Proof. We summarize the parser structures to determine complexities. USTEP and Drain use searching tree structures to determine the closest leaf, once the leaf is identified, a list of templates is evaluated to find the most similar. Search tree structures are more memory-intensive but help to reduce processing time. For instance, while Spell exhibits a linear space complexity with respect to the discovered templates, its time complexity is quadratic in terms of the maximum token length. On the other hand, USTEP presents a larger space complexity but it is the only method with a constant worst-case time complexity.

For OpenStack-extended dataset, Drain performs worse than USTEP. Figure 4.6 which illustrates the processing time for each incoming log, depicts a trend of increasing processing times for Drain. As Drain doesn't bound the number of templates attached to a leaf, in the long run this increases the processing time as more templates are discovered. USTEP maintains a more stable processing time thanks to ϕ parameter that limits the number of similarity comparisons.

USTEP-UP

The time complexity analysis of USTEP displays demonstrates a consistent parsing time relative to the number of tokens inside the message. Results of our experimental evaluation support this claim. However, within fast-paced ou multi-sources logging environments, even a constant parsing time can be a limitation for real-time parsing. We address this scalability issue, by presenting in this section USTEP-UP, a way of running multiple slightly modified USTEP-like instances in parallel.

Agrawal & al [START_REF] Agrawal | Logan: A distributed online log parser[END_REF] used an LCS-based log parsing algorithm similar to Spell and distributed it on top of Spark. Still, their evaluation displays a lower bound in parsing time due to the

USTEP-UP

increased overhead induced by the knowledge merge steps between the instances. A similar approach applied to Drain would run into an even bigger problem, as it's more than likely that the tree structures of the instances will diverge, leading to inconsistent parsing where different templates can be associated with the same log message depending on the instance parsing it.

This inconsistency is a problem because identical messages could appear as being different for downstream applications.

Like Drain instances, USTEP ones are stateful and need to share information; otherwise, they might encode different parsing rules. To address this, USTEP-UP runs modified USTEP instances. Those skip the leaf division step and never divide saturated leaves. A monitoring instance is added to the architecture, it spreads the knowledge and refines the instances by punctually merging its memory structure with theirs. q ← FIFOQueue()

3:
q.enqueue(υ 0 1 , υ l 1 )

4:

while not q.isempty() do 5:

(node1, node2) ← q.dequeue()

6:
for child in node2.childrens do 7:

if not child.isLeaf() then 8:

q.enqueue((node1.getChild(child.label), child)) return P 0 24: end function a First in First Out (FIFO) Queue to iterate over all the nodes of P l in a breadth-first fashion.

Crossed Internal nodes are enqueued paired with their equivalent in P 0 . When encountering a leaf node, if no child with the same label exists in P 0 , the leaf is added to the currently pointed node in P 0 . Otherwise, all the templates from the P l leaf are parsed through P 0 starting at the current node and not at the root node. MergeTree(P 0 , P l ) function time complexity is in O(| E|), as we need to iterate over every template within P l and parse them with P 0 . As this function performs operations in place, its space complexity is in

O(|V 0 | × | E 0 | + |V l | × | E|).
Figure 4.8 illustrates the merging process between the knowledge tree P 0 (Tree 1) and an instance tree P l (Tree 2). In this example, we consider σ = 0.5 ϕ = 3. The process starts by visiting (υ 0 1 , υ l 1 ), the two root nodes. In this example, υ l 1 is linked to one leaf node υ 1 1 , and

υ 1 1 .label = 5.
As there is already a node with node.label = 5 inside υ 0 1 .children, all the templates attached to υ1 1 are parsed by υ 0 1 . υ 1 1 .templates return four elements: 1/ "Send * bytes to *" will end up in the left leaf according to the descent rules as simF("Send * bytes to *", "Send * bytes to 1.2.3.4") = 0.8, it will update the template within P 0 .2; 2/ "Send 42 packets to *" will be affected to the already more refined template "Send * packets to *"; 3/ "Received * packages from *" will create a new leaf as no leaf with label = "package" exists yet. 4/ "Received * packets from *" will end up in the same leaf as "Send * packets to *" as a new template.

Perspectives

By separating variables from the template, the parsing process opens a new source of information for intelligent monitoring, and it is a required step for log-based anomaly detection methods.

With the proposition of USTEP -a high-performing online log parsing algorithm in terms of accuracy, robustness, and effectiveness, along with USTEP-UP, an architecture designed to run multiple parallel USTEP instances, we effectively tackle the challenge of parsing logs in environments characterized by high log statement evolution and huge data volume.

One limitation of USTEP is that it can't group logs sharing the same template but with a different number of tokens. This limitation is inherent to its memory structure and it's shared with Drain. Still, our comparative evaluation on 14 datasets displays that USTEP outperforms existing methods in terms of PA and robustness. We believe that progress in the parsing field can stimulate progress in many log-based applications and not just the anomaly detection ones as it allows using log message embedded information for a wide variety of tasks. Also, we discovered that parsing algorithms like USTEP can end up having unusual applications such as the classification of JIRA 1 tickets or the extraction of invoice-embedded information by some 3DS OUTSCALE teams.

Introduction

In this chapter we present Monilog, a log-based alert system architecture designed to detect anomalies at a cloud scale. It is able to produce comprehensive reports regarding the reported anomaly including the concerned system and applications. Unlike existing methods depicted in Chapter 2, Monilog operates without needing specific knowledge of the relationship between logs, thereby facilitating near real-time alerts through a streaming mechanism.

We start by exploring the architecture of Monilog and our work toward implementing Monilog inside 3DS OUTSCALE infrastructure. We successfully used this implementation to monitor equipments from the European region of 3DS OUTSCALE during 11 consecutive days. Details about this experimentation, its context, and the obtained results are to be found under Chapter 6. 2) It seeks anomalous events and generates alert candidates (in white); The last step (in yellow), consolidates alert candidates by grouping related ones and acquiring their context (Section 5.2.3). On the architectural schema (Figure 5.1), we indicate 7 components enumerated from 1 to 7, within this chapter, we will refer to those using a number within curly braces (e.g. {1} refers to the raw logs message queue).

Monilog Architecture

Vectorizing Logs

Log Filtering

Logs are timestamped and generated by machines at runtime. Monilog input({1}) is an infinite and temporally ordered sequence of tuples L = ((t i , l i ), i = 1, 2, ...), with t i the timestamp associated to a log l i . In our implementation of Monilog, this input {1} is a Kafka1 stream exposing the last 3 hours of logs. With Kafka being a distributed event streaming platform presenting good performance in terms of scalability and fault tolerance.

In first-hand, logs are filtered based on header embedded criteria ({2}). Filtering is an optional step that allows you to retain only the logs linked to the system you want to monitor.

For instance, in our experimental use case around serves crashes (Chapter 6), we choose to retain only logs issued by Kernel Virtual Machines from 3DS OUTSCALE European cloud region.

Template Mining

We define the log event template (e) to be the combination of the application name and the severity of the considered log message. For instance, the template of the raw log presented in In Chapter 4, we presented our proposal log parsing method USTEP that outperforms ex-isting log parsers in terms of accuracy and robustness. However, USTEP still makes parsing errors and rarely achieves 1.0 PA on log datasets. To avoid dealing with falsely parsed templates we choose to craft templates using only header-embedded information as this is a trivial and non-error-prone operation. Log parsers can generate useful information based on the log messages. We believe that in the future, replacing the proposed template mining step by a log parser such as USTEP could improve the system performances, although this requires dealing with the induced potential parsing errors.

Log Aggregation

Logs with the same host and template are aggregated by w nanoseconds windows. We define L h,e (t) as the count of logs associated to template e for host h at timestamp t, and the logs count window as a time serie C h,e (w).

C h,e (t) = i∈[t,t+w] L h,e (i) (5.1)
The tuples of size 4 are then queued within another message queue {3} following the shape (t + w, h, e, C h,e (t)). Messages queued into {3} are to be consumed by monilog-brain {4} component, in charge of identifying alert candidates within the log series associated to each host.

We separate the logic of the monilog-log-consumer component {2} and monilog-brain component {4}. By doing so, the structure becomes more modular and it is easier to enforce new filtering rules or to experiment with different anomaly detection models. In our practical implementation, the monilog-log-consumer is implemented using Spark1 , an engine for executing computing tasks on a cluster of machines. Having its logic decoupled from the monilog-brain component allowed us to take full advantage of Spark streaming features.

Vectorization

monilog-brain {4} component reads the message queue {3}, and writes the obtained tuples in a cold storage database {5}. This database aims to provide a convenient way of replaying specific sequences to test new models or to inspect suspicious results. In our implementation of the Monilog system, we choose InfluxDB1 , a high-speed read and write database designed for time series. We also built a visualization module {6} on top of {5}, allowing us live monitoring of interesting metrics such as the log count per time window or the time delay or the prediction error of the different models over time.

When training models, every unique log template within the training set is associated to an id (see template mining step). We define T as the ensemble of unique log templates within the model training set, and f e : T → 1, 2, .., |T|, a bijective function. The id e i of log template is defined by the following equation:

e i =        f e (i) if i ∈ T |T| + 1 otherwise (5.
2)

The event count vector V (t) is constructed as follows:

V (t)(i) = e i =i
C h,e (t) (5.3)

|V (t)| = |T| + 1 (5.4)

Detecting Anomalies

Model Predictions:

Models forecast the log traffic using a matrix regrouping r previous observations. Their output is a prediction for the next log count vector, each model can be assimilated to a prediction

Anomaly Consolidation

Context Acquisition

Forecasting error metrics are computed using the forecasted vector and the observation vector.

Each record inside one of those vectors is linked to an identified template. It is therefore possible to quantify the influence of a given template over the final error. This information is used to retrieve the most contributing templates to the total error. Since the templates are forged using the application name and the severity, reverting this process enables us to identify the application and severity of the templates that contribute most to the errors.

Alert Consolidation

{ " Timestamp " : " 2 0 2 2 -1 1 -0 2 1 0 : 3 2 : 0 0 " , " Data Centers " : [ { " Name " : " DC -1 " , " Hosts " : [ { " Name " : " Host -1 3 " ,

Classification of Log-Based Anomalies

Within the Litterature

Anomalies encompass various types of events, ranging from breakdowns to performance issues, including security attacks. These anomalies possess distinct characteristics that often necessitate the involvement of different teams, each with their own set of priorities. Effectively classifying a detected anomaly and providing contextual information facilitates the identification of the most appropriate team to address the issue at hand. In our review of the literature, we discovered three studies that explore the automated classification of log-based anomalies.

Zou et al. proposed Uilog [START_REF] Zou | Uilog: Improving log-based fault diagnosis by log analysis[END_REF] (2016), an anomaly classifier that built a fault keyword matrix to classify anomalies. This matrix encodes the probability of a given token to appear in each fault type. This matrix is dynamically learned during the training phase thanks to labeled anomalies. Author's evaluation on classic error affecting CentOS based virtual machines in a cloud illustrates the relevance of their proposal to correlate tokens with specific anomalies.

With WEAC [START_REF] Pande | Weac: Word embeddings for anomaly classification from event logs[END_REF] (2017), Pande et al. also exploit token embedded information to detect and classify anomalies by using word2vec techniques. Authors illustrates the superiority of their method comarent to Isolation Forest [START_REF] Liu | Isolation forest[END_REF] (IF), and HDBSCAN [START_REF] Campello | Hierarchical density estimates for data clustering, visualization, and outlier detection[END_REF], two traditionnal mahcine learning techniques.

LogClass (2018), by Meng et al., is a classifier trained over log anomalies [START_REF] Meng | Device-agnostic log anomaly classification with partial labels[END_REF]. Authors applied their proposal to a set of switch logs issued from data centers to detect and classify common errors (e.g., FAN FAILED, POWER DOWN, SYSTEM REBOOT). Logclass approach is based on the bag-of-words technique and learns anomalies labels based on the frequently associated tokens.

A common limitation of these approaches is their reliance on manually established labels to train their classifiers. Acquiring such labeled data is a laborious task, and the resulting trained classifier may not be adaptable to newly detected anomalies. Consequently, these approaches become challenging to maintain, which, in our view, is a key factor contributing to the lack of active research on log-based anomaly classification.

With Monilog

Classifying anomalies can be a subjective task, different organizations may have different ways of handling the same type of events and will assign them different priorities based on their business model. Therefore, instead of trying to group anomalies around known errors, we focused on a way to bring up as much relevant information as possible about an anomaly.

Unlike other approaches, Monilog does not incorporate a classifier module. Instead, when it detects a potential anomaly, it generates a comprehensive report containing relevant information.

As depicted in Section 5.2.3, this report includes the concerned equipment, the applications responsible for the anomalies alongside the criticity of the logs linked to those applications. The proportional contribution to the error score for each log template (appname + criticity) is also provided.

Organizations can easily create a tailored anomaly handling process on the Monilog platform by leveraging business rules and expert knowledge. An example at cloud scale could be for instance to group anomalies occurring at the same time if they concern equipment inside a given subnet/data center. One could also use the involved applications to create error handling scenarios based on identified error templates. We believe the flexibility offer by Monilog to be a key factor for his adoption by industrial practitioners in production.

Perspectives

In this chapter, we presented Monilog, an automated system for detecting anomalies based on log data. Monilog is specifically designed to address the challenges posed by cloud computing platforms, which often consist of numerous complex components of diverse types and undergo frequent updates reflected in log statements. One of the key strengths of Monilog is its capability to generate comprehensive statements that provide detailed information about the detected anomalies and the specific components involved. This inherent flexibility offered by Monilog has been instrumental in its adoption by industrial practitioners operating in real-world production environments.

Introduction

To assess the practical applicability of our proposal, we implemented a version of Monilog using the logging infrastructure provided by 3DS OUTSCALE and tested it at a cloud region scale. The implemented architecture was running three different models and three different error metrics detailed below. It allows us to validate the real-time performances of our approach as well as its relevance to forecast anomalous events and provide useful insight regarding the anomalous periods.

In this section, we provide a comprehensive overview of the logging environment, collection mode, experimental context, selected models, and metrics. We also discuss their performance in forecasting anomalous events. We have focused our assessment work around two research questions:

• RQ3. What is the precision of each of the considered model/metric pairs for the detection of abnormal events ?

• RQ4. Can the considered model/metric pairs forecast the major events that impacted the monitored systems during the experimental period ?

In Section 6.3.1, we share and comment the result of a precision evaluation of the tested model/metric couples. We used manually generated labels regarding the relevance of the identified anomalous candidates. The section 6.3.3 is dedicated to RQ4, as we explore the capability of our proposal to forecast three critical anomalous events that occurred during the experimental period.

Experimental Setup

Dataset

A Kernel Based Virtual Machine or KVM refers to a Linux system turned into an hypervisor that allows a host machine to run multiple isolated virtual environments, called virtual machines.

A KVM exposes four types of resources to its hosted virtual machines: Computing resources, memory, network access and storage access. Server hard failures are rare events but they can have a negative impact on the hosted client virtual machines.

During 11 consecutive days, our Monilog implementation was reading a Kafka stream filled by syslog messages issued from all the Kernel Based Virtual Machines (KVMs) linked to the 3DS OUTSCALE European cloud region. Several billion lines of logs were consumed during the monitored period at the average rate of 238 logs per host per minute.

Anomalies

During the monitored period, operational teams reported and timestamped three severe crashes.

However, during the same period, Monilog alerting system generates a higher number of alert candidates. To evaluate the relevance of Monilog architecture as well as the performance of each model/metric combination, we ran a manual inspection of all the reported abnormal events to assign them one of the following label: Probable Anomaly (PA) for the legitimate abnormal events; False Positive (FP) for the identified events that are not likely to represent anomalies, and Unknown for the remaining ones.

We assign to the Probable Anomaly category events such as a high concentration of kernel errors, dropped log messages due to excessive rates or network-related errors. Unusual but not abnormal events such as a large number of virtual machines being deployed simultaneously, booting operations, or admin operations such as network configurations changes were labeled as False Positive. Events inside the Unknown are linked to log events referring to external systems or metrics. Assigning a clear label to such events is a difficult task as it requires to correlate the information from multiple systems. Given the significant number of such events in our case, we made the decision to set them aside, as inspecting each one in detail was not practically feasible.

Models

The cloud computing logging environment is inherently chaotic, significantly influenced by various client operations. Some of these operations are periodic and run by scripts like the backup or the cleaning of data where other ones are punctual (e.g., manually triggered API calls). It is common for equipment, such as servers, to transition rapidly from an inactive state to a fully loaded state. To address this dynamic nature, forecasting models are utilized to predict the subsequent log count vector based on the previous vectors. In our experimental evaluation, we have chosen and implemented three distinct types of models for this purpose.

We set r = 24, the number of last count vectors forwarded to the model. As each vector represents a five-minute time window, this is the equivalent of asking the model to predict the next five minutes based on the last two hours of logs.

Repetitor

This model always predicts the most recent count vector as the next output. With this model:

V (t) = V (t) (6.1)
The repetitor model offers a rather simplistic approach to forecasting log traffic, assuming its consistency over time. However, it presents some interesting aspects. As it only considers the last log count vector, it is less influenced by trailing noises. Obtained result displays that coupled with the appropriate error metric, this model can achieve a good precision even if it tends to miss a large number of abnormal events.

Mean Channel Value

This model returns the channel wise mean of the last r observations. For this model, V (t) is constructed as below:

V (t)(i) = r j=1 M (j)(i) r (6.2)
Another straightforward approach to forecasting log traffic is through the mean channel value model. It aims to mitigate the influence of the chaotic changes such as wide variation affecting one channel by using its over time mean.

LSTM Autoencoder

LSTM [START_REF] Hochreiter | Long short-term memory[END_REF] is a type of Recurrent Neural Network (RNN) that allows the network to retain long-term dependencies between data at a given time from many time steps before. They have been used by a large panel of sequence learning tasks including handwriting recognition, speech recognition, and sentiment analysis. An LSTM-based encoder is used to map an input sequence to a vector representation of fixed dimensionality. The decoder is another LSTM network which uses this vector representation to produce the target sequence.

LSTM autoencoders displayed promising results for multivariate time series forecasting in contexts like supply chain management [START_REF] Nguyen | Forecasting and anomaly detection approaches using lstm and lstm autoencoder techniques with the applications in supply chain management[END_REF], multi-sensor anomaly detection [START_REF] Malhotra | Lstm-based encoder-decoder for multi-sensor anomaly detection[END_REF], or spacecraft monitoring [START_REF] Hundman | Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding[END_REF]. To test the relevance of such models in our cloud computing logging context, we trained an LSTM encoder decoder to reconstruct instances of normal time series. Training log data were collected during the two weeks prior to our 11 days train dataset, they are issued from the same KVM but there is no overlap between the training and the evaluation period.

A linear layer on top of the LSTM decoder layer is used to predict the next log count vector V (t)(i).

Error Metrics

Accuracy measurement in a forecasting problem is always a subject of debate because of its importance. Workloads can significantly vary between two identical devices and change drastically over time. For instance, a server charge can operate between 0 and 100% of its computing capacity and quickly evolve within this range of values without abnormal. We have chosen three distinct metrics for this experimental evaluation. Each metric is given a forecast vector V and an observation vector V and returns a numeric value. We assume here that vectors have the same dimension |T| + 1.

Root Mean Square Error

The Root Mean Square Error (RMSE) is a commonly used metric in statistics. It is defined as the square root of the mean square error between the forecast vector V and an observation vector V :

RMSE = |T|+1 i=1 ( V (i) -V (i)) 2 |T| + 1 (6.3)
As it doesn't involve volumetric comparisons, this metric is heavily influenced by the most represented log templates. Two unpredicted logs associated to a rare logging event will have the same weight in the final error than two unpredicted logs associated to a frequent logging event.

Symmetric Mean Absolute Percentage Error

The Symmetric Mean Absolute Percentage Error (SMAPE) is also a commonly used metric in statistics, it expresses the accuracy as a ratio defined by the following formula:

SMAPE = 100 |T| + 1 |T|+1 i=1 | V (i) -V (i)| |V (i)| + | V (i)| (6.4)
It is the absolute difference between an observation V and the forecast V divided by half the sum of absolute values of the observation and the forecast. The value of this calculation is summed for every fitted point and divided again by the number of fitted points |T|+1. The lower the SMAPE value of a forecast, the higher is the model accuracy. This error metric involves volumetric comparisons between the expected and obtained number of each logging template.

With this metric, rare unpredicted events will have a stronger impact on the final value and fluctuation of the number of common events will be less impacting. One limitation to SMAPE is that if the actual value or forecast value is 0, the value of error will approach for the concerned channel 100%.

Log Accuracy Ratio

Mean Absolute Percentage Error metric systematically promotes methods with lower predictions.

The Log Accuracy Ratio (LAR) was introduced by Tofallis in 2015 [START_REF] Tofallis | A better measure of relative prediction accuracy for model selection and model estimation[END_REF] to avoid this bias. LAR is defined as the square logarithm of the ratio observation over the forecast:

LAR = 1 |T| + 1 |T|+1 i=1 ln( V (i) V (i) ) 2 (6.5)
Its properties are similar to SMAPE ones, as it also induces volumetric comparison between the forecasted and the actual values.

Evaluation

Precision Evaluation

In order to be accepted by technical teams, it is more important for an alerting model to be precise than to be accurate. A significant proportion of false positive can hide the relevant alerts and, in the long run leads the system to be abandoned. Computing the accuracy of each model/metric pair would require knowing all the abnormal events that occurred during the monitored period. As previously discussed, this is impracticable due to the number of logs inside the dataset. However, we spent a consequent amount of time inspecting and labeling all the abnormal sequences raised by the different model/metric pairs. Each sequence was assigned one of the following labels: Relevant Alert, False Positive or Unknown. More details about those labels can be found in Section 6.2.1.

The precision for each model/metric combination is computed based on those labels using the following equation: It is the ratio of relevant alerts over the total number of alerts. Note, we left aside all the Unknown labeled events. As we weren't able to assign them a clear label, we chose to ignore them when computing the precision. Table 6.1 regroups the obtained results for each of the nine model/metric combinations. Precision for the LSTM-AE/RMSE should be 1.0. Assigning a label to anomalies reported by this pair was a complicated task and, we end up assigning the Unkown label to almost all of them (11 315 over 11 316). We consider this to be a strong sign that reported anomalous period will not be easily exploitable to detect effective anomalies.

P
Regarding this, we set aside the precision result for this combination.

Overall, SMAPE appears to be the precision wise best performing metric, and LSTM-AE/SMAPE the most precise pair. This combination achieves a precision of 0.933 while also being the most verbose one with 13 219 anomaly candidate, 11 203 if we remove the unknown ones. RMSE on its side is the worst performing metric for each model. We explain this by its sensibility to volume change regarding the common templates. Most of the false positives raised by the model/RMSE pairs are linked to a massive deployment of virtual machines, automated CRON jobs or maintenance operations. All of those operations tend to generate a usually high number of log messages. The model/LAR pairs produces in between result, Tofallis [START_REF] Tofallis | A better measure of relative prediction accuracy for model selection and model estimation[END_REF] concern regarding metrics favoring underforecasting models does not seem to be harmful in our case.

Regards to the previously stated results, SMAPE appears to be the go for choice to optimize the precision regarding generated anomaly candidates.

Despite its simplicity, the repetitor model surprisingly achieves high precision. It obtained a precision of 0.891 when coupled with the SMAPE error metric, and a precision of 0.822 when coupled with the LAR error metric. We believe this result to be driven by the chaotic aspect of logs. A repetitor model with a proportional metric such as SMAPE or LAR is good to detect short term major evolution in logging behaviors. Also, the 5-minute duration of the window is huge compared to cloud operations periods that are usually measured in milliseconds. However, the total number of relevant alerts (840 for Repetitor/SMAPE) is way lower than the 10 458 reported by LSTM-AE/SMAPE.

Anomaly Duration

In the lower part of Table 6.1, we indicate the mean anomaly length for the reported anomaly candidates as well as the percentage of the total dataset duration that is flagged as part of an abnormal period (in %×10 -3 ).

Anomalies reported by LSTM-AE/metric pairs are all of the same duration (5 min), which is also the window length duration. Thus meaning that such model/metric combination never flagged two consecutive periods as abnormal. LSTM-AE forecasting seems to quickly adapt its output to the newly detected logging behaviors. This is a strong point regarding the capability of LSTM-AE neural structure for multivariate time series forecasting as well as for anomaly detection. Still, this fast adaptability seems to be penalizing regarding the discovery of longterm events such as KVM crashes. We discuss this more in the next section when analyzing the ability of model/metric pairs to forecast critical events.

Anomalies reported by MeanPredictor/metric like pairs tend to last around one hour (54.3 minutes for the MeanPredictor/SMAPE pair). This corresponds to almost half of the total duration of past observations forwarded to the model. One explanation is that drastic changes in the logging behavior impact the forecasting output of the model for the next two hours. A positive side of this behavior is that the model is more sensitive to the accumulation of errors over the time.

Critical Events Forecasting

Throughout the 11-day experimental period, the operational teams reported three server crashes.

As each crash affected a different KVM and occurred on a different day, we assumed here that they are independent anomalies. Predicting these events could aid the monitoring team in reducing their impact on client-hosted virtual machines. In this section we answer question RQ4. by investigating the capacity of the considered model/metric pairs to forecast the reported crashes.

We consigned the obtained results under Table 6.2. For each crash we reported the model/-metric pairs that detected the crash (i.e., crash time occurs during one of the reported anomaly candidate time periods), and if applicable the forecast. The forecast is computed using the time difference between the reported crash time and the beginning of the associated abnormal period if applicable.

The results indicate that two model/metric pairs detected at least one crash, while only one managed to detect all of them. The MeanPredictor/LAR detects two crashes and the MeanPredictor/SMAPE detects all of the three reported crashes. For the five detected crashes, the forecasting time is superior to 80 minutes. Giving the low number of alleged crashes, we are working with, it is difficult to reach any hard conclusion. However, obtained results support our hypothesis that KVM log files can be used to predict server crashes.

MeanPredictor model was the only one able to forecast some of the reported crashes. We believe this to be linked to this model sensitivity to the accumulation of small anomalies over time. When diving into prior to crash KVMs logging behavior, we observed a drift within the hours before the crash. A higher than the usual number of logs with error or warning severity was reported during this period. MeanPredictor/SMAPE pair appears to be specially efficient regarding the detection as well as the long-time alerting regarding those drifts. error value is the highest for the considered KVM during the experimental period.

Despite their abnormal nature, both the Repetitor and LSTM-AE models swiftly adapt to new behaviors. Repetitor and LSTM-AE models coupled with SMAPE or LAR metrics raised anomaly at the same time as the MeanPredictor/SMAPE and MeanPredictor/LAR models for the alleged crashes. However, they stop raising alert after a short period, around 30 minutes for Repetitor/metric pairs and after one abnormal period (5 min) for LSTM-AE/metric pairs.

Regarding this, we could debate about the ability of such models to forecast crashes because to a certain extent they detect their premises. When looking at the alert candidate detailed report raised by repetitor and LSTM-AE models, it was obvious that it was an alleged anomaly but hard to predict a crash. On the other hand, MeanPredictor generated reports give us more clues.

Repetitor

Perspectives

The results obtained in our study provide strong support for our hypothesis regarding the utility of logs in detecting and forecasting abnormal events. The implementation of our Monilog system demonstrated accurate identification of abnormal periods and offered valuable insights into the associated applications within a KVM environment. Moving forward, we aim to expand the monitoring capabilities of Monilog by conducting experiments on a wider range of cloud computing equipment and software. A key concern of our work is reducing the number of false alerts. As we broaden the monitoring scope, it is expected that abnormal events will generate multiple alert candidates as they become linked to more devices. To address this, we plan to experiment with combining Monilog's reported anomaly candidates with existing key performance indicators (KPIs) and metrics, resulting in fewer alerts with enhanced contextual information. Additionally, we believe that leveraging cloud-specific rules, such as geographic distribution of devices, can facilitate the combination of anomaly candidates and their associated contexts.

Regarding the labeling of the anomalies in our experimental study, we choose to put admin operations in the False positive category, such events can be the sign of malicious events run by an infiltrated agent with admin privileges. As the focus of this work is on the platform and resources monitoring, we chose to set such scenario aside and classified the linked event as normal. Also, the error metric should be picked carefully and be driven by your context [START_REF] Flores | A pragmatic view of accuracy measurement in forecasting[END_REF].

In our cloud computing environment, non-homogeneity is a major concern. Error metric that includes proportional comparisons (SMAPE and LAR) gave us better results than the other one (RMSE). In the future we can imagine crossing information from multiple error metrics to alert on specific scenarios.

CHAPTER 7

Conclusion and Perspectives

Conclusion

Cloud computing platforms offer a globally accessible pool of shared and highly available processing and storage resources. These platforms have gained immense popularity due to their cost-effective distribution of computing infrastructure among multiple organizations. However, effectively monitoring the growing number of IT resources on the provider side presents a significant challenge. Logs serve as a valuable resource for various tasks, including inspection, maintenance, development, and alerting.

In this manuscript, we present our research, which aims to address the issue of automated log-based anomaly detection within cloud computing platforms. Our work aligns with the fields of data mining, cloud computing, deep learning, and distributed systems. The key contributions of our research are as follows:

1. Literature Review: We conduct a comprehensive review of existing research in the areas of log parsing and log-based anomaly detection. Throughout the manuscript, we provide in-depth discussions and analysis for each of the above points, utilizing a combination of open-source and 3DS OUTSCALE log data. We believe that our work significantly contributes to the advancement of automated log-based anomaly detection in cloud computing platforms and offers valuable insights for practitioners and researchers in this field.

Contributions to the Log-Parsing Field

Labeling logs in complex environments can be a laborious task. For an automated log-based anomaly detection system, it is crucial to have a log parsing solution that is both robust to log changes and adjustable without relying on labeled data. In Chapter 3 of our study, we delve into the impact of hyperparameter tuning on Spell and Drain, two widely used log parsing methods. Our investigation reveals that there are no universally applicable values for these hyperparameters, emphasizing the necessity for more robust parsing solutions.

In Chapter 4, we introduce USTEP, an online log parsing approach based on an evolving tree structure, able to discover and encodes new parsing rules on the run. The comparative complexity analysis highlights the fact that USTEP is the only existing method to achieve constant time complexity as it is not influenced by the state of its memory structure. Experimental results on a panel of 14 datasets including one industrial dataset issued from 3DS OUTSCALE logging infrastructure show the superior parsing effectiveness (+3.4% at 93.7%) and robustness of USTEP when compared to other state-of-the-art techniques.

Log parsing is a mandatory first step for many log mining methods. Therefore, parsing time can become a bottleneck for solutions requiring real-time processing in log intensive environments. In those cases, centralized log parsing fails to produce on the fly output. The USTEP-UP framework introduced in Chapter 3 helps to run multiple USTEP instances in parallel while conserving parsing consistency.

Automated Log-Based Anomaly Detection in Cloud Environment

In Chapter 5 of our research, we introduce Monilog, an anomaly detection system specifically designed for the cloud computing context. Monilog is capable of detecting anomalous events in environments where the relationships among logs are not easily identifiable. Monilog utilizes a model to forecast the log traffic and error metrics, which enables the evaluation of its forecasting accuracy. An abnormal period is identified when the associated error reaches an unusual value, indicating potential anomalies. One of the unique aspects of Monilog lies in its parsing and embedding of log events. This approach enables the system to generate comprehensive summaries pertaining to the reported anomaly candidates, including details about the systems and applications involved. By leveraging these capabilities, Monilog enhances log-based anomaly detection in the cloud computing environment, offering valuable insights for identifying and addressing irregular events that may occur within complex log structures.

In Chapter 6, we conducted a comprehensive evaluation of our proposal at an industrial scale using 11 consecutive days of logs obtained from the KVMs in the European cloud region operated by 3DS OUTSCALE. The evaluation aimed to assess the effectiveness of our approach in raising relevant alerts for anomaly detection. During the evaluation, we considered three different forecasting models: Repetitor, MeanPredictor, and LSTM-AE, along with three error metrics: RMSE, SMAPE, and LAR. Our primary focus was to evaluate the performance of these nine model/metric pairs in identifying and alerting abnormal periods. Based on the results, the SMAPE error metric emerged as the best performing metric, exhibiting the highest efficacy in detecting abnormal periods. Additionally, the LSTM-AE model demonstrated the highest effectiveness in detecting such abnormal periods. Throughout the monitored period, three instances of KVM crashes were reported. Notably, the MeanPredictor model successfully predicted all of these crashes, providing forecasts at least 80 minutes prior to each event. This outcome serves as a validation of the effectiveness and relevance of Monilog in forecasting abnormal events within cloud computing infrastructure. The evaluation conducted on real-world data from the European cloud region underscores the practical utility and reliability of our proposed approach, reinforcing its potential for enhancing anomaly detection and forecasting within cloud computing environments.

Log-Based Anomaly Detection

The experimental results conducted on a cloud platform scale scale demonstrate the promising potential of Monilog in achieving automated log-based anomaly detection. Further exploration of Monilog's capabilities should focus on multi-component anomaly detection. While our practical analysis solely utilized logs from the KVM, it would be interesting to extend monitoring to a broader range of logging sources, such as storage devices and cloud hypervisors. The range of abnormal events that Monilog can detect is extensive, with our current focus being on virtualization-induced server crashes. However, this detection capability could be expanded to include security events or any other form of unusual logging behavior.

On a more technical note, advancements in the field of machine learning can greatly benefit Monilog by providing opportunities to experiment with new log forecasting models. Conducting experiments with different model/metric combinations would enhance our understanding of how to optimize the Monilog architecture specific contexts.

Regarding the embedding step of Monilog, we deliberately opted not to employ log parsing. This decision was driven by the desire to assess the potential of Monilog using the simplest approach as a benchmark. However, in the future, we believe that incorporating log parsing would be advantageous, as it would enable the utilization of a wider range of information for anomaly detection and the generation of anomaly reports. This, however, would require dealing with parsing errors.

Monilog is designed to detect concept drifts in system logging behavior and provide detailed outputs related to such drifts. Nevertheless, this approach can also be adapted to capture various events. For instance, we envision Monilog being applied to track the propagation of software updates within a cloud platform or to analyze usage patterns across different system components.

Safety and Monitoring

By carefully examining logs, security professionals can identify potential security breaches, malicious activities, and unauthorized access attempts. Logs enable the detection of suspicious patterns, anomalies, and indicators of compromise, facilitating early warning and timely incident response. Additionally, logs play a crucial role in forensic investigations, providing a detailed record of events and actions that can be analyzed to reconstruct timelines and identify the root cause of security incidents. Security use cases usually required fine-grained analysis as attackers take care of leaving the smallest number of traces.

Security use cases typically necessitate fine-grained analysis, as attackers are adept at minimizing their traceable activities. A challenge extensively addressed in the literature is that the majority of log analyses are conducted post-mortem [START_REF] Svacina | On vulnerability and security log analysis: A systematic literature review on recent trends[END_REF]. Consequently, such approaches are unable to promptly address ongoing attacks. In our future research, we aim to explore automated methodologies for swiftly detecting security attacks by leveraging multiple data sources, including logs.

Correlating log data with other data sources such as KPIs, metrics or architectural information could help refine the analysis of abnormal events, reduce the number of false positive, detect weaker signals or enrich the anomalous context. We believe that working with heterogenous sources is key to develop better monitoring systems able to detect a wider range of abnormal events including malicious and hardly traceable events. This is challenging as working with a single data sources can already be a big data challenge as this was discuss a lot inside this manuscript. Combining multiple sources can therefore pose an even bigger data challenge [START_REF] Zuech | Intrusion detection and big heterogeneous data: a survey[END_REF]. Still we believe this research field to be the next big step in building intelligent monitoring system for complex platforms such as cloud ones.

Introduction

Contributions

Nous avons organisé nos travaux de recherches autour de 3 axes:

Axe 1: Structuration des Logs

Les logs informatiques sont des données semi-structurées. Chaque log a un entête qui suit un format défini par convention, incluant des champs tels que la date et l'heure, le système, l'application concernée, ou la criticité. Le format des messages de logs lui, est libre et laissé à la discrétion des développeurs. Cette flexibilité a permis une adoption massive des logs en pratique, mais est également un frein à leur exploitation automatique. Afin de récupérer les données relatives à l'état courant du système il nécessaire de pouvoir les extraire du message.

Dans cet axe, nous explorons les méthodes existantes de structuration des messages logs et nous nous intéressons à l'optimisation de leurs hyper paramètres afin d'identifier des valeurs génériques. Les résultats de notre étude mettant en limite la non-existence de telles valeurs nous ont poussés à proposer USTEP une nouvelle méthode de structuration des messages logs, plus robuste aux différents environnements et présentant une meilleure complexité temporelle que l'état de l'art. Le besoin de temps réels étant un prérequis fort pour un système de détection d'anomalies, nous avons également introduit USTEP-UP une architecture pour faire tourner plusieurs instances d'USTEP en parallèle et permettre le passage à l'échelle.

Axe 2: Détection des Anomalies dans des Infrastructures Cloud

Les logs sont déjà activement utilisés pour la détection d'anomalies et depuis 2017, de nombreuses méthodes basées sur des approches de deep learning ont été publiées [24; 29; 71]. Ces méthodes présentent des précisions élevées, cependant elles ont besoin de pouvoir regrouper les logs en séquences afin de détecter des anomalies. En pratique, à l'échelle d'une plateforme de cloud computing, ce prérequis est pratiquement impossible à obtenir à cause de la forte volumétrie, de la multiplicité des sources de logs et de l'absence d'identifiants uniques de transactions dans de nombreuses sources.

Nous avons proposé Monilog, un système autonome de détection des anomalies adapté aux besoins de robustesse et de rapidité des plateformes cloud et qui ne nécessite pas de devoir lier les logs entre eux. Monilog est inspiré du domaine des séries temporelles multivariées, il prédit l'activité log d'une machine à l'aide d'un modèle et se sert de son erreur de prédiction pour détecter des comportements anormaux. Notre implémentation de Monilog à des fins de surveillance de la région Europe de 3DS OUTSCALE nous a permis de valider son efficacité pour la prédiction de comportements anormaux liés aux serveurs de virtualisation (KVM).

Axe 3: Caractérisation des Anomalies

Lorsqu'une anomalie est détectée, l'information remontée à une équipe humaine doit indiquer le plus précisément possible la cause et le contexte du problème. Pour assurer une résolution efficace, l'identification du contexte est aussi importante que l'anomalie elle-même afin de cibler les experts les plus qualifiés pour la gérer. Les travaux liés à cet axe comprennent une revue Présentée en 2017, Deeplog [START_REF] Du | Deeplog: Anomaly detection and diagnosis from system logs through deep learning[END_REF] est la première méthode proposée basée sur des réseaux de neurones. Elle utilise un réseau de neurones de type LSTM pour détecter des anomalies dans des séquences de logs. Deeplog a rapidement été étendu par des chercheurs de Microsoft [START_REF] Zhang | Robust log-based anomaly detection on unstable log data[END_REF] et de Huawei [START_REF] Meng | Loganomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs[END_REF]. Les deux équipes ont utilisé une structure LSTM bidirectionnelle et ajouté des mécanismes de résilience face aux évolutions des comportements logs des systèmes ciblés.

Depuis, de nombreuses approches d'apprentissage profond ont été proposées, les méthodes utilisées suivant les tendances et les avancées du domaine. Ces méthodes partagent cependant un mécanisme commun, ils dépendent de solution de structuration des logs pour extraire les patrons sous-jacents et autres informations utiles des messages de logs.

Structuration des Logs

Les approches de structuration des logs visent à séparer les parties injectées du patron d'un message afin de reconstituer la ligne de code qui l'a générée. Elles permettent ainsi de regrouper ensemble des logs tu mêmes types et d'accéder à des informations intégrées aux messages. Les avancées dans le domaine de la structuration de logs ont un impact direct les performances des méthodes de détection automatisée des anomalies.

Les méthodes existantes peuvent être regroupées en deux familles. Les approches permettant

Étude de Méthodes Robustes pour la Structuration des Logs

Les méthodes de structuration des logs étant utilisées en amont des méthodes de détection d'anomalies, leur précision peut avoir une influence directe sur les méthodes en aval. Une de nos études sur Deeplog couplé avec Drain montre qu'une baisse 20% de la précision de structuration peut entrainer une baisse de 70% de la précision de détection des anomalies [START_REF] Vervaet | Ustep: Unfixed search tree for efficient log parsing[END_REF].

Impact des valeurs des Hyperparamètres sur les Méthodes de Référence

Les principaux facteurs influant sur la précision d'une méthode de structuration sont les valeurs de ses hyperparamètres et les expressions régulières qui lui sont fournies pour prétraiter les messages. Isoler un jeu de données représentatif d'un environnement cloud et le labéliser pour paramétrer les méthodes est une tâche ardue et non pérenne dans le temps.

Nous nous sommes penchés sur l'existence de valeurs par défaut permettant des performances stables sur différents jeux de données pour Spell et Drain. Notre étude met en lumière l'inexistence de telles valeurs ainsi que les performances fluctuantes en termes de précision maximale obtenue sur des logs issus de systèmes différents [START_REF] Vervaet | Automatisation de la structuration des logs pour le cloud computing[END_REF]. Ces résultats nous ont motivés à explorer et proposer USTEP, une méthode de structuration des logs plus précise et plus robuste.

Conclusion Générale

Dans ce manuscrit, nous présentons nos travaux de recherches visant à proposer un système autonome et innovant de détection des anomalies dans les infrastructures de cloud computing en exploitant les logs. Nos contributions sont les suivantes:

• Un état de l'art du domaine de la détection d'anomalies à partir de logs et du domaine de structuration des logs.

• Une étude de la robustesse de Spell et Drain ainsi que de leur paramétrage automatique.

• Nous avons proposé USTEP, un algorithme de structuration des logs et USTEP-UP sa version distribuée.

• Nous avons introduit une nouvelle architecture pour la détection automatique d'anomalie à partir de message logs, capable de générer des rapports sur les anomalies détectées.

• Une évaluation à l'échelle d'un cloud de notre proposition en exploitant des logs issus de serveurs de virtualisation.

8 . 4 1 Introduction Contents 1 . 1 2 1. 1 . 1 4 1. 2 . 1 5 1. 2 . 2 1 . 2 . 4 9 1. 2 . 5

 84111211421522124925 Proposition de Monilog un Système Innovant de Détection d'Anomalies à Partir de Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.1 Architecture de Monilog . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.2 Utilisation de Monilog pour la Surveillance de KVM . . . . . . . . . . . . 8.5 Conclusion Générale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References x List of Figures 1.1 A Sample of a Log File From an HPC System . . . . . . . . . . . . . . . . . . . . 1.2 3DS OUTSCALE Simplified Cloud Platform Architecture . . . . . . . . . . . . . 1.3 Log Statement, Log Event and Parsed Log, an Example . . . . . . . . . . . . . . 2.1 Multi-Layer Logging Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Parsing Accuracy Impact on Deeplog Performances . . . . . . . . . . . . . . . . . 3.2 Influence of the Hyperparameters on the Parsing Accuracy . . . . . . . . . . . . 4.1 USTEP Evolving Research Tree Structure . . . . . . . . . . . . . . . . . . . . . . 4.2 Example of USTEP Search Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Example of Leaf Division with Pivot set to 3 . . . . . . . . . . . . . . . . . . . . 4.4 Robustness of Log Parsers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Cumulated processing time by dataset . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Processing Time Evolution for OpenStack-extended . . . . . . . . . . . . . . . . 4.7 USTEP-UP architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.8 Example of merging tree process . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Monilog Architecture and Workflow . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 An Example of Error Time Serie . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1 USTEP Structurelle Mémorielle Arborescente . . . . . . . . . . . . . . . . . . . . 8.2 Architecture et Fonctionnement de Monilog . . . . . . . . . . . . . . . . . . . . . 8.3 Un Exemple de Série Temporelle d'Erreurs . . . . . . . . . . . . . . . . . . . . . xi List of Tables 2.1 Log-Based Anomaly Detection Methods, an Overview . . . . . . . . . . . . . . . 3.1 Online Log Parsers Presented in [1]. . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Characteristics of Datasets Used by [1] Benchmark . . . . . . . . . . . . . . . . . 3.3 Regular Expressions Used in the Preprocessing Step . . . . . . . . . . . . . . . . 3.4 Influence of the Preprocessing on the Parsing Accuracy . . . . . . . . . . . . . . 4.1 Datasets Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Log Parser Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 USTEP Parameters Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Parsing Accuracy of Log Parsers . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Memory Usage in MB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Worst-case complexities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Predictor Alerting Precision per Selected Error Metric . . . . . . . . . . . . . . . 6.2 Model/Metric Pairs Ability to Forecast the Major Anomalous Events . . . . . . . 8.1 Méthodes de Détection Automatisée des Anomalies, un Aperçu . . . . . . . . . . 8.2 Précision des Méthodes de Structuration . . . . . . . . . . . . . . . . . . . . . . . 8.3 Précision des Paires Modèle/Métrique . . . . . . . . . . . . . . . . . . . . . . . . CHAPTER Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Landscape of Cloud Monitoring . . . . . . . . . . . . . . . . . . . . 3 1.1.2 An Introduction to Logging . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2 Research Objectives and Axis Covered by This Manuscript . . . . . Elements of Vocabulary . . . . . . . . . . . . . . . . . . . . . . . . . . . An Overview of 3DS OUTSCALE Cloud Platform and Logging Activities 6 1.2.3 Axis 1: Extraction of Structured Data From Log Files . . . . . . . . . . 7 Axis 2: Anomaly Detection in Highly Distributed IT Infrastructures . . Axis 3: Contextualization of Anomalies for Decision Support . . . . . . 9 1.3 An Overview of Our Contributions . . . . . . . . . . . . . . . . . . . . 10 1.4 Organization of This Manuscript . . . . . . . . . . . . . . . . . . . . . 12 1.1 Context

Figure 1 . 1 :

 11 Figure 1.1: A Sample of a Log File From an HPC System

  the company is designed to be operated in an API-firsts fashion, meaning that the first entry point for client requests is an ensemble of network-accessible endpoints. Incoming requests are forwarded to the cloud orchestrator (TINA OS) that will interact with other services as well as hardware components to perform the appropriate tasks to satisfy customers' demands.

Figure 1 .

 1 Figure 1.2 is a simplified version of the company cloud platform architecture.

Figure 1 . 2 :

 12 Figure 1.2: 3DS OUTSCALE Simplified Cloud Platform Architecture

Figure 1 .

 1 Figure 1.3 displays a log statement in Python and an example of a log event generated by this statement.

Contents

3. 1

 1 Log Parsing for Log-Based AD Methods . . . . . . . . . . . . . . . . 3.1.1 Impact of the Parsing Accuracy on Anomaly Detection Methods . . . . 3.1.2 Parsing Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Online Log Parsers and Hyperparameters Tuning . . . . . . . . . . . 3.2.1 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Experimental Motivations . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.3 Experimental Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.4 Impact of the Hyperparameters on the Parsing Accuracy . . . . . . . . 3.2.5 Impact of the Preprocessing on the Parsing Accuracy . . . . . . . . . . 3.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Log Parsing for Log-Based AD Methods

Figure 3 . 1 :

 31 Figure 3.1: Parsing Accuracy Impact on Deeplog Performances

Figure 3 . 2 :

 32 Figure 3.2: Influence of the Hyperparameters on the Parsing Accuracy

  the shape of "*b*eb-*d*a-*aa*-ba*b-*d*f*f*c" in three templates. Without preprocessing, these

Figure 4 Figure 4 . 1 : 2 . 4 .Figure 4 . 2 :

 4412442 Figure 4.1: USTEP Evolving Research Tree Structure

Figure 4 .

 4 Figure 4.2 displays a tree structure with a root node, one internal node, 3 leaves, and 4

1 :

 1 function Parse(t i ) 2:

Figure 4 . 3 :

 43 Figure 4.3: Example of Leaf Division with Pivot set to 3

p

  : e i ∈ leaf.templates} ▷ Set of tokens at position p end for 26: pivot = argmax{|S p | : p ∈ 1, . . . length } return pivot 28: end function UpdateTree algorithm and the related methods are described in Methods 4. The leaf division process starts by searching for the position with the highest number of different tokens among the templates. This position p is set to be the pivot. The leaf is then transformed into an internal node with the same label and a key function key(l i ) = t p . For each unique token at position p within the templates, a new leaf is created with the appropriate label. All the templates attached to the initial leaf node are transferred to a newly created leaf following the new descent rule.

Figure 4 .

 4 Figure 4.3 illustrates the division process for ϕ = 3. The considered leaf has 4 templates

Figure 4 . 4 :

 44 Figure 4.4: Robustness of Log Parsers

4. 2 . 4

 24 Efficiency of the Log Parsers 4.2.4.1 Experimental Parsing Time

Figure 4 . 5 :

 45 Figure 4.5: Cumulated processing time by dataset

  So, both have the same space complexity O(T × N ) and a searching time complexity O(d + max{c leaf ∀leaf} × |t i |) where d is the tree depth and max{c leaf ∀leaf} is the maximum number of candidate templates in leaves. In the worst-case scenario, Drain attaches all discovered templates to the same leaf having max{c leaf ∀leaf} = T , however, USTEP bounds this number by the parameter ϕ. In both cases, the depth of the tree doesn't exceed the maximum token length |t|. USTEP searching time complexity is therefore in O(|t i | + ϕ × |t i |), with ϕ a constant. Spell does not partition its search space and will use the LCS algorithm to compare t i with all the existing templates. LCS algorithm complexity is in O(|t i | × |t|). Thus Spell time complexity is in O(|t i | × |t| × T ). LenMa starts by iterating over the log message to create a word length vector representing the character length of each token. It then compares this vector, with all the discovered vectors with the same length (|t i |). Therefore, its time complexity is in O(|t i | + T × |t i |). In the worst case, SHISO tree descent algorithms can visit all the existing nodes in the structure and compare the log with all the templates. Its worst-case time complexity is then in O(T × |t i |).

Figure 4 . 6 :

 46 Figure 4.6: Processing Time Evolution for OpenStack-extended

Figure 4 .

 4 Figure 4.7: USTEP-UP architecture

Figure 4 . 8 :

 48 Figure 4.8: Example of merging tree process

Figure 5 . 1 :

 51 Figure 5.1: Monilog Architecture and Workflow

Figure 1 .

 1 Figure 1.3 would be "serviceManager INFO". This embedding bounds the number of different possible templates by the number of running applications multiplied by the number of distinct severity levels (8 for syslog formatted logs). Also, it keeps track of the related application and the severity, this property is exploited by the context acquisition step to generate contextual anomaly reports.

Figure 5 . 2 :

 52 Figure 5.2: An Example of Error Time Serie

Figure 5 . 2 displays

 52 the over time error of the SMAPE error metric over MeanPredictor forecasts. The crash occurred during Day 3, the crash error time is clearly visible as the associate reported

Contents 7. 1 86 7. 1 . 1

 18611 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Contributions to the Log-Parsing Field . . . . . . . . . . . . . . . . . . . 87 7.1.2 Automated Log-Based Anomaly Detection in Cloud Environment . . . . 87 7.2 Perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 7.2.1 Data Mining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 7.2.2 Log-Based Anomaly Detection . . . . . . . . . . . . . . . . . . . . . . . 90 7.2.3 Safety and Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

2 .

 2 Robustness and Tuning of Spell and Drain: We investigate the robustness and fine-tuning aspects of Spell and Drain, two state-of-the-art online parsing methods. 3. Novel Parsing Techniques: We propose novel, more robust and efficient approaches for parsing log messages in an online and distributed manner.4. Automated Anomaly Detection Architecture: We introduce Monilog, an automated architecture designed for detecting anomalies within a cloud infrastructure through log analysis.This architecture leverages recent advancements in deep learning and provides explanations for the flagged events.5. Cloud-Scale Experimentation:We present a large-scale experimentation of Monilog, an anomaly detection framework, using KVM logs.

8. 1 . 1 Contexte

 11 La flexibilité et la haute disponibilité des ressources informatiques mise à disposition par les plateformes de cloud computing stimulent une adoption massive du modèle. Le secteur jouit d'une croissance toujours dynamique avec une prévision de 29.8% pour le secteur public en 2023 [4]. Du côté des fournisseurs, le maintien en activité et le développement d'un parc d'équipements et de services en croissance constante posent des contraintes d'automatisation. Dans le cadre de cette thèse, nous étudions les registres machines (logs) comme source d'information pour la création d'un système innovant et autonome de détection des anomalies dans les plateformes de cloud computing. Nos travaux ont été réalisés dans le cadre d'un partenariat industriel avec 3DS OUTSCALE, fournisseur français cloud souverain. Cette collaboration nous a permis d'échanger avec des experts techniques du domaine, de développer une meilleure connaissance des enjeux et des limitations des systèmes existants et de réaliser des expériences à l'échelle sur des données réelles.

  de traiter des messages en flux, et celles qui ont besoin d'effectuer plusieurs passes sur des jeux de données. Dans notre contexte, les seules méthodes pertinentes tombent dans la première catégorie, les logs étant générés en flux. À notre connaissance, 4 méthodes de structuration des logs en flux existent dans la littérature: SHISO [100] et LenMa [101] basées sur des méthodes de partitionnement, Spell [102] qui utilise l'algorithme de plus grande séquence commune; et Drain [103; 104] une approche basée sur un arbre de recherche. Parmi elles, Drain est souvent citée comme la méthode la plus précise et recommandée par plusieurs études [1; 29; 105].

8. 4

 4 Proposition de Monilog un Système Innovant de Détection d'Anomalies à Partir de Logs Les limites que nous avons identifiées dans notre état de l'art des méthodes de détection des anomalies sont: 1 une sensibilité aux erreurs de structuration; 2 le besoin de pouvoir regrouper les logs en séquences. Le point 2 étant non atteignable de manière réaliste dans le cadre d'une plateforme cloud, de par la diversité des environnements de logs impliqués et leur évolution, nous avons proposé Monilog, une méthode de détection d'anomalie ne nécessitant pas ce prérequis. Monilog est également capable de générer des rapports sur les erreurs remontées incluant le système et les applications concernées.

8. 4 . 1

 41 Architecture de Monilog

Figure 8 . 2 :

 82 Figure 8.2: Architecture et Fonctionnement de Monilog

Figure 8 .

 8 Figure 8.3: Un Exemple de Série Temporelle d'Erreurs

  Tout au long du manuscrit, nous fournissons des discussions et des analyses détaillées pour chacun des points susmentionnés, basées une combinaison de données de journal open source et 3DS OUTSCALE. Nous pensons que notre travail contribue à l'avancement de la détection automatisée des anomalies basée sur les logs dans les plateformes de cloud computing et fournit des informations précieuses aux praticiens et aux chercheurs dans ce domaine.

  capacités prédictives de Monilog sont 7.2.1 Data Mining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.2 Log-Based Anomaly Detection . . . . . . . . . . . . . . . . . . . . . . . . 7.2.3 Safety and Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Extended Summary in French 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1.1 Contexte . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1.2.1 Axe 1: Structuration des Logs . . . . . . . . . . . . . . . . . . . 8.1.2.2 Axe 2: Détection des Anomalies dans des Infrastructures Cloud 8.1.2.3 Axe 3: Caractérisation des Anomalies . . . . . . . . . . . . . . . 8.2 État de l'Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.1 Détection des Anomalies à l'aide de Logs . . . . . . . . . . . . . . . . . . 8.2.1.1 Méthodes Traditionnelles . . . . . . . . . . . . . . . . . . . . . . 8.2.1.2 Méthodes Basées sur de L'apprentissage Profond . . . . . . . . . 8.2.2 Structuration des Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3 Étude de Méthodes Robustes pour la Structuration des Logs . . . . . . . . . . . 8.3.1 Impact des valeurs des Hyperparamètres sur les Méthodes de Référence . 8.3.2 USTEP, une Méthode de Structuration plus Robuste et plus Rapide . . .

  1.2 Research Objectives and Axis Covered by This Manuscript abnormal events. However, a high precision means a low proportion of False Positive and therefore few false alarms.

	P recision =	True Positive True Positive + False Positive	(1.1)
	1.2.2 An Overview of 3DS OUTSCALE Cloud Platform and Logging Activ-
	ities
	As a cloud provider, 3DS OUTSCALE has two key assets: 1/ the hardware infrastructure built

in partnership with CISCO, Intel, Netapp and Nvidia; and 2/ a proprietary orchestrator, TINA OS, to manage the infrastructure and allocate resources to users

[22]

. The cloud platform of

  Reconstructing Log Sequences . . . . . . . . . . . . . . . . . . . . . . . 24 2.6.2 A Limited Number of Open Datasets . . . . . . . . . . . . . . . . . . . . 25

	2.1 Log Mining Applications Within the Literature
	2.6.1
	CHAPTER 2
	State of the Art
	Contents
	6, we present experimental results of Monilog applied
	to the monitoring of Virtualization servers at cloud scale. Chapter 7 sum-ups the presented work
	and discuss future work based on this thesis.

  8 for OpenStack, 3.2 Online Log Parsers and Hyperparameters Tuning

	Dataset	Regular expressions
		((\d+\.){3}\d+,?)+
	OpenStack	/.+?\s
		\d+
	Android	(/[\w-]+)+', r'([\w-]+\.){2,}[\w-]+ \b(\-?\+?\d+)\b|\b0[Xx][a-fA-F\d]+\b|\b[a-fA-F\d]{4,}\b
	HDFS	blk -?\d+ (\d+\.){3}\d+(:\d+)?

Table 3 .

 3 3: Regular Expressions Used in the Preprocessing Step

	Dataset	Spell (without) PA #Templ.	Spell (with) PA #Templ.	Drain (without) PA #Templ.	Drain (with) PA. #Templ.
	Android 0.60	425	0.91 (×1.5) 180 (×0.42) 0.67	217	0.91 (×1.4) 171 (×0.79)
	HDFS	0.28	684	1.00 (×3.6)	14 (×0.02)	1.00	17	1.00 (×1)	16 (×0.94)
	O.Stack	0.23	692	0.77 (×3.3) 451 (×0.65) 0.84	75	0.73 (×0.8) 299 (×3.99)

Table 3 .

 3

4: Influence of the Preprocessing on the Parsing Accuracy 0.85 for Android, a nd 0.5 for HDFS. The choice of τ has the most significant effect on the HDFS dataset, with half of the considered values leading to precision lower than 0.7, and even dropping to 0. On the other hand, for OpenStack and Android, selecting an appropriate value of τ can result in accuracy improvements of up to 15% and 20%, respectively. Drain's behavior differs among the datasets. While high values of ST (> 0.7) do not yield good results, the choice of the depth hyperparameter is crucial for obtaining accurate results

  Methods 2 USTEP Template Mining Methods function InitializeTemplate(t i )

	2:	e ← t i
		return e
	4: end function
	8:	return e
		end function
	10:
		function getVariables(e, t i )
	12:

6: function UpdateTemplate(e, t i ) e ← (e j if e j = t i j else * : j = 1 . . . |t i |)

  υ k else InitializeTemplate(t i ) ▷ New leaf to include t i Create a new leaf υ j with a templates list [e i ]. i ← argmax{simF (t i , e) : e ∈ leaf i .templates} if simF (t i , e i ) > σ then 22: e i ← InitializeTemplate(t i ) ▷ New template to include t i Add new template e i in leaf i tempaltes list

	12: e i ← 14: end if
		end while
	16:	return υ j
		end function
	18:
		function SelectTemplate(leaf i , t i )
	20:
	24:	else
		e i ← UpdateTemplate(e i , t i )
	26:	end if
		v

e i ← getVariables(e i , t i ) 28:

  Methods 4 Tree Update Methods function UpdateTree(leaf i ) pivot position pivot and an empty list of leaves L for all e i ∈ leaf.templates do ▷ Share templates between new leaves pivot ∀e j ∈ x.templates then Append e i to template list of x

	2:	if |leaf i .templates| > ϕ then
		DivideLeaf(P, leaf i )
	4:	end if
		end function
	6:
		function DivideLeaf(leaf )
	8:	pivot ← SelectPivot(leaf )
		Create an inner node υ with
	10:
	12:	if ∃x ∈ L : e (j) pivot = e	(i)
	14:	else
		Create a new leaf with template e i and append to L
	16:	end if
		end for
	18:	Replace leaf by inner node υ in P
		end function
	20:
		function SelectPivot(leaf )
	22:	Let's length templates length of leaf
		for p ← 1, . . . length do
	24:	S p ← {e

a

Table 4 .

 4

	4.2.1.1 Datasets
	Name	#log	size(B)	#template	#t/line	%var
	Apache	2 000	168K	6	6.3	23%
	BGL	2 000	310K	120	6.3	25%
	Hadoop	2 000	376K	114	8.4	37%
	HDFS	2 000	282K	14	7.4	45%
	HPC	2 000	148K	46	3.5	18%
	Mac	2 000	312K	341	9.4	28%
	OpenSSH	2 000	220K	27	8.7	28%
	OpenStack	2 000	582K	43	9.0	45%
	Thunderbird	2 000	318K	149	8.5	16%
	Zookeeper	2 000	274K	50	6.3	18%
	HDFS-2	11 175 629	1.5G	30	7.4	N.A.
	OpenStack-2	207 820	54M	43	9.0	N.A.
	Internal-1	1 750 916	1.2G	N.A.	28.2	N.A.
	Big-comp	2 × 10 9	150 GB	910	6.6	33%

1: Datasets Characteristics

Table 4 .

 4

	4.2.1.2 Log Parsers
	Log parser Year	Method	#parameters
	Drain	2017	Fixed depth parsing tree	3
	LenMa	2016	Clustering	1
	SHISO	2013	Clustering	4
	Spell	2016 Longest common subsequence	1
	USTEP	2021	Evolving parsing tree	2

2: Log Parser Characteristics

Table 4

 4 is an online log parser based on the Longest Common Subsequence algorithm (LCS). It uses different pre-filtering techniques to accelerate the matching process.

	The principal characteristics of each method are presented in Table 4.2. Each log parser uses
	different parameters (e.g., σ, ϕ for USTEP), for fairness of comparison each presented experiment
	was run multiple times to fine-tune the parameters and retain only the best value.
	4.2.1.3 Additional Tuning
	Dataset	σ	ϕ	Dataset	σ	ϕ
	Apache	0.5	4	Internal-1	0.3 50
	BGL	0.45 18	Mac	0.6 18
	Big-comp 0.3	8	OpenSSH	0.75 3
	Hadoop	0.3	3	OpenStack	0.9	2
	HDFS	0.35 2	OpenStack-2 0.9	2
	HDFS-2	0.35 2	Thunderbird 0.3	4
	HPC	0.6	2	Z.keeper	0.63 3

.3: USTEP Parameters Values

Table 4 . 4

 44

	1; 102], which corre-

: Parsing Accuracy of Log Parsers log message is considered correctly parsed when it aligns with the same template as its ground

Table 4 .

 4

			5: Memory Usage in MB
	Dataset	Drain LenMa SHISO Spell USTEP
	Apache	1.917	1.796	1.714 2.007	1.784
	BGL	3.777	3.987	3.581 3.878	3.687
	Hadoop	3.079	3.095	2.875 3.228	3.035
	HDFS	2.835	2.738	2.636 2.957	2.712
	HPC	2.700	2.704	2.479 2.753	2.598
	Mac	3.368	3.732	3.161 3.456	3.442
	OpenSSH	2.734	2.637	2.541 2.854	2.622
	OpenStack	4.004	4.009	3.769 4.342	4.233
	Thunderbird 4.053	4.062	3.812 4.151	4.058
	Zookeeper	2.988	2.926	2.795 3.035	2.894

Considered parsing methods are based on different parsing structures and therefore can have a different memory footprint. In Table

Table 4 .

 4

	6: Worst-case complexities
	Log parser Searching time complexity Space complexity
	USTEP

Table 6 .

 6 1: Predictor Alerting Precision per Selected Error Metric

	recision =	relevant alerts relevant alerts + f alse positives	(6.6)

Table 6 .

 6 2: Model/Metric Pairs Ability to Forecast the Major Anomalous Events6.4 Lesson LearnedIn this chapter, we presented Monilog a log-based anomaly detection system designed for the cloud computing context. It is able to detect anomalous events in contexts where log relationships are not identifiable and to generate detailed output regarding the reported events including the concerned system and the applications whose behavior led to raise an alert. We conducted an evaluation of our proposal at industrial scale using 11 consecutive days of logs issued from the KVMs of the european cloud region operated by 3DS OUTSCALE. For this evaluation, we considered three forecasting models: Repetitor, MeanPredictor and LSTM-AE and three error metrics: RMSE, SMAPE, LAR. Considering the accuracy of each model/metric pair, SMAPE emerged as the top-performing error metric, with LSTM-AE proving the most effective model for detecting anomalies. During the monitored period, three KVM crashes were reported, and the MeanPredictor/SMAPE combination was able to predict all of them with at least 80 min of forecasts.

			RMSE SMAPE LAR
	Crash 1	Detected Forecast (minutes)	✗ N.A	✗ N.A	✗ N.A
	Crash 2	Detected Forecast (minutes)	✗ N.A	✗ N.A	✗ N.A
	Crash 3	Detected Forecast (minutes)	✗ N.A	✗ N.A	✗ N.A
		MeanPredictor
			RMSE SMAPE LAR
	Crash 1	Detected Forecast (minutes)	✗ N.A	✓ 120	✓ 120
	Crash 2	Detected Forecast (minutes)	✗ N.A	✓ 80	✓ 80
	Crash 3	Detected Forecast (minutes)	✗ N.A	✓ 85	✗ N.A
		LSTM-AE
			RMSE SMAPE LAR
	Crash 1	Detected Forecast (minutes)	✗ N.A	✗ N.A	✗ N.A
	Crash 2	Detected Forecast (minutes)	✗ N.A	✗ N.A	✗ N.A
	Crash 3	Detected Forecast (minutes)	✗ N.A	✗ N.A	✗ N.A

Table 8 .

 8 1: Méthodes de Détection Automatisée des Anomalies, un Aperçu 8.2.1.1 Méthodes Traditionnelles À notre connaissance, la première méthode de détection d'anomalies basée sur des logs a été évoquée par Liang et al. en 2007 [64]. Le système proposé est basé sur l'ensemble de classificateurs SVM (Support Vector Machines) et vise à détecter les défaillances affectant le supercalculateur IBM BlueGene/L. Dans la catégorie des méthodes traditionnelles, on retrouvera par la suite des approches basées sur d'autres méthodes d'apprentissage automatique telles que Principal Component Analysis [65], Invariant Mining [67] ou du partitionnement [69; 73].

	Année Papier	Méthode	Année Papier	Méthode
	2007	[64]	SVM	2019	[29]	Bi-LSTM
	2009	[65]	PCA	2020	[66]	IF + Autocencoder
	2012	[67]	IM	2020	[68]	BERT + Bi-LSTM
	2016	[69]	Clustering	2021	[70]	Transformer
	2017	[71]	LSTM	2021	[72]	Clustering + CNN
	2017	[73]	Clustering	2021	[74]	Transformer
	2018	[75]	CNN	2021	[76]	GAN
	2018	[77]	Clustering	2021	[78]	Adversarial Network
	2019	[79]	Transformer	2021	[80]	Transformer
	2019	[24]	Bi-LSTM	2022	[81]	TCN

8.2.1.2 Méthodes Basées sur de L'apprentissage Profond

Table 8 .

 8 8.3.2 USTEP, une Méthode de Structuration plus Robuste et plus Rapide USTEP Structurelle Mémorielle ArborescenteBasée sur une structure arborescente qui évolue au fil des messages traités, USTEP est capable traiter en flux des messages de logs. Sa structure mémorielle est basée sur 4 types d'éléments (Figure8.1): 1/ une racine qui sert de point d'entrée aux actions de recherche; 2/ des noeuds internes qui encodent les règles de parcours de l'arbre; 3/ des feuilles qui regroupent les patrons; et 4/ les patrons déjà découverts.Les messages entrent par la racine et suivent les règles des noeuds internes jusqu'à rejoindre une feuille. Ils sont alors comparés aux patrons déjà découverts pour savoir s'ils peuvent y être assimilés. Le cas échéant, un nouveau patron est créé à partir du message courant. Quand une feuille contient trop de patrons, elle divise, se transformant en un noeud interne relié à plusieurs nouvelles feuilles. Cette étape d'éclatement permet de garder un temps de traitement rapide tout en créant des règles de descentes spécifiques aux logs rencontrés. 2: Précision des Méthodes de Structuration Notre évaluation d'USTEP[START_REF] Vervaet | Ustep: Unfixed search tree for efficient log parsing[END_REF] sur 11 jeux de données issus de différents systèmes réels nous a permis de mettre en valeur la supériorité de notre méthode en termes de précision (+3.34% par rapport à Drain) et de robustesse avec un écart type deux fois plus faible que Drain. USTEP est également la seule méthode obtenant une complexité temporelle constante en pire cas. Afin de pouvoir s'adapter aux changements de charge, nous proposons USTEP-UP, une architecture permettant de faire fonctionner plusieurs instances d'USTEP de manière distribué, sans partage de mémoire, mais avec une consistance de structuration[START_REF] Vervaet | Ustep: Unfixed search tree for efficient log parsing[END_REF]. 8.4 Proposition de Monilog un Système Innovant de Détection d'Anomalies à Partir de Logs

		ROOT
	NODE	LEAF	LEAF
		Template	Template
	LEAF	LEAF	Template Template
			Template
	Template Template	Template
	Template
	Figure 8.1:

Table 8

 8 .3) mettent en avant la SMAPE comme la métrique permettant d'obtenir la meilleure précision sur tous les jeux de données. Et le LSTM-AE comme le 8.4 Proposition de Monilog un Système Innovant de Détection d'Anomalies

				à
				Partir de Logs
	Repetitor
		RMSE SMAPE LAR
	# Alertes vérifiées	281	840	453
	# Faux Positifs	577	103	98
	# Inconnu	1 911	2	3
	Total	2 769	945	554
	Precision	0.327	0.891	0.822
	MeanPredictor
		RMSE SMAPE LAR
	# Alertes vérifiées	37	825	352
	# Faux Positifs	60	497	363
	# Unknown	1 274	818	849
	Total	1 371	2 140	1563
	Precision	0.381	0.624	0.493
	LSTM-AE
		RMSE SMAPE LAR
	# Alertes vérifiées	1	10 458	4 085
	# Faux Positifs	0	745	865
	# Inconnu	11 315	2 016	5 842
	Total	11 316 13 219 10 792
	Precision	N.A	0.933	0.825

Table 8 .

 8 3: Précision des Paires Modèle/Métrique modèle plus précis, quand couplé avec la SMAPE. Nous avons inspecté manuellement tous les événements remontés, cependant nous n'avons pas toujours pu établir un diagnostic franc. Nous avons donc fait le choix d'introduire un label Inconnu pour ces événements. Durant la période d'évaluation, trois crashs de KVM ont été remontés par les équipes. Pour ces événements majeurs, seule les combinaisons MeanPredictor/SMAPE et MeanPredictor/LAR ont pu prédire les événements. Cependant, dans le cas de MeanPredictor/SMAPE la prédiction est chaque fois 80 minutes avant occurrence.

https://aws.amazon.com/fr/message/41926/

https://github.com/outscale/ustep-online-log-parser

https://hadoop.apache.org/

https://www.redhat.com/fr/technologies/jboss-middleware/application-platform

https://httpd.apache.org/

https://www.mantisbt.org/

https://www.mysql.com/

https://hadoop.apache.org/

https://hadoop.apache.org/

https://www.openstack.org/

The work presented in this section is mainly extracted from our paper Vervaet, Arthur, Raja Chiky, and Mar Callau-Zori. "Automatisation de la structuration des logs pour le cloud computing." Extraction et Gestion des Connaissances: Actes EGC'2021 (2021).[START_REF] Vervaet | Automatisation de la structuration des logs pour le cloud computing[END_REF].

The work presented in this chapter is heavily based on our paper Vervaet, Arthur, Raja Chiky, and Mar Callau-Zori. "Ustep: Unfixed search tree for efficient log parsing."

IEEE International Conference on Data Mining (ICDM). IEEE, 2021.[START_REF] Vervaet | Ustep: Unfixed search tree for efficient log parsing[END_REF]

https://www.atlassian.com/fr/software/jira

https://kafka.apache.org

https://spark.apache.org

https://www.influxdata.com

Acknowledgements

Pursuing a Ph.D. can be a lonely and challeging activity. During this time I receive tremendous and invaluable support from academic supervisors, colleagues at Outscale and at ISEP, friends and family. I will be forever thankful to them.

USTEP-UP

Subsequently, we will refer to it as U 0 = {P 0 , σ, ϕ}. U 0 , never processes logs, it updates its tree P 0 by merging it with other trees (P l ), the resulting tree also replaces the P l . Initially, every worker starts with identical parsing trees, before any parsing operations: P 0 = P l , ∀l.

As they process logs, those trees will grow and extend their parsing knowledge. To avoid ending with conflictual parsing rules (descent rules) within two trees, we run a modified USTEP version that skips the leaf division step and therefore never splits saturated leaves. Bypassing this step ensures that no internal node is created by a parsing instance, it also influences the time complexity of modified instances now in O(maxT

for classic USTEP instances. With maxT the maximum number of templates attached to one leaf. The main risk is then a continuous slowdown of instances with the increase in the number of log templates. To avoid this, it is the knowledge manager that divides saturated leaves when it merges its tree with an instance one. Delegating the split step to one instance guarantees that the encoded parsing rules won't diverge.

Load balancing systems are widely studied, and many different approaches have already been detailed within the literature [115; 116]. As its intrinsic workflow doesn't affect USTEP-UP design, we will not detail any LBU here. If you want to work with this architecture, we recommend using a state-of-the-art load balancer.

Merging knowledge

Delegating the leaf division process only to K.M. assures that the instances trees won't discover diverging parsing rules. As the worker's instances can't divide leaves, it is guaranteed that at any moment, any internal nodes in a tree P l also exist in P 0 . To distribute the knowledge and refine the instance tree structures, K.M. punctually merges its tree P 0 with a selected instance tree P l , the resulting tree replaces both P 0 and P l . This merging process adds the knowledge collected by the instance to K.M., but it also distributes the knowledge acquired by K.M. toward the other instances to U l . The merging process (Algorithm 5) starts by pointing at the root node of each tree. It uses function P:

(5.5)

With M(t) a matrix composed of the r last count vectors at time t. M(t) is formally defined as follows:

In the case of a perfect predictor:

Compute Forecasting Error

Previous steps model prediction V (t -w) are compared to the actual observation V (t) using different error metrics. They evaluate the forecasting accuracy of the models for the elapsed time step. The performance of Monilog is heavily impacted by the selected model/metric combination.

In our experimental evaluation phase, we used three forecasting models, and three error metrics resulting in 9 different error time series, one for each model/metric pair. We found useful to store error metric outputs into the cold-storage database {5}. This allows us to visualize the error series in real time but also to investigate again certain patterns later without having to repeat the model prediction step, which is expensive in terms of computing units.

Alert Candidate Generation

The error time series are used to generate alert candidates. Points that are considered abnormal regarding the three sigma rules based on the weekly mean error are labeled as alert candidates.

" Anomaly Score " : 4 . 5 , " Applications " : [ { " Name " : " App -1 3 " " Score Weight " : " 9 2 . 4 %" } ] } , { " Name " : " Host -2 7 " , " Anomaly Score " : 3 . 2 , " Applications " : [ { " Name " : " App -1 3 " " Score Weight " : " 5 4 . 2 %" } , { " Name " : " App -9 " " Score Weight " : " 3 6 . 8 %" }

False positive or irrelevant alerting are the main obstacles toward the adoption of an anomaly detection system by the operational teams [START_REF] Debnath | Loglens: A real-time log analysis system[END_REF]. At the end of the previous step, for each alert candidate, the system returned the host as well as the top contributing applications. In the alert consolidation step, it is possible to define additional rules to reduce the number of alerts.

For instance, alerts linked to similar equipment inside a data center can be grouped, or alerting scenarios can be defined based on the involved applications.

An example of an alert report is provided in Listing 5.1, showing how alert candidates from equipment connected to the same data center are used. For each host, the anomaly scores as well as the weights of the top contributing applications are listed. The ability to generate detailed alert report is a strong point toward Monilog acceptance by operational teams as it gives them detailed information about the potential issue while allowing them to fine-tune the alert settings based on their experience and domain knowledge.

We were able to implement an instance of Monilog to monitor the European region of 3DS OUTSCALE. In Chapter 6, we illustrate the practical utilities of Monilog by applying it to the detection of Kernel Virtual Machines crashes. Our proposal is quite flexible and many improvements can still be made. We share our thoughts and perspectives on this in Section 7.2 at the end of this manuscript.

Perspectives

Data Mining

Our research is dedicated to the identification of anomalies within cloud computing platforms, specifically focusing on the analysis of log data. Within this context, we have developed USTEP, a log parsing solution that we propose, along with its distributed version USTEP-UP, which provides a more robust and efficient approach to log parsing.

Although significant progress has been made, parsing errors remains a concern. It is important to acknowledge that no parsing solution can achieve flawless parsing in all logging contexts.

While USTEP has exhibited exceptional performance, we believe there is scope for improvement.

Currently, it lacks the ability to group together logs originating from the same log statement but containing a different number of tokens.

To address this limitation, we propose exploring the incorporation of virtual links between log templates using techniques such as the Longest Common Subsequence (LCS) algorithm.

Although this enhancement may marginally slow down the parsing process, it has the potential to enhance the accuracy of log grouping.

The extraction of data from logs is a critical step for log-based applications in general. The most precise information about the current state of a system is often embedded within the log message and can only be accessed through parsing. Currently, little utilization is made of nonnumerical variables, as existing approaches primarily focus on template mining. This limitation is associated with the diverse nature of the embedded variables, including text, IP addresses, URLs, numbers, hashes, and more. Exploiting this information can be pivotal for numerous log-based applications in domains such as monitoring, security, and usage mining.

Furthermore, we posit that log parsers can be employed for a broader range of data mining tasks beyond log parsing itself.