Li Wu
email: li.wu@elastisys.com

Johan Tordsson
email: johan.tordsson@elastisys.com

Erik Elmroth
email: erik.elmroth@elastisys.com

Odej Kao
email: odej.kao@tu-berlin.de

MicroRCA: Root Cause Localization of Performance Issues in Microservices

Keywords: cause analysis, performance degradation, microservices

Software architecture is undergoing a transition from monolithic architectures to microservices to achieve resilience, agility and scalability in software development. However, with microservices it is difficult to diagnose performance issues due to technology heterogeneity, large number of microservices, and frequent updates to both software features and infrastructure. This paper presents MicroRCA, a system to locate root causes of performance issues in microservices. MicroRCA infers root causes in real time by correlating application performance symptoms with corresponding system resource utilization, without any application instrumentation. The root cause localization is based on an attributed graph that model anomaly propagation across services and machines. Our experimental evaluation where common anomalies are injected to a microservice benchmark running in a Kubernetes cluster shows that MicroRCA locates root causes well, with 89% precision and 97% mean average precision, outperforming several state-of-the-art methods.

I. INTRODUCTION

More and more applications are using microservices architectures (MSA) in domains such as internet of things (IoT) [START_REF] Butzin | Microservices approach for the internet of things[END_REF], mobile and cloud [START_REF] Francesco | Research on architecting microservices: Trends, focus, and potential for industrial adoption[END_REF], to build large-scale systems that are more resilient, robust and better adapted to dynamic customer requirements. With MSA, an application is decomposed into self-contained and independently deployable services with lightweight intercommunication [START_REF] Newman | Building Microservices[END_REF].

To operate microservices reliably and with high uptime, performance issues must be detected quickly and their root causes pinpointed. However, it is difficult to achieve this in microservices systems due to the following challenges: 1) complex dependencies: the number of services can often be hundreds, or thousands, (e.g., Uber has deployed 4000 microservices [START_REF]How Uber Is Monitoring 4,000 Microservices with Its Open Sourced Prometheus Platform[END_REF]). Consequently, the dependencies among services are much more complex than for traditional distributed systems. A performance degradation from one service can propagate widely and cause multiple alarms, making it difficult to locate the root causes; 2) numerous metrics: the number of monitoring metrics available is very high. According to [START_REF] Thalheim | Sieve: Actionable insights from monitored metrics in distributed systems[END_REF], Netflix exposes 2 million metrics and Uber exposes 500 million metrics. It would cause a significant overhead if all these metrics were to be used for performance issue diagnosis; 3) heterogeneous services: technology heterogeneity [START_REF] Dragoni | Microservices: Yesterday, today, and tomorrow[END_REF], one key benefit of MSA, enables development teams to use different programming languages and technology stacks for their services. However, performance anomalies manifest differently for different technology stacks, making it hard to detect performance issues and locate root causes; 4) frequent updates: microservices are frequently updated to meet customers' requirements, (e.g., Netflix updates thousands of times per day [START_REF] Netflix | Amazon, and Apple Care About Microservices[END_REF]). This highly dynamic environment aggravates the difficulty in root cause localization.

To date, many studies have been conducted on root cause diagnostics in distributed systems, clouds and microservices. These either require the application to be instrumented (e.g., [START_REF] Fonseca | X-trace: A pervasive network tracing framework[END_REF], [START_REF] Gan | Seer: Leveraging big data to navigate the complexity of performance debugging in cloud microservices[END_REF]) or numerous metrics to be analyzed (e.g., [START_REF] Thalheim | Sieve: Actionable insights from monitored metrics in distributed systems[END_REF], [START_REF] Mariani | Localizing faults in cloud systems[END_REF]). A third class of approaches [START_REF] Wang | Cloudranger: Root cause identification for cloud native systems[END_REF]- [START_REF] Kim | Root cause detection in a service-oriented architecture[END_REF] avoid these limitations by building a causality graph and inferring the causes along the graph based on application-level metrics. With this approach, potential root causes are commonly ranked through the correlation between back-end services and frontend services. However, this may fail to identify faulty services that have little or no impact on front-end services.

In this paper, we propose a new system, MicroRCA 1 , to locate root causes of performance issues in microservices. MicroRCA is an application-agnostic system designed for container-based microservices environments. It collects application and system levels metrics continuously and detects anomaly on SLO (Service Level Objective) metrics. Once an anomaly is detected, MicroRCA constructs an attributed graph with services and hosts to model the anomaly propagation among services. This graph does not only include the service call paths but also include services collocated on the same (virtual) machines. MicroRCA correlates anomaly symptoms of communicating services with relevant resource utilization to infer the potential abnormal services and ranks the potential root causes. With the correlation of service anomalies and resource utilization, MicroRCA can identify abnormal noncompute intensive services that have non-obvious service anomaly symptoms, and mitigate the effect of false alarms to root cause localization. We evaluate MicroRCA by injecting various anomalies to the Sock-shop2 microservice benchmark deployed on Kubernetes running in Google Cloud Engine (GCE) 3 . The results show that MicroRCA achieves a good diagnosis result, with 89% in precision and 97% in mean average precision (MAP), which is higher than several stateof-the-art methods.

In summary, our contributions are threefold:

• We propose an attributed graph with service and host nodes to model the anomaly propagation in containerbased microservices environments. Our approach is purely based on metrics collected at application and system levels and requires no application instrumentation. • We provide a method to identify the anomalous services by correlating service performance symptoms with corresponding resource utilization, which adapts well to the heterogeneity of microservices. • We evaluate MicroRCA by locating root causes from different types of faults and different kinds of faulty services. On average of 95 test scenarios, it achieves 13% precision improvement over the baseline methods. The remainder of this paper is organized as follows. Related work is discussed in Section II. Section III gives an overview of the MicroRCA system. Root cause localization procedures are detailed in Section IV. Section V describes the experimental evaluation and Section VI concludes the paper.

II. RELATED WORK

In recent years, many solutions have been proposed to identify root causes in distributed systems, clouds and microservices. Log-based approaches [START_REF] Xu | Logdc: Problem diagnosis for declartively-deployed cloud applications with log[END_REF]- [START_REF] Tan | Kahuna: Problem diagnosis for mapreduce-based cloud computing environments[END_REF] build problem detection and identification models based on logs parsing. Even though log-based approaches can discovery more informational causes, they are hard to work in real time and require abnormal information to be hidden in logs. Similarly, trace-based approaches [START_REF] Fonseca | X-trace: A pervasive network tracing framework[END_REF], [START_REF] Gan | Seer: Leveraging big data to navigate the complexity of performance debugging in cloud microservices[END_REF], [START_REF] Zhou | Fault analysis and debugging of microservice systems: Industrial survey, benchmark system, and empirical study[END_REF]- [START_REF] Chen | Pinpoint: Problem determination in large, dynamic internet services[END_REF] gather information through complete tracing of the execution paths, then identify root causes through analyzing the deviation of latencies along the paths, e.g., based on machine learning. These approaches are very useful to debug distributed systems. However, it is a daunting task for developers to understand source code well enough to instrument tracing code.

In addition, there are many metrics-based approaches [START_REF] Thalheim | Sieve: Actionable insights from monitored metrics in distributed systems[END_REF], [START_REF] Gan | Seer: Leveraging big data to navigate the complexity of performance debugging in cloud microservices[END_REF]- [START_REF] Kim | Root cause detection in a service-oriented architecture[END_REF], [START_REF] Chen | Causeinfer: Automated end-toend performance diagnosis with hierarchical causality graph in cloud environment[END_REF], [START_REF] Sharma | Cloudpd: Problem determination and diagnosis in shared dynamic clouds[END_REF], as well as this work. These use metrics from applications and/or additional infrastructure levels to construct a causality graph that is used to infer root causes. Seer [START_REF] Gan | Seer: Leveraging big data to navigate the complexity of performance debugging in cloud microservices[END_REF] and Loud [START_REF] Mariani | Localizing faults in cloud systems[END_REF] use multiple-level metrics to identify root causes. Seer identifies the faulty services and which resource, like CPU overhead, that causes the service performance degradation. It is a proactive method which applies deep learning on a massive amount of data to identify root causes. This approach requires instrumentation of source code to get metrics, meanwhile, its performance may decrease when microservices are frequently updated. Loud identifies any faulty components that generate metrics. It uses the causality graph of KPI metrics from the anomaly detection system directly and locates the faulty components by means of different graph centrality algorithms. However, it requires anomaly detection to be performed on all gathered metrics, which would cause a significant overhead.

MonitorRank [START_REF] Kim | Root cause detection in a service-oriented architecture[END_REF], Microscope [START_REF] Lin | Microscope: Pinpoint performance issues with causal graphs in micro-service environments[END_REF] and CloudRanger [START_REF] Wang | Cloudranger: Root cause identification for cloud native systems[END_REF] identify root causes based on application level metrics only. MonitorRank considers internal and external factors, and proposes a pseudo-anomaly clustering algorithm to classify external factors, then traverses the provided service call graph with a random walk algorithm to identify anomalous services. Microscope considers communicating and non-communicating dependencies between services and constructs a service causality graph to represent these two types of dependencies. Next, it traverses the constructed graph from the front-end service to find the root cause candidates and ranks them based on the metrics similarity between candidate and front-end service. CloudRanger constructs an impact graph with causal analysis and proposes a second-order random walk algorithm to locate root causes. All of these approaches achieve a good performance in identifying faulty services that impact a frontend service. However these methods commonly fail to identify root causes from backend services of the type that scarcely impact front-end services. Similar to these works, we also use a graph model, and localize root causes with an algorithm similar to random walk. However, we correlate anomalous performance symptoms with relevant resource utilization to comprehensively represent services anomaly, which improves the precision of root cause localization.

III. SYSTEM OVERVIEW

In this section, we briefly introduce MicroRCA and its main components.

A. MicroRCA Overview

Figure 1 shows the overview of MicroRCA. Data collection module collects metrics from application and system levels. The application level metrics continuously, in particular the response times of microservices, are used to detect performance issues, and metrics from both levels are used to locate root causes. Once anomalies are detected, the cause analysis engine constructs an attributed graph G with service and host nodes to represent the anomaly propagation paths. Next, the engine extracts an anomalous subgraph SG based on detected anomalies and infers which service is the most likely to cause the anomalies.

B. Data Collection

MicroRCA is designed to be application-agnostic. It collects application and system levels metrics from a service mesh [START_REF]What's a service mesh? And why do I need one?[END_REF] and monitoring system separately and stores them in a time series database. In container-based microservices, system-level metrics include container and host resource utilization, as illustrated by container and host in "Model Overview" in Figure 1. Application level metrics include response times between two communicating services, etc.

C. Anomaly Detection

Anomaly detection is the starting point of root cause localization. MicroRCA leverages the unsupervised learning algorithm Distance-Based online clustering BIRCH [START_REF] Gulenko | Detecting anomalous behavior of black-box services modeled with distance-based online clustering[END_REF] anomaly detection method. We use the slow response time of a microservice as the definition of an anomaly.

D. Cause Analysis Engine

Once anomalies are detected, the cause analysis engine starts to locate root causes. The engine is composed of three main procedures: attributed graph construction, anomalous subgraph extraction, and faulty service localization. The engine constructs an attributed graph to represent the anomaly propagation through services and hosts. Next, it extracts the anomalous subgraph and locates the faulty services with a graph centrality algorithm named Personalized PageRank [START_REF] Jeh | Scaling personalized web search[END_REF].

IV. ROOT CAUSE LOCALIZATION

In this section, we describe the three procedures to identify root causes, namely attributed graph construction (Section IV-A), anomalous subgraph extraction (Section IV-B) and faulty services localization (Section IV-C).

A. Attributed Graph Construction

MicroRCA constructs an attributed graph to represent the anomaly propagation in microservices environments, which is based on the observation that anomalies propagate not only among services along the service call paths but also to services collocated on the same (virtual) machines [START_REF] Weng | Root cause analysis of anomalies of multitier services in public clouds[END_REF], [START_REF] Ibidunmoye | Performance anomaly detection and bottleneck identification[END_REF].

Our attributed graph consists of a set of services nodes S = {s 1 , s 2 , ..., s k } and host nodes H = {h 1 , h 2 , ..., h m } as shown in Figure 1. For each service node s i , we add edges to all other service s j it communicates with, and all hosts h k it runs on.

We discover the graph nodes and their dynamic relationships by enumerating and parsing the metrics monitored at application and system levels. The graph nodes are interconnected as follows. When service s i sends requests to service s j , we add a directed edge from s i to s j . If a containerized service s i is allocated in host h j , we add a directed edge from s i to h j .

To construct the attributed graph, we use the metrics of a certain time frame before the anomaly was detected. We assume all microservices run reliably during this time frame and the collected metrics can precisely provide relationships among all communicating services and located hosts.

B. Anomalous Subgraph Extraction

After constructing the attributed graph, we proceed to extract the anomalous subgraph based on detected anomalies and assign anomaly-related attributes to the nodes. An anomalous subgraph SG is a connected subggraph that represents the anomaly propagation through services and hosts. Note that we use the response times between communicating services, which are the edges between service nodes in the attributed graph as the anomaly detection targets. Once the response time of an edge is determined to be an anomaly, we consider the edge as an anomalous edge and the origin of the edge as an anomalous node.

To obtain the anomalous subgraph, we first extract the anomalous service nodes. In Figure 2(a), the response times between (s 1 , s 2 ), (s 2 , s 3 ), (s 2 , s 4 ) are anomalous. Therefore, we extract the anomalous edges and take the origins of the edges (s 2 , s 3 , s 4 in the example), as the anomalous service nodes. Figure 2(c) depicts the anomalous edges as dashed lines and anomalous services nodes as shadowed nodes. Next, we compute the average of anomalous response times for each anomalous service node, denoted as rt a, to a node attribute. Finally, we add nodes and edges that are connected to anomalous service nodes. Figure 2(c) shows the extracted anomalous subgraph, where adjacent services nodes s 1 and s 5 and adjacent host nodes h 1 and h 2 probably are impacted.

C. Faulty Services Localization

Once an anomalous subgraph is extracted, we start to locate the most likely faulty services. We firstly compute the similarity between connected nodes by weighing the subgraph, then assign anomaly scores to anomalous service nodes, and finally locate root causes based on a graph centrality algorithm.

Anomalous Subgraph Weighing: Edge weights represent similarity between pairs of nodes. Similar to previous work [START_REF] Mariani | Localizing faults in cloud systems[END_REF]- [START_REF] Kim | Root cause detection in a service-oriented architecture[END_REF], we use the Pearson correlation function to measure similarity, henceforth denoted as corr(i, j). In particular, weights are computed in following three ways:

• Weight of an anomalous edge w a . From anomalous subgraph extraction (Section IV-B), we get a list of anomalous edges. To each anomalous edge, we assign a constant value α ∈ (0, 1], which denote the anomaly detection confidence. If the real anomalies are correctly detected, we would set α higher, and vice versa. In Figure 2(d), weights w (s1,s2) , w (s2,s3) , w (s2,s4) are assigned α. • Weight between an anomalous service node and a normal service node w n . The weight is assigned the correlation between two response times: the response time between an anomalous service node and a normal service node, and the average anomalous response time rt a of an anomalous service node. In Figure 2(d), the weight w (s1,s3) between the normal service node s 1 and the anomalous node s 3 is the correlation between the response time rt (s1,s3) between s 1 and s 3 , and the response time rt (s2,s3) which is the average anomalous response time of s 3 . • Weight between an anomalous service node and a normal host node w h . We use the maximum correlation coefficient between the average anomalous response time rt a of an anomalous service node, and the host resource utilization metrics U h , including CPU, memory, I/O, network, to represent the similarity between service anomaly symptoms and the corresponding resource utilization.

Given that the response time of both normal and abnormal services have strong correlations with the host resource utilization, we take the average of in-edge weights w in as a factor of the similarity. Thus, w h ij between service node i and host node j can be formulated as:

w h ij = max k:u k ∈U h (j) corr(u k , rt a(i)) • w I (i) (1)
To summarize, the weight w ij between node i and node j is given by Equation 2, where rt a(i) denotes the average anomalous response time of anomalous service node i.

w ij =          α,
if rt (i,j) is anomalous, corr(rt (i,j) , rt a(j)), if i ∈ S and normal, corr(rt (i,j) , rt a(i)), if j ∈ S and normal, as per Equation 1if j ∈ H and normal.

(

) 2
The procedure of anomalous subgraph weighing is presented in Algorithm 1. In this algorithm, we iterate over the anomalous nodes and compute the weights for in-edges in lines L1-L8 and for out-edges in lines L9-L15.

Assigning Service Anomaly Score: We calculate anomaly scores for anomalous service nodes and assign them to a node attribute, denoted as AS ∈ [0, 1].

To quantify the anomaly, we take an average weight of service node w(s j ) that indicates the impact to linked nodes. Furthermore, In container-based microservices, we assume container resource utilization is correlated with service performance. We thus complement the service anomaly with the maximum correlation coefficient between the average anomalous response time rt a of the anomalous service node, and its container resource utilization U c . To summarize, given anomalous service node s j , the anomaly score AS(s j ) is defined as:

AS(s j ) = w(s j ) • max k:u k ∈Uc(sj ) corr(u k , rt a(s j )) (3)
Localizing Faulty Services: We locate the faulty services from the anomalous subgraph with a graph centrality algorithm -Personalized PageRank [START_REF] Jeh | Scaling personalized web search[END_REF], which proved a good performance in capturing anomaly propagation in previous work [START_REF] Mariani | Localizing faults in cloud systems[END_REF], [START_REF] Wang | Cloudranger: Root cause identification for cloud native systems[END_REF], [START_REF] Kim | Root cause detection in a service-oriented architecture[END_REF], [START_REF] Weng | Root cause analysis of anomalies of multitier services in public clouds[END_REF]. In Personalized PageRank, the Personalized PageRank vector (PPV) v is regarded as the root cause score for each node. To compute PPV, we first define P as the transition probability matrix, where P ij = wij j wij if node i links to node j, and P ij = 0 otherwise. A preference vector u denotes the preference of nodes, which we assign the value of anomaly scores of the nodes in our method. This way, the anomaly-related service nodes are visited more frequently when the random teleportation occurs. Formally, the personalized PageRank equation [START_REF] Jeh | Scaling personalized web search[END_REF] is defined as:

v = (1 -c)P v + cu (4)
where c ∈ (0, 1) is the teleportation probability, indicating that each step jumps back to a random node with probability c, and with probability 1 -c continues forth along the graph. Typically c = 0.15 [START_REF] Jeh | Scaling personalized web search[END_REF]. After ranking, we removed the host nodes from the ranked list as MicroRCA is designed to locate faulty services. As the link between service nodes in the anomalous subgraph represents the service call-callee relationship, we need to reverse the edges before running the localization algorithm. We give an example of the ranked list of root causes in Figure 2(e).

V. EXPERIMENTAL EVALUATION

In this section, we present the experimental setup, experimental results, a comparison with state-of-the-art methods, and discuss the characteristics of our approach.

A. Experimental Setup

Testbed: We evaluate MicroRCA in a testbed established in Google Cloud Engine (GCE) where we set up a Kubernetes cluster, deploy the monitoring system and service mesh, and run the benchmark named Sock-shop 2 . There are four worker nodes and one master node in the cluster, three of the worker nodes are dedicated to microservices and one for data collection. In addition, one server of the cluster runs the workload generator. Table I describes the detail configuration of the hardware and software in the testbed.

Benchmark: Sock-shop 2 is a microservice demo application that simulates an e-commerce website that sells socks. It is a widely used microservice benchmark designed to aid demonstration and testing of microservices and cloudnative technologies. Sock-shop consists of 13 microservices, which are implemented in heterogeneous technologies and intercommunicate using REST over HTTP. In particular, frontend serves as the entry point for user requests; catalogue provides a sock catalogue and product information; carts holds shopping carts; user provides the user authentication and store user accounts, including paymenet cards and addresses; orders places orders from carts after user log-in through the user service, then process the payment and shipping from the payment and shipping services separately. To each microservice, we limit the CPU resource to 1vCPU and memory to 1GB. The replication factor of each microservice is set with 1.

Workload Generator: We develop a workload generator using Locust4 , a distributed, open-source load testing tool that simulates concurrent users in an application. The workload is selected to reflect real user behavior, e.g., more requests are sent to the entry points front-end and catalogue, and fewer to the shopping carts, user and orders services. We distribute requests to front-end, orders, catalogue, user, carts with five locust slaves, and provision 500 users that in total generate about 600 queries per second to sock-shop. The request rate of each microservice is listed in Table II.

Data Collection: We use the istio 5 service mesh, which in term uses Prometheus 6 , to collect service-level and containerlevel metrics and node-exporter 7 to collect host-level metrics. Prometheus is configured to collect metrics every 5 seconds and sends the collected data to MicroRCA. In service-level, we collect response time between each pair of services. In both container-level and host-level, we collect CPU usage, memory usage, and the size of total sent bytes.

Faults Injection: Our method is applicable to any type of anomaly that manifests itself as increased microservice response time. In this evaluation, we inject three types of faults commonly used in the evaluation of the state-of-theart approaches [START_REF] Mariani | Localizing faults in cloud systems[END_REF], [START_REF] Lin | Microscope: Pinpoint performance issues with causal graphs in micro-service environments[END_REF], [START_REF] Kang | Peerwatch: A fault detection and diagnosis tool for virtualized consolidation systems[END_REF] to sock-shop microservices to simulate the performance issue. (1) Latency, we use the tc8 to delay the network packets; (2) CPU hog, we use stress-ng9 , a tool to load and stress compute system, to exhaust CPU resources. As microservice payment is non-compute intensive whose CPU usage is only 50mHz, we exhaust its CPU heavily with 99% usage. (3) Memory leak: we use stress-ng to allocate memory continuously. As microservice carts and orders are CPU and memory intensive services, and memory leak causes CPU overhead [START_REF] Gulenko | Detecting anomalous behavior of black-box services modeled with distance-based online clustering[END_REF], we only provision 1 virtual machine. The details of the injected faults are described in Table III.

To inject performance issues in microservices, we customize the existing sock-shop docker images with installing above faults injection tools. Each fault lasts 1 minute. To increase the generality, we repeat the injection process 5 times for each fault. This produces a total of 95 experiment cases.

Baseline Methods: We compare MicroRCA with some baseline methods as follows:

• Random selection(RS): Random selection is a way that an operating team use without specific domain knowledge of the system. Every time, the operating team randomly selects one microservice from the uninvestigated microservices to investigate until they find the root cause.

• MonitorRank [START_REF] Kim | Root cause detection in a service-oriented architecture[END_REF]: MonitorRank uses a customized random walk algorithm to identify the root cause. It is similar with personalized PageRank [START_REF] Jeh | Scaling personalized web search[END_REF] we adopt in MicroRCA.

As we only consider the internal faults in microservices, we do not implement the pseudo-anomaly clustering algorithm to identify the external factors. To implement MonitorRank, we use their customized random walker algorithm to identify root causes in the extracted anomalous subgraph .

• Microscope [START_REF] Lin | Microscope: Pinpoint performance issues with causal graphs in micro-service environments[END_REF]: Microscope is another graph-based method to identify faulty services in microservices environment and it also takes sock-shop as the benchmark.

To implement Microscope, we construct the services causality graph with paring the service-level metrics, then use their cause inference, which traverses the causality graph with the detected anomalous services nodes, to locate root causes. The anomaly detection methods in MonitorRank and Microscope fail to detect anomalies, especially for non-obvious anomalies, like payment service. As we in our experiments aim to compare the root cause localization performance, we thus use the same results from our anomaly detection module.

Evaluation Metrics: To quantify the performance of each algorithm on a set of anomalies A, we use following metrics:

• Precision at top k denotes the probability that the top k results given by an algorithm include the real root cause, denoted as P R@k. A higher P R@k score, especially for small values of k, represents the algorithm correctly identifies the root cause, Let R[i] be the rank of each cause and v rc be the set of root causes. More formally, P R@k is defined on a set of given anomalies A as:

P R@k = 1 |A| a∈A i<k (R[i] ∈ v r c ) (min(k, |v r c |)) (5)
• Mean Average Precision (MAP) quantifies the overall performance of an algorithm, where N is the number of microservices:

M AP = 1 |A| a∈A 1≤k≤N P R@k. (6)

B. Experimental Results

Figure 3 shows the results of our proposed root cause localization method for different types of faults and sock-shop microservices. We observe that all services achieve a MAP in the range of 80%-100% and an average PR@1 over 80% except shipping and payment. The average precision of shipping and payment that is lower than other services is likely for three reasons. First, payment is a non-compute intensive service, even though we exhaust the resource of CPU and memory, its response time is scarcely impacted. Second, there are few requests to shipping and payment and they do not request any other services, which makes the response time increase less obviously than other faulty services. Third, in order to detect anomalies in their experimental cases, we use a small threshold in anomaly detection module, which causes more false alarms.

Table IV demonstrates the performance of MicroRCA in different types of faults and microservices. It shows that MicroRCA can achieve almost 90% in terms of PR@1 and effectively locate all root causes in the top three faulty services.

C. Comparisons

To evaluate the performance of MicroRCA further, we apply it and the baseline methods on all experimental cases.

We compare the overall performance of all methods and their performances in identifying different types of faults. Table V shows the performance, in terms of PR@1, PR@3 and MAP, for all methods. We can observe that MicroRCA outperforms the baseline methods in overall. In particular, MicroRCA achieves a precision of 89% and MAP of 97%, which are at least 13% and 15% higher than the baseline methods separately. In general, Microscope performs well in CPU hog and memory leak when the anomalies are detected correctly but worse in fault latency when more false alarms are detected. However, MicroRCA performs well in all types of faults, and achieves average improvement of 24% in MAP comparing to MonitorRank and Microscope.

Next, we compare the performance of each method on different microservices. Figure 4 shows the comparison results, in terms of PR@1, PR@3 and MAP, on different services. We can see that MonitorRank performs well in identify dominating nodes which have large degrees, like microservice orders. However it fails to identify leaf nodes, like microservice payment. This is because MonitorRank calculates the similarity based on the correlation between front-end services and backend services. The anomaly of microservice payment decreases the correlation during propagation and thus the root cause localization fails. On the contrary, Microscope performs better in identifying leaf nodes, such as the microservice payment, but worse in identify dominating node, like microservice orders. This is because Microscope traverses the graph based on the detected anomalies and put the anomalous child nodes into a list of potential causes, which makes it fail to identify the dominating nodes when alarms are reported from child nodes.

Comparing to MonitorRank and Microscope, MicroRCA generally has a good performance for all the services, no matter dominating nodes or leaf nodes. Finally, we compare the root cause localization performance of each method against the anomaly detection results to evaluate their sensitivity to the performance of anomaly detection. We compute the F1-score of each experimental case and count the number of different ranks of all experiment cases to different F1-scores. F1-score is computed as the harmonic mean of precision and recall: 2 × precision×recall precision+recall . In our experiment, we only inject one fault to one microservice, which means only one root cause in a case. Thus in each case, the recall is 1 if the root cause is in the ranked list, otherwise 0. Precision is the number of detected root causes divided by the number of detected anomalies. As RS(random selection) is not related to anomaly detection results, we analyze MonitorRank, Microscope and our MicroRCA only.

Figure 5 shows the number of different ranks against F1score for MonotorRank, Microscope and MicroRCA separately. To show the results compactly, we set rank with 10, which is higher than the total number of microservices, when the root cause is failed to locate. We can see all of them have a good performance when the faulty services are correctly detected. However, when false alarms are frequent, Microscope cannot identify root causes in some cases; MonitorRank can identify root causes, but root causes are always low-ranked; MicroRCA identifies the root cause correctly both in scenarios with high and low F1-score.In particular, MicroRCA identifies 13 out of 18 faults in top 1 when F1-score is less than 0.4, which is higher than the other two methods.

D. Discussion

Here we discuss the overhead and sensitivity of our proposed MicroRCA system.

Overhead: The overhead of MicroRCA on the microservices system is mostly caused by the data collection module, which collects the application-level and system-level metrics continuously. Table VI shows the overhead of data collection and the execution time of modules in MicroRCA. We can see that the execution time of MicroRCA is short enough to run in real-time given a data collection interval of 5 seconds. However the overhead of the data collection module is a little high. In the future, we would like to explore lightweight monitoring tools to reduce this overhead.

Sensitivity: To evaluate the sensitivity of MicroRCA to the anomaly detection confidence(α), which is assigned to the anomalous edges in weighing anomalous subgraph (section IV-C). We analyze the performance of MicroRCA with different values of α. Figure 6 shows the performance of MicroRCA, in terms of overall PR@k(k=1,2,3) and MAP of different types of faults over all the test cases, when α ranges from 0.15 to 1.0. We can see that the performance of MicroRCA changes with the weight assigned to the anomalous nodes. PR@1 increases to the maximum when α = 0.55 and drops when α 0.7; PR@3 is always 1 when α is less than 0.55, and drops after that. MAP of all type of faults are relatively stable. and keeps over 96%. In this case, we choose 0.55 as the weight of anomalous edges where the PR@1 and PR@3 are the maximum. In practise, as different anomaly detection methods have different performances, we need to tune the anomaly detection confidence(α) according to the anomaly detection module.

Furthermore, we analyze the sensitivity to anomaly detection results with different thresholds in anomaly detection module. Figure 7 two baseline methods in solid lines and F1-score in dash line when threshold ranges from 0.01 to 0.1. Note that we set α to the constant 0.55 when MicroRCA regularly performs well. We can see that the three methods share the similar pattern with F1-score, and MicroRCA performs well in a range from 0.02 to 0.065, which is wider than other two methods. As all three methods performs well when threshold is 0.045, we use it in the anomaly detection module for all above analysis.

VI. CONCLUSION AND FUTURE WORK

In this paper, we proposed a new application-agnostic system, MicroRCA, for root cause localization of performance anomalies in container-based microservices. It constructs an attributed graph model and correlates service anomalous performance symptoms with corresponding resource utilization to infer the anomalous microservices. Experimental evaluations with a microservice benchmark show that MicroRCA achieves 89% in precision and 97% in mean average precision (MAP). In particular, our method improves the precision in identifying root causes from both dominating services with large degrees and leaf services with non-obvious anomaly symptoms, where the state-of-the-art methods fall short.

As the involved tools (Kubernetes, Istio, etc) are not designed to operate with sub-second monitoring intervals, this adds a limitation to how fast anomalies can be detected with our approach, rendering our system unable to identify anomalous services for which the anomalies propagate faster than our monitoring interval. In the future, we would like to improve our system to overcome this limication. Furthermore, we would like to study the automatic remediation methods for performance degradations in microservices environments.

Figure 1 .

 1 Figure 1. Overview of MicroRCA main components and root cause localization workflow.

  Figure 2(b) shows an example of a constructed attributed graph from the initial microservices environment.

  Figure 2(d) shows these three types of weights along the edges.

Algorithm 1 : 6 8 end 9 for 13 Assign

 168913 Anomalous Subgraph Weighing Input: Anomalous subgraph SG, anomalous edges, anomalous nodes, anomaly response time rt a, host metrics U h , response times of edges rt Output: Weighted SG 1 for node v j in anomaly nodes do 2 for edge e ij in in-edges of v j do 3 if rt (i,j) in anomalous edges then 4 Assign α to w ij ; 5 else Assign corr(rt a(j), rt (i,j) ) to w ij 7 end edge e jk in out-edges of v j do 10 if node v k is service node then 11Assign corr(rt a(j), rt (j,k) ) to w jk ; 12 else avg(w in (j)) × max(corr(rt a(j), U h (k)))

Figure 3 .

 3 Figure 3. The result of PR@1 and MAP.

  shows the MAP of MicroRCA and other

Figure 4 .Figure 5 .

 45 Figure 4. The comparison results of PR@1,PR@3 and MAP for different microservices.

Figure 6 .

 6 Figure 6. Performance against anomaly detection confidence(α).

Figure 7 .

 7 Figure 7. Performance against anomaly detection threshold.

  as the

			Data Collection
							Cause Analysis Engine
		Anomaly Detection		anomalies	Attributed Graph Construction	G	Anomalous Extraction Subgraph	SG	Localization Faulty Service
	s 1	RT: 45ms	s 2	container{cpu, mem,
					…}			service node
	h 1		h 2	host{cpu: 75%, mem: 62%,			host node
	Model Overview	…}

Table I HARDWARE

 I AND SOFTWARE CONFIGURATION USED IN EXPERIMENTS.

		Hardware Configuration
	Component	Master node	Worker node(x4)	Workload generator
	Operating System	Container-Optimized OS	Container-Optimized OS	18.04.2 LTS
	vCPU(s)	1	4	6
	Memory(GB)	3.75	15	12
		Software Version
	Kubernetes	Istio	Prometheus	Node-exporter
	1.14.1	1.1.5	2.3.1	v0.15.2

Table II REQUEST

 II RATES SENT TO MICROSERVICES.

	Microservices	front-end	catalogue	user	carts	orders
	Concurrent users		100	100		100	100	100
	Request rate(/s)		200	300		50		50	20
			Table III
	THE DETAIL OF INJECTED FAULTS.
	Microservices	front-end	catalogue	user	carts	orders	payment	shipping
	Latency(ms)	200	200	200	200		200		200	200
	CPU Hog(vcpu*%)	-	2*95	2*95	2*95	2*95	2*99	2*95
	Memory Leak(vm*MB)	-	2*1024	2*1024	1*2048	1*2048	2*1024	2*1024

Table V PERFORMANCE

 V OF EACH ALGORITHM.

	Metric	RS	MonitorRank	Microscope	MicroRCA	Improvement to	Improvement to
						MonitorRank (%)	Microscope(%)
	Overall
	PR@1	0.21	0.41	0.79	0.89	118	13.3
	PR@3	0.46	0.65	0.86	1.0	53.2	15.9
	MAP	0.58	0.73	0.85	0.97	33.7	14.7
	Latency
	PR@1	0.17	0.23	0.66	0.89	287	34.8
	PR@3	0.46	0.66	0.71	1.0	51.5	40.8
	MAP	0.58	0.73	0.7	0.97	32.9	38.6
	CPU Hog
	PR@1	0.23	0.6	0.87	0.9	50	3.5
	PR@3	0.5	0.67	0.93	1.0	49.3	7.5
	MAP	0.61	0.77	0.92	0.97	26	5.4
	Memory Leak
	PR@1	0.23	0.43	0.87	0.9	109	3.5
	PR@3	0.37	0.63	0.97	1.0	58.7	3
	MAP	0.57	0.68	0.95	0.98	44.1	3.2

Table VI THE

 VI OVERHEAD OF MICRORCA.

	Modules	Cost
	Data collection	0.6 vCPU and 1511MB RAM
	Anomaly Detection	0.01s (8 cores)
	Attributed Graph Construction	3.3s (8 cores)
	Root Cause Localization	0.03 (8 cores)

MicroRCA stands for Microservices Root Cause Analysis

Sock-shop -https://microservices-demo.github.io/

Google Cloud Engine -https://cloud.google.com/compute/

Locust -https://locust.io/

Istio -https://istio.io/

Prometheus -https://prometheus.io/

Node-exporter -https://github.com/prometheus/nodeexporter

tc -https://linux.die.net/man/8/tc

stress-ng -https://kernel.ubuntu.com/ cking/stress-ng/

ACKNOWLEDGMENT

This work is part of the FogGuru project which has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 765452. The information and views set out in this publication are those of the author(s) and do not necessarily reflect the official opinion of the European Union. Neither the European Union institutions and bodies nor any person acting on their behalf may be held responsible for the use which may be made of the information contained therein.


Alejandra Ruiz
email: alejandra.ruiz@tecnalia.com

Javier Puelles
email: javier.puelles@tecnalia.com

Jabier Martinez
email: jabier.martinez@tecnalia.com

Thomas Gruber
email: thomas.gruber@ait.ac.at

Martin Matschnig
email: martin.matschnig@siemens.com

Bernhard Fischer
email: bernhard.bf.fischer@siemens.com

Preliminary Safety and Security Co-engineering Process in the Industrial Automation Sector

Keywords: Safety, Security, Co-Engineering, Dependability, Quality assurance, IEC 61508, ISA 62443

The Industrial Automation Sector has a long tradition of showing compliance on functional safety. Ultimately, security was taken into account only at production phase and with a reactive approach. However, this domain is experimenting an increasing need to incorporate cybersecurity mechanisms and to provide evidences on security-related standards and applying security by design principles. Both domains have their own regulations defining specific life-cycles. In this work we analyzed IEC 61508 (safety-related) and ISA 62443 (security-related) standards to 1) identify commonalities and create a mapping model, and 2) propose a combined process in the context of safety and security co-engineering. Our approach is qualitatively evaluated by experts on the standards and by practitioners of this domain.

Introduction

It is becoming increasingly apparent that organizations should look at the safety and security challenges of their systems in a unified way, which is contradictory with a long tradition of safety silos and security silos [START_REF] Pomante | The AQUAS ECSEL Project Aggregated Quality Assurance for Systems: Co-Engineering Inside and Across the Product Life Cycle[END_REF]. Security is usually a requirement in order to ensure safety [START_REF] Netkachova | Security-informed safety[END_REF], but different teams are usually responsible for these two tasks separately (one for safety and another one for security). This is justified because of the intrinsic complexity of each area making it difficult to be experts on both. They follow different lifecycles, they need to carry out certification for their own specialized and complex standards, and they use different terminologies complicating their communication.

Combined analysis requires additional effort during early development and it is not certain if it will pay off later. Unfortunately, there are no strong evidences about the cost-effectiveness of dealing with safety and security together. However, safety and security separation led to redundant work and late identification of trade-offs in safety and security requirements [START_REF] Pomante | The AQUAS ECSEL Project Aggregated Quality Assurance for Systems: Co-Engineering Inside and Across the Product Life Cycle[END_REF]. For example, introducing a ciphering security mechanism can increase the time needed to perform a safety-critical system task leading to exceed the worst-case execution time. The motivation of this work is thus to avoid work redundancies optimizing the use of human resources and the time needed while favouring the early identification of potential trade-offs.

This paper is focused on the Industrial Automation sector. Industrial Control Systems (ICS) such as SCADA, Real Time Units, Programmable Logic Controllers or Distributed Control Systems, are connected to hardware and physical systems and they are used to control industrial processes (e.g., processing plants, factories, power plants) and to support key infrastructures (e.g., transportation, airports, seaports and buildings). Failures that end up affecting the ICS may imply not only harming the system itself, or negatively affecting the production or service availability, but also may cause environmental problems or endanger the health and safety of human lives. Examples exploiting security vulnerabilities in safetycritical industrial systems are numerous (e.g., [START_REF]arstechnica: Hackers trigger yet another power outage in ukraine, ars TECHNICA[END_REF][START_REF]BBC: report 'timeline: How stuxnet attacked a nuclear plant[END_REF]3]).

Security concerns are being introduced in the ICS domain, which has been traditionally safetyoriented, focusing on security informed safety [START_REF] Netkachova | Security-informed safety[END_REF] where security issues could impact the system's overall safety. ICSs are increasingly more software-intensive and "smart" (e.g., ICS are key in the Industry 4.0 paradigm [START_REF] Lu | Industry 4.0: A survey on technologies, applications and open research issues[END_REF]), with needs for data management and communication with external entities. Thus, cyber-security requirements and risks analysis are critical. New challenges arose, next to only considering security informed safety, privacy issues (a relevant aspect of cyber-security) where safety is not impacted must also be considered (e.g., personal data can be exposed in case of a data breach).

ICSs typically include proprietary software or hardware, as well as communication protocols with limited security. This is mainly because they were designed to meet functional properties in an effective and safe way, and usually overlooking security criteria as they were not traditionally connected with external networks or integrated in a highly-connected and dynamic systems-of-systems context. Besides, the vulnerability of these systems is aggravated due usually to large number of functional entities present in various production installations (e.g., distributed control systems). Also, the locations can be remote and, in some cases, an entity can be even running without human supervision. These entities were usually provided by different vendors, used to have a long operational life, and the intellectual property of the organizations used to be embedded in the deployed systems. Also, most of the industrial processes controlled by ICSs use to be available and functional 24 hours per day, 365 days per year. As summary, the exposure and surface for cyber-attacks on security vulnerabilities on safety-critical ICSs are increasingly high.

The focus of this paper is on a collaborative approach for co-engineering functional safety and cybersecurity. We focused on the IEC 61508 international standard entitled "Functional Safety of Electrical/ Electronic/ Programmable Electronic Safety-related Systems" [START_REF]Functional safety of electrical/electronic/programmable electronic safety-related systems[END_REF], and the ISA 62443 international standards for implementing secure ICSs [START_REF]Industrial communication networks network and system security[END_REF] (more concretely ISA 62443 Part 4.1 "Secure product development life-cycle requirements") were analyzed to identify commonalities and differences between the former (functional safety) and the latter (cyber-security). This in-depth analysis enabled us to propose a combined life-cycle satisfying the needs of the two normative spaces and reducing the risk of duplicated efforts during the certification processes. This first attempt to create a co-engineering process is intended to identify the common links during the project life-cycle and at which phase is most interesting to define interaction points for trade-off discussions. Thus, this process can be used to replace current practices towards a more effective use of the resources when compliance is required for these two widely required standards for ICSs.

This paper is structured as follows: Section 2 presents related works. Section 3 presents background information on the targeted standards. Section 4 presents the methodology and results of the analysis of the standards. Section 5 presents the suggested co-engineering life-cycle. Section 6 presents a qualitative discussion of the results, its strengths and limitations. Finally, Section 7 concludes and presents the further plans.

Related work

Some attempts have been made in the past to compare and harmonize different standards analyzing their similarities and divergences. For example, in [START_REF] Blanquart | Criticality categories across safety standards in different domains[END_REF][START_REF] Machrouh | A cross-domain comparison of software development assurance standards[END_REF][START_REF] Machrouh | Cross domain comparison of System Assurance[END_REF][START_REF] Ruiz | Reuse of safety certification artefacts across standards and domains: A systematic approach[END_REF], different functional safety standards were analyzed across different domains where certification is requested. For instance, in [START_REF] Ruiz | Reuse of safety certification artefacts across standards and domains: A systematic approach[END_REF] a systematic approach is proposed for reusing evidences of a safety-critical component created to comply with a railway's functional safety standard during the compliance process with another standard from the avionics domain. While the focus of trying to reduce redundant work in co-certification is similar; these works focused only on functional safety standards. On the contrary, our focus is on safety and security, requiring a co-engineering process.

According to an international survey [START_REF] Kaspersky | The state of industrial cybersecurity 2019[END_REF], managing risks for cyber-security is the activity with more clear priority for companies with ICSs (79% classified it as a main priority and 21 as minor priority). Followed by better complying with regulations, and improving the efficiency on these processes. Regarding this last point, lack of communication between engineering disciplines and their different focus and approaches on security are considered a major issue [START_REF] Kaspersky | The state of industrial cybersecurity 2019[END_REF]. Some approaches have studied the idea of combining hazards and threats analysis such as SAHARA (Security-aware Hazard Analysis and Risk Assessment) [START_REF] Macher | Sahara: A security-aware hazard and risk analysis method[END_REF] or FMVEA (Failure Mode Vulnerability and Effect Analysis) [START_REF] Schmittner | Security application of failure mode and effect analysis (fmea)[END_REF]. In [START_REF] Verma | Combined approach for safety and security[END_REF], the idea has progressed and they proposed a combination of different techniques such as Fault Tree Analysis (FTA) [START_REF]IEC 61025: Fault tree analysis[END_REF][START_REF] Ruijters | Fault tree analysis: A survey of the state-of-the-art in modeling, analysis and tools[END_REF], Stochastic Colored Petri Net (SCPN) [START_REF] Gehlot | An introduction to systems modeling and simulation with colored petri nets[END_REF], Attack Trees Analysis (ATA) and FMVEA for safety-security analysis. Our approach is not to provide a new method for safety and security co-analyses but to propose a combined life-cycle process to reduce the efforts to comply with the standards in isolation and detect the most suitable times for trade-off discussion between the different teams. A related work is also the preliminary co-engineering concepts envisioned for the industrial drives domain [START_REF] Gruber | Co-engineering-in-the-loop[END_REF].

3 Background on the standards under study

IEC 61508

The IEC 61508 "Functional Safety of Electrical/Electronic/Programmable Electronic Safety-related Systems (E/E/PE, or E/E/PES)" standard [START_REF]Functional safety of electrical/electronic/programmable electronic safety-related systems[END_REF] is applicable to all kind of programmable electronic safety systems in any industry. It is a general standard proposing a generic approach and it is usually derived for each application domain for specialization. Even though the standard is seen as a regulation for product developers, it is a general regulation that applies to end-users, system integrators and product developers. It is the main standard on the topic of functional safety for ICSs and it defines the requirements of programmable safety systems for the safety of facilities.

The standard covers the whole safety life-cycle, from the initial concept until the system decommissioning or disposal. IEC 61508 proposes three complementary life-cycles. The system level life-cycle can be considered as the leading one. One of its phases, Realisation, is decomposed in two life-cycles which are executed in parallel: The E/E/PE system safety life-cycle (related to hardware) and the software safety life-cycle. IEC 61508 follows a risk-based approach to elicit the safety requirements, however, it does not cover mechanisms that might be necessary to prevent unauthorised persons damaging or affecting the functional safety achieved by the system [START_REF]IEC: Functional safety and iec 61508[END_REF].

ISA 62243

ISA 62443 [START_REF]Industrial communication networks network and system security[END_REF] has been developed by the ISA99 and IEC committees to improve the security, availability, integrity and confidentiality of the components or systems used in industrial automation and control. If IEC 61508 standard is considered the reference standard for functional safety, similarly it is expected to happen with ISA 62443 for cyber security. At the moment of writing this paper, not all the parts that will be part of the ISA 62443 series are published and some are still considered draft.

The ISA 62443 clearly identifies three main roles and their compliance responsibilities: the asset owner, the system integrator and the product supplier. The standard only addresses the development of the components and system which are part of an IACS (Industrial Automation and Control System) but does not cover the design, installation or operation of that IACS. The standard distinguishes between the product development process (ISA 62443-4-1) and the system or subsystems integration process performed by the system integrator (ISA 62443-2-4) with an acquisition perspective where components might be created by different suppliers and an integrator needs to ensure security when components work together after the integration. The standard also includes security measures as well as security conformance metrics in other parts.

Standards analysis

We first present the methodology for the standards analysis in Section 4.1 and then in Section 4.2 we discuss and present examples of the identified commonalities and differences.

Methodology

For the two standards analyses, two standardization experts lead the work. One of the experts has a long experience as consultant for standards appliance, specially on process improvement and on domains related to critical infrastructures. The other expert has a strong background on functional safety standards evolution and related research. Both of them have cyber-security knowledge. The work was performed during six weeks. The objective was to identify the commonalities (activities, requirements, roles, evidences, techniques, methods) between both standards.

The work was done in an iterative way. First a study of the standards was performed followed by initial joint sessions for discussing the standards regarding their structure, scope, techniques, management approaches etc. Then, the experts conducted two workshops with industrial partners with a special focus on security aspects. One of the discussions was on how the ISA 62443-4-3 was applied in a specific project. This part of the standard deals with precise technical requirements. The second discussion was on how the process of a company was being defined to apply ISA 62443 in a systematic way. After these two workshops, the interaction continued with focused discussions on the objectives of each practice (both from ISA 62443 Part 4.1 and IEC 61508). As a result of these discussions the mappings between the practices were identified.

Equivalence mappings

The IEC 61508 and ISA 62443 standards have been analyzed in order to search for equivalences between the different standards requirements. Even though we knew that each of the standards has a particular scope as described in Section 3, we assumed a scenario where a functional safety standard is already in place and there is a need to expand the compliance process to comply with cyber-security regulations. This assumption is aligned with the safety-oriented tradition of ICSs where security concerns arrived later. Thus, in our study the identified equivalences are true in one direction and we did not check whether the equivalences are bidirectional. For example, an identified equivalence can be related to the fact that the security requirement is conceptually contained in the safety requirement but, this does not mean that satisfying only the security requirement is enough to satisfy the safety one.

In the beginning, equivalences were sought for the management of cyber-security of a component (ISA 62443-4-1) with the management of the functional safety of an electronic device (IEC 61508). When looking for equivalences, it has been conceptually assumed that when we talk about functional safety we also include cyber-security that affects functional safety.

Table 1 shows two examples of what we called "a full equivalence mapping", that is a requirement identified in both standards which can be considered compatible. Thus, once you provide evidence for compliance for IEC 61508, same evidence can be used to prove compliance with ISA 62443. This is the case of the requirement dealing with identifying people responsible of the tasks. This is requested in A process shall be employed that identifies the organizational roles and personnel responsible for duties for each of the processes required by this standard.

An organisation with responsibility for an E/E/PE safety-related system, or for one or more phases of the overall, E/E/PE system or software safety life cycle, shall appoint one or more persons to take overall responsibility for: the system and for its life cycle phases; coordinating the safety-related activities carried out in those phases; (many other items were not included for space limitations) A general product development/maintenance/support process shall be documented and enforced that is consistent and integrated with commonly accepted product development processes (for example, ISO 9001 certified processes) that include, but are not limited to:

configuration management with change permission controls and audit record logging; product description and requirements definition with requirements traceability; software or hardware design and implementation practices, such as modular design; repeatable testing verification and validation process; review and approval of all development process artifacts; and life-cycle support.

Procedures shall be developed for ensuring prompt follow-up and satisfactory resolution of recommendations relating to E/E/PE safety-related systems, including those arising from:

hazard and risk analysis (see 7. A process shall be employed for defining security training and assessment programs to ensure that personnel assigned to the organizational roles and duties specified in 6.3, SM2 -Identification of responsibilities, have demonstrated security expertise appropriate for those processes.

-6.2.3, -6.2.12, -6.2.13, -6.2.14, -6.2.15, -6.2.16 both standards. Another example is the development process requirement; while the fist example is quite straightforward, in this second requirement example we have taken into account how the development process is applied in both cases. For instance, in the IEC 61508 the configuration management should be applied to the different artefacts requested in the development so the system description, the hardware, the software designs and all the validation and testing results are included. Also, the validation and verification plans should be systematic, repeatable and traceable. The quantity of documentation and evidences required for the development process is a good example of how relevant is to avoid this redundant work. In some cases the equivalence is not a one-to-one mapping of two articles or statements but oneto-many. This is the case of the last example included in Table 1 where the security expertise process is not clearly identified in one of the requirements of IEC 61508 but in six of them.

During the analysis it was revealed that the mentioned "full equivalence map" is not frequent and the common situation was "a partial equivalence map" as the examples presented on Table 2. In this case, the requirements can be found in both standards but they cannot be considered as being completely equal. When looking at these two examples (process verification and third-party suppliers support), we can identify parallel process activities where safety and security could interact and discuss trade-off decisions as each of the requirements is focusing either on safety or on security. It might be considered as a unique process activity and equivalent only if the deployed product development process has considered co-engineering practices. SM8: Third party embedded component security Part 1 -6.2.17 Requirement A process shall be employed to identify and manage the security risks of all externally provided components used within the product. Suppliers providing products or services to an organization having overall responsibility for one or more phases of the overall, E/E/PE system or software safety life-cycles, shall deliver products or services as specified by that organization and shall have an appropriate quality management system.

SM7: Development environmental security

Part 1 -6.2.3 c) Requirement A process that includes procedural and technical controls shall be employed for protecting the integrity of the development environment, production and delivery, including private keys, and the design, implementation and release of a product or product update (patch).

Software configuration management shall [...] maintain accurately and with unique identification all configuration items which are necessary to meet the safety integrity requirements of the E/E/PE safety related system. Configuration items include at least the following: safety analysis and requirements; software specification and design documents; software source code modules; test plans and results; verification documents; pre-existing software elements and packages which are to be incorporated into the E/E/PE safety related systems; all tools and development environment which are used to create or test or carry out any action on the software of the E/E/PE safety related system.

Proposed co-engineering process

As a result of the mapping study presented in Section 4, we propose the definition of a co-engineering process where full equivalence requirements will be taken into account just once, and the partially equivalent ones will be integrated in a unique activity that consider both perspectives. This preliminary co-engineering process is created with the aim to identify common links during the project life-cycle and at which phase it is most interesting to define interaction points for trade-off discussions. Table 3 presents the links of the different life-cycle phases of our proposed life-cycle regarding the related standards' parts and articles. This mapping can be used by practitioners to reference to the specific requirements in each of the phases.

The proposed life-cycle contains a set of phases. We include a brief summary of them below but the details can be found in the referenced standards' parts and articles of Table 3. We designed our coengineering process based upon the well-known V-model [START_REF] Forsberg | The relationship of system engineering to the project cycle[END_REF] inspired by the implicit V-model of IEC 61508 with the idea to increase the comprehension of the concepts (see Figure 1). The dark gray V represents the system life-cycle and the light gray V's represent the software life-cycle (left) and the hardware life-cycle (right) which need to be synchronized. The arrow between the V's represent the validation direction where the final hardware validation is performed once the software has been deployed in the specific hardware.

-Development Management: This activity is performed along the development life-cycle as functional safety management and security management should be coordinated.

-Concept: This is mainly a safety phase where the concept of the system is defined.

-Overall scope definition: The basic blocks and approaches for safety and security should be agreed.

-Overall requirements allocation: In this phase the requirements to fulfil the safety concepts and security concepts should be derived into requirements and allocated to blocks of the future system design.

-System requirement specification: System requirements are agreed from safety and security perspectives.

-Software requirement specification: This phase focused on the software and special attention should be given to reduce security vulnerabilities and eliminate possible security breaches. -Validation planning: In this phase validation should be planned from the functional perspective but also ensuring that hazards and threats are sufficiently mitigated or deleted. -Software architecture: This phase is focused on the architecture definition and both safety and security mechanisms should be allocated to the different components of the architecture. -Software system design: This phase is mainly safety-driven software design.

-Coding: This phase should be security-driven. Although best practices for coding and coding standards should be followed, security practices should be prioritised to avoid vulnerabilities. -Module testing: The different modules of the system should be tested to ensure the fulfilment of the safety and security requirements. -Integration testing (module): In this phase it is important to ensure that no emergent behaviour appears which could affect to safety or security. Both properties should be simultaneously taken into consideration. -Integration testing (components, subsystems and programmable electronics): Similarly to previous phase but considering the deployment of the software into the hardware. -Validation testing: Validation should ensure that safety and security mechanisms and mitigation strategies are working as expected. -Overall installation and commissioning: It is important to take into account in this phase that no security breaches are introduced while ensuring safety. -Operation, maintenance and repair: Each modification done to the system is agreed between safety and security teams so no new vulnerabilities or hazards could be introduced.

Qualitative evaluation and discussion

This section presents the conclusions of three industrial and standardization experts after presenting them our results and the proposed life-cycle. This qualitative discussion is an evaluation on the soundness of our approach. First, they wanted to emphasize the relevance of the objective and motivation presented in Section 1. In the industrial context, where time and money are of utmost importance, combining safety and security in a single process has the potential to save a lot of effort by applying activities that satisfy both, safety and security. Though they are in favor of combined methods, they acknowledge that there are arguments, in particular from practitioners in industry with a traditional mindset, why treating safety and security simultaneously can create difficulties. One of those arguments is that safety experts and security experts mostly "think in a different manner" (i.e., solve their problems according to their own paradigms and, thus, have difficulties to follow the arguments of the experts of the other discipline or are not willing to do this). Another argument is that standardization for the two quality attributes is still widely separate. So these safety and security experts prefer separating the development to address both aspects independently. The problem with the separate approach is, however, that realizing safety functions and security controls independently may cause double work because the required functions are partly the same. Even worse, the solutions resulting from independent safety and security engineering flows are likely to be incompatible due to mutual influences not being treated. The necessary iterations for removing redundancies or correcting the incompatibilities are costly. Therefore the three experts agree that the conducted standards analysis to systematically identify mappings is a correct approach.

Industry shrinks back from investing effort into training, which is doubtlessly necessary when introducing the new co-engineering processes. However, on the long run, the savings from combined processes with less iterations will prevail over the investment in training and the additional costs for purchasing or licensing advanced co-engineering tools. Moreover, rising complexity of future systems should even increase the long-term advantage. It is also important to recognize that co-engineering can be introduced for individual phases and, thus, step-by-step. In the following, we discuss in more detail the effectiveness of the approach.

For the proposed co-engineering, one example for synergistic activities is the fulfillment of integrity requirements. Integrity requirements in the safety context mean that sensor data must not be corrupted before being processed, whereas in the security domain integrity is focused towards data not being willingly manipulated to damage the system's operation. Integrity is a requirements set that is present in both, ISA 62443 and IEC 61508. By only looking at the safety side, a simple CRC (Cyclic Redundancy Code) calculated over payload and attached to data packets is sufficient for most cases. However, security attacks are able to manipulate that data including the CRC. In this case, introducing a MAC (Message Authentication Code) prevents that data from being changed maliciously and ensures that a change of data can be detected, thus enabling the system to take safety measures (for example introducing redundancies in combination with an adjudicator can satisfy safety and security requirements [START_REF] Gashi | Diversity for safety and security in embedded systems[END_REF]). Implementing security measures always seem to be an expensive investment dealing with unknowns and with little visibility of its benefits (only when security measures fail). With a combined co-engineering process, some of these efforts can be saved due to the synergies emerged by common process activities that may result in a partial design solutions that can fulfill many requirements at once (instead of creating one design solution for a single requirement). One example for such a process activity is interference analysis where requirements are analyzed for their interference on each other. This requires that quality attribute requirements (such as safety and security) have to be analyzed together in a common process. The goal of interference analysis is to find requirements that are influenced by, or have commonalities with others. In this case, the analysis of the safety data integrity requirement and the security data requirement result in a single solution that satisfies both requirements (MAC).

Security can hardly be measured and modelled statistically, which prevents safety methods being adopted as they are into the security domain, i.e. FTA or Failure Mode and Effects Analysis (FMEA) have to be refined and extended for security analysis (FMVEA). Thus, established safety analysis methods can not be re-used in a synergistic way. Also, one major problem of safety-related systems arises if software has to be updated. This leads to additional efforts for maintaining the (already gained) certifications. Since modern Industrial Automation Control Systems (IACS) have interfaces for external data exchange and software/systems are never truly secure, updates must be implemented on a regular basis. The proposed combined co-engineering flow eases re-certification efforts, considering the example above, only the functionality of the MAC (which is able to provide data integrity handling and additionally indication of data integrity and authentication) has to be proven again since the MAC satisfies both, the safety integrity requirement and the security integrity requirement.

Overall, modern ICSs all need safety and security measures. There might only be some rare cases (very small systems) where safety can be implemented encapsulated and not being subject to security issues. The proposed life-cycle, which emerges from the analysis of commonalities and differences of the standards, provides guidelines for cost-effective co-engineering in ICSs.

Conclusions

We have shown the feasibility to define a development and co-certification life-cycle for functional safety and security. In this preliminary work, we aligned the phases of the life-cycle identifying common link points. We focused on IEC 61508 and ISA 62443 to propose a co-engineering solution for the Industrial Automation sector. The presented discussions from practitioners help to understand the approach benefits (e.g., higher confidence of a correct design gained by combined analysis in early design stages) and limitations (e.g., initial effort of setting up the new co-engineering flow, higher effort and involvement of experts in early life-cycle stages). As further work, we aim to refine the process with empirical data of its usage (at least a part of it), as well as proposing and integrating advanced co-engineering practices (e.g., safety-security interaction points and interference analysis [START_REF] Pomante | The AQUAS ECSEL Project Aggregated Quality Assurance for Systems: Co-Engineering Inside and Across the Product Life Cycle[END_REF]) in the definition of the process.

Part 4 - 1

 41 Practice 1 Security management. SM4: Security Expertise process Part 1 -Requirements:

Fig. 1 . 1

 11 Fig. 1. The V-shaped proposed co-engineering process

Table 1 .

 1 Examples of equivalent process requirements

	ISA 62443 Cyber security	IEC 61508 Functional safety
	Part 4-1 Practice 1 Security management. SM2:	Part 1 -6.2.1 Requirement
	Identification of responsibilities

Table 2 .

 2 Example of a partially equivalent process requirements

	ISA 62443 Cyber security	IEC 61508 Functional safety
	Part 4-1 Practice 1 Security management.	Part 2 -7.9.2 and Part 3 -7.9.2 Requirements
	SM11: Process verification

A process shall be employed for verifying that, prior to product release, all security-related processes required by this specification have been carried out. P2-7.9.2.1 The verification of the E/E/PE safety-related systems shall be planned concurrently with the development (see 7.4), for each phase of the E/E/PE system safety lifecycle, and shall be documented. P3-7.9.2.1 The verification of software shall be planned (see

[START_REF] Gehlot | An introduction to systems modeling and simulation with colored petri nets[END_REF]

.3) concurrently with the development, for each phase of the software safety lifecycle, and shall be documented.

Acknowledgments:

The research leading to this paper has received funding from the AQUAS project (H2020-ECSEL grant agreement 737475).