Marius Stübs

Tobias Mueller

Kai Bavendiek
email: kai.bavendiek@tuhh.de

Manuel Loesch
email: loesch@fzi.de

Sibylle Schupp
email: schupp@tuhh.de

Hannes Federrath

Provably Privacy-Preserving Distributed Data Aggregation in Smart Grids

Keywords: smart grid security, smart metering, formal model, automated decentralized security

The digitalization of power systems leads to a significant increase of energy consumers and generators with communication capabilities. Using data of such devices allows for a more efficient grid operation, e.g., by improving the balancing of power demand and supply. Fog Computing is a paradigm that enables efficient aggregation and processing of the measurements provided by energy consumers and generators. However, the introduction of these techniques is hindered by missing trust in the data protection, especially for personal-related data such as electric consumption. To resolve this conflict, we propose a privacy-preserving concept for the hierarchical aggregation of distributed data based on additive secret-sharing. To increase the trust towards the system, we model the concept and provide a formal proof of its confidentiality properties. We discuss the attacker models of colluding and non-colluding adversaries on the data flow and show how our scheme mitigates these attacks.

Introduction

The electricity consumption of private households is usually not monitored nor managed in real-time. Traditionally, the power meter aggregates the electricity flow over the year and the aggregated value is only read for billing. However, the continuing increase in the share of renewable energies will require small-scale consumers and producers to actively participate in the demand-supply matching process. In order to gain more insight in the power flows within their grids, grid operators have strong interests in obtaining additional measurements from their customers. Power suppliers also want to allow their customers to benefit from price fluctuations at wholesale electricity markets. The roll-out of smart meters sets the foundation for Smart Grid services such as selective data aggregation and distributed power balancing [START_REF] Van Aubel | Smart metering in the Netherlands: what, how, and why[END_REF]. For the grid operators, an important use case is power system state estimation, where measurements from different sources are aggregated to provide a more precise overview of the respective distribution grid.

In the context of Smart Grid, privacy is a major concern for consumers and prosumers [START_REF] Lodge | Developing GDPR Compliant Apps for the Edge[END_REF]. Depending on the precision of the measurement data, researchers were even able to distinguish between TV channels and thereby identify the movie that was being displayed on a specific type of home TV screens [START_REF] Abeykoon | Real time identification of electrical devices through power consumption pattern detection[END_REF]. This can also reveal other personal information and daily routines, such as how many inhabitants are home and when they leave or return. In Europe, the legislator demands data to be processed in ways that have been designed to respect the privacy of the users ("Privacy by Design and Default") [24, §25]. Whether households agree to participate in local market schemes or choose to demand cloud-based Smart Grid services therefore heavily depends on the perception of these services as serious and trustworthy. One way to increase the plausibility and transparency of cloud-based applications is to incorporate security measures already in an early stage and explain as well as verify these measures. This is where formal modeling and automated proofs come into play to back the security-related claims of the service providers.

Fog Computing in Power Grids

In power grids, the control of consumers and generators can be hierarchically aggregated for provisioning of smart grid services at different grid levels. Smart grid services are, e.g., power adjustments required in the demand-supply matching process. Technically, they are realized by Energy Management Systems (EMSs). The aggregation of control options and data allows for improved decisions as further information such as the grid structure can be considered at higher aggregation levels.

At the lowest aggregation level, Nano Grids can be recognized. Examples for Nano Grids are buildings with an EMS that locally processes data and balances power demand and supply within the building. They also may provide grid services and data to external Smart Grid Service Providers or higher-level aggregators. Nano Grids are always dependent on the connection to the main grid [START_REF] Martin-Martínez | A literature review of Microgrids: A functional layer based classification[END_REF]. At a higher aggregation level Micro Grids can be recognized. Micro Grids are often defined to be self-sufficient in the sense that they can support islanding and that they, in case of emergency, can encapsulate themselves from the higher-level distribution grid. Examples for Micro Grids are districts with an EMS that processes data of multiple buildings for balancing the district's power demand and supply. Control options and data provided by multiple Micro Grids can be aggregated by an EMS at the level of the corresponding distribution grid which is operated by a Distribution System Operator (DSO). Finally, control options and data of multiple distribution grids can be aggregated by an EMS on the level of the transmission grid which is operated by a Transmission System Operator (TSO). EMSs at different aggregation levels allow for different smart The hierarchical structure of Fog Computing is visualized in fig. 1. It fits well to the structure of electric grids, especially when extending the fog node layer to a multi-layer architecture. In this paper we propose to extend previous approaches that aggregate values in Fog nodes and to implement a hierarchical aggregation scheme for sensor data that resembles the structure of the power lines. Aggregated sensor data (such as aggregated power values of multiple districts within a distribution grid) set the foundation for the realization of Smart Grid services that are ensuring grid stability.

Smart Metering & Data Security

In order to facilitate the integration of fluctuating renewable energy resources and to improve the demand-supply matching, the European Union directive 2009/72/EG requires member states to install smart metering infrastructures. As a consequence, it can be expected that a large share of households will deploy EMSs in the near future [START_REF] Van Aubel | Smart metering in the Netherlands: what, how, and why[END_REF]. In this context, Smart Meter Gateways (SMGWs) provide a communication link between Nano Grids such as Smart Buildings and external parties. This communication link offers access to smart meter data and hence sets the foundation for the provisioning of grid services to Smart Grid Service Providers or higher-level aggregators.

The aggregation of data is a security critical function that needs to maintain the users' privacy by keeping the data as confidential as possible. Data about electricity consumption contains information about the user's habits. The more fine-grained the data, the better can the attacker infer details about the subject. Not only has it been shown that it is possible to detect the appliances a consumer is using [START_REF] Kim | Nonintrusive Load Monitoring Based on Advanced Deep Learning and Novel Signature[END_REF], it is also possible to infer what TV program the consumer is watching [START_REF] Greveler | Multimedia content identification through smart meter power usage profiles[END_REF]. The aggregation of data must thus be private in that the parties involved learn as little as possible.

Data Flow Modeling and Automated Proofs

Several approaches exist to show that a system works as intended. Testing is a well known technique, which, however, cannot demonstrate the absence but only the presence of errors. Formal methods on the other hand follow another approach by formalizing the system in way such that certain properties can be proven. One example of a formal method is model checking where a formal model is verified to follow a certain specification. The specification can be expressed using formal properties. To actually verify formal properties of a system, one has to make a formal description of the system. The complexity of real-life systems might be neither beneficial nor necessary for the verification of certain properties. Therefore, a formal model can be a good abstraction of, for instance, the data flow of a system.

Structure of the Paper

Section 2 comprises a comparative review of related work. In section 3 the fundamental design decisions and goals are elaborated. The general data flow privacy validation scheme is described in section 4. The concept of hierarchical data aggregation is elaborated in section 5 and is then evaluated in the security discussion in section 6. We conclude the paper with a short summary in section 7.

Related Work

The proposed concept combines two areas of research, namely Fog Computing based privacy-preserving data aggregation and modeling of privacy-respecting data flows. In this section, we present the work related to each field.

The transition to smart metering and the Smart Grid evoked a lively scientific discussion about the impacts on privacy the transition entails [START_REF] Mrabet | Cyber-security in smart grid: Survey and challenges[END_REF][START_REF] Sultan | Privacy-preserving metering in smart grid for billing, operational metering, and incentive-based schemes: A survey[END_REF][START_REF] Asghar | Smart Meter Data Privacy: A Survey[END_REF]. With the discussion, several approaches, mitigations, and solutions have been proposed ranging from adding noise to the actual usage pattern to make analysis harder [START_REF] Kalogridis | Privacy for Smart Meters: Towards Undetectable Appliance Load Signatures[END_REF] to privacy-preserving data aggregation techniques to be used in Smart Grids [START_REF] Ferrag | A systematic review of data protection and privacy preservation schemes for smart grid communications[END_REF][START_REF] Fan | Privacy-Enhanced Data Aggregation Scheme Against Internal Attackers in Smart Grid[END_REF]. A different approach on private data protection is using batteries to add noise to hide usage pattern [START_REF] Kalogridis | Privacy for Smart Meters: Towards Undetectable Appliance Load Signatures[END_REF]. Over the course of the last decade, the use of secret-sharing, additive or otherwise, for aggregating smart meter readings has been proposed [START_REF] Kursawe | Privacy-Friendly Aggregation for the Smart-Grid[END_REF][START_REF] Danezis | Smart meter aggregation via secret-sharing[END_REF][START_REF] Dimitriou | Secure and scalable aggregation in the smart grid resilient against malicious entities[END_REF], as have other privacy preserving mechanisms [START_REF] Finster | Privacy-Aware Smart Metering: A Survey[END_REF][START_REF] Jawurek | Plug-In Privacy for Smart Metering Billing[END_REF], such as ensuring -differential private aggregation [START_REF] Barthe | Verified Computational Differential Privacy with Applications to Smart Metering[END_REF][START_REF] Eibl | Differential privacy for real smart metering data[END_REF].

Due to the recent interest on Cloud and Fog Computing, several publications (e.g. [START_REF] Green | High performance computing for electric power systems: Applications and trends[END_REF][START_REF] Simmhan | Cloud-Based Software Platform for Big Data Analytics in Smart Grids[END_REF][START_REF] Rehmani | Software Defined Networks-Based Smart Grid Communication: A Comprehensive Survey[END_REF][START_REF] Antoniadis | Security and Privacy for Smart Meters: A Data-Driven Mapping Study[END_REF][START_REF] Diovu | Enhancing the security of a cloud-based smart grid AMI network by leveraging on the features of quantum key distribution[END_REF]) that apply cloud-based solutions to deal with the data explosion challenge in a Smart Grid. Especially when it comes to reducing the communication bandwidth between the smart meter and cloud in cloud computing, regional [START_REF] Stübs | Topology-aware distributed smart grid control using a clustering-based utility maximization approach[END_REF] or hierarchical [START_REF] Stübs | Blockchain-based multi-tier double-auctions for smart energy distribution grids[END_REF] aggregation schemes based on Fog Computing are still not fully researched. Moreover, data security and privacy are also critical issues when sensitive smart meter data is aggregated in only partially trusted environments such as Fog nodes and cloud applications.

In the field of privacy-respecting data flow verification different approaches exist to model data flow and verify privacy properties. A survey paper by Gürses, Troncoso, and Diaz shows different privacy properties and case studies including homomorphic encryption in an automated toll pricing system [START_REF] Gürses | Engineering Privacy by Design[END_REF]. A Smart Grid case study with focus on smart metering is described in [START_REF] Efthymiou | Smart Grid Privacy via Anonymization of Smart Metering Data[END_REF]. It has the same research question but makes a distinction between accountable and private readings. Application-specific approaches range from e-government [START_REF] Hoepman | Crossing Borders: Security and Privacy Issues of the European e-Passport[END_REF][START_REF] De Jonge | Privacy-Friendly Electronic Traffic Pricing via Commits[END_REF] over e-healthcare [START_REF] Li | Data security and privacy in wireless body area networks[END_REF][START_REF] Kart | A Distributed e-Healthcare System Based on the Service Oriented Architecture[END_REF] and medical registers [START_REF] Bavendiek | Automatically Proving Purpose Limitation in Software Architectures[END_REF] to cloud computing [START_REF] Bohli | Security and Privacy-Enhancing Multicloud Architectures[END_REF][START_REF] Wei | Security and privacy for storage and computation in cloud computing[END_REF]. Approaches based on the applied pi-calculus [START_REF] Kremer | Analysis of an electronic voting protocol in the applied pi calculus[END_REF][START_REF] Backes | Automated Verification of Remote Electronic Voting Protocols in the Applied pi-Calculus[END_REF][START_REF] Delaune | Automatic Verification of Privacy Properties in the Applied Pi Calculus[END_REF] are popular for protocoloriented applications with focus on data integrity (often in e-voting systems). ProVerif is one of the few tools in this field, which implements a typed version of the applied pi-calculus. Another category of formal methods are approaches based on type systems [START_REF] Gaboardi | Linear Dependent Types for Differential Privacy[END_REF][START_REF] Maffei | Security and privacy by declarative design[END_REF][START_REF] Fournet | ZQL: A Compiler for Privacy-Preserving Data Processing[END_REF]. These papers usually aim at achieving differential privacy by using typing rules. Other approaches, like this paper, base their modeling on software architectures. Some of these employ modal logics like modified epistemic logic [START_REF] Antignac | Privacy Architectures: Reasoning about Data Minimisation and Integrity[END_REF][START_REF] Antignac | Trust Driven Strategies for Privacy by Design[END_REF][START_REF] Halpern | Complete Axiomatizations for Reasoning about Knowledge and Time[END_REF][START_REF] Bavendiek | Privacy-Preserving Architectures with Probabilistic Guaranties[END_REF] or temporal logic for reasoning about data minimization [START_REF] Barth | Privacy and Contextual Integrity: Framework and Applications[END_REF]. Of the former ones CAPRIV [START_REF] Antignac | Privacy Architectures: Reasoning about Data Minimisation and Integrity[END_REF] and CAPVerDE [START_REF] Bavendiek | Privacy-Preserving Architectures with Probabilistic Guaranties[END_REF] are tools that support the modeling and verification of data minimization properties in software architectures. However, to the best of our knowledge, the tool CAPRIV and its source are not openly accessible. The open source tool CAPVerDE is a very similar tool that operates on software architectures and a modified epistemic logic with focus on data minimization and access control.

Models and Design Goals

We consider the electricity supply and demand in a Micro Grid. We assume that the Micro Grid is properly designed such that a portion of the electricity demand related to basic living usage (e.g., lighting) from the residents, termed basic usage, can be guaranteed by the minimum capacity of the Micro Grid. There is randomness in both electricity supply (due to, e.g., weather change) and demand (e.g., entertainment usage in weekends). To cope with the randomness, the Micro Grid works in the grid-connected mode and is equipped with energy storage systems (ESSs), such as an electrochemical battery, superconducting magnetic energy storage, flywheel energy storage, etc. The ESSs store excess electricity for future use.

Modeling of Privacy-Respecting Data Flows

For this paper we make use of a logic and tool called CAPVerDE [START_REF] Bavendiek | Privacy-Preserving Architectures with Probabilistic Guaranties[END_REF]. The logic consists of a formal description language for software architectures, a data flow property language, and a rule-based verification system. With these building blocks we can describe the data flow of a software system, express properties like "Actor A should have access to Data d", and verify whether the specified model satisfies said properties. The tool aids us by automatically solving the inference-rule based satisfaction problem.

The process of using this technique is to first identify the relevant data flow, actors, and data dependencies of the system. Then use a formal description language to describe the data flow. Setting up a formal model is already a big step because one decides which details are important and where to abstract. Once the system is modeled, one has to formalize the design goals for the system. This is done using a logic that allows for certain statements about the model. The verification rules allow for checking whether such a property holds for the provided model. If the verification returns that the model satisfies all properties, the model meets the design goals.

For the specific use case of this paper, we want to look at privacy design goals. These are usually expressed by properties that state that an 'untrustworthy' actor should not have access to data that is considered personal or worth of protection. For a Smart Grid architecture this means, for instance, protecting measurements of smart meters against curious intermediate aggregators. To verify such a property, we have to model the data flow from smart meters to service providers, including the implicit data flow, i.e., dependencies between data.

In order to achieve the goals that we have modeled, we make use of an additive secret-sharing-based multi-party computation as suggested in the past [START_REF] Kursawe | Privacy-Friendly Aggregation for the Smart-Grid[END_REF][START_REF] Danezis | Smart meter aggregation via secret-sharing[END_REF][START_REF] Dimitriou | Secure and scalable aggregation in the smart grid resilient against malicious entities[END_REF]. With that technique, we can achieve the private aggregation of values and prevent disclosure of single measurements. However, we are also interested in hiding the aggregated values from unauthorized intermediary nodes, so we additionally share a secret with the intended receiver of the aggregated result. With this approach we can have the values aggregated by any number of nodes in the Fog, but only the intended recipient can remove the blinding and obtain the actual result.

Network Model

The roles of the proposed encryption scheme are depicted in fig. 2. Actors in the distribution grid are a) Smart Meter Gateways (SMGWs) that collect and disseminate measurement data, b) Intelligent Energy Devices (IEDs) that are controllable via a interface and consume electricity or in case of distributed energy resources inject power into the distribution grid, c) data aggregators that operate on Fog nodes to collect, aggregate and redirect measurement data, and d) Smart Grid Service Providers who receive the aggregated measurements. The e) key management authority enables confidential end-to-end connections between the other actors. a) Smart Meter Gateway (SMGW): The measurements from the IEDs are encrypted and disseminated by the SMGW. The data is encrypted using secretsharing, so that is can be aggregated without decryption. b) Intelligent Energy Devices (IED): For our scheme, the readings from the The KMA provides a central trust infrastructure for the data flow. To establish a trusted link between the SMGWs, the DAs and the Smart Grid Service Providers, the KMA distributes secrets for each hierarchical aggregation layer.

Threat Model

The presented scheme is designed to provide the confidentiality of data with respect to hierarchical layers in the aggregation architecture. Following the formal modeling approach, we can distinguish between two types of adversaries: Colluding and non-colluding attackers.

Non-Colluding Attackers For the non-colluding attacker, we consider the honest-but-curious attacker model [START_REF] Derryberry | Compiling an Honest but Curiuos Protocol (5 2003)[END_REF], where the respective node complies to the protocol in such that they are only collecting and analyzing the data that they lawfully receive. This type of adversary does not forget any data once received. They try to computationally and statistically break any obfuscation on the received data to break confidentiality where possible. The attacker model is called honest-but-curious because they do not actively request data if not obliged to by the protocol.

For the evaluation, we consider all individual actors -that is edge nodes, Fog nodes and cloud services -as potential attackers and formally model the requirement that the proposed scheme has to protect the personal data from unauthorized access.

Colluding Attackers In extension of the previous model, we also formally describe that nodes communicate their knowledge among each other to collaborate on breaking the confidentiality. The colluding attacker actively shares information with other nodes and combines this acquired knowledge. Manipulation of data, selective forwarding and forging attacks are not considered because the focus of this paper is on privacy and data flow. However, we consider out-of-band forwarding (via gossiping) of confidential communication to colluding nodes. We assume an attack to be successful, if two (or more) colluding nodes can derive knowledge that exceeds the knowledge from both (all) respective nodes individually.

Formal Privacy Modeling Methodology

This section presents the formal method to model and analyze the Smart Grid system under investigation and its desired privacy properties.

Data Flow Description Language

Table 1 shows a relevant excerpt of the syntax of the architecture description language of CAPVerDE. An architecture A consists of so-called relations R that model explicit and implicit data flows between the actors (called components C i ). The relation Has i (X) is the entry point, i.e., it models how data enters the system. The relations can be read as "Component C i has access to a variable X" for example via a sensor. Rec i,j ({X}) is the relation that models the passing of data from one actor to another. It can be read as "Component C i receives the set of variable {X} from component C j ." Comp i (X = T ) is the relation that lets actors derive new data from already known data. The natural language description of this relation is "The Component C i can compute variable X from some term T ", where term T consists of already known variables. The language also allows for implicit data flow, that is, calculations that are not intended in the system but possible. Relation Dep i (Y, {X 1 , ..., X n }) states that "Component C i has the computational power to derive variable Y from the set of variables X i ." For example if it is known that a = b + c, all actors in the system will have the relation Dep i (a, {b, c}). The absence of such a relation denotes the fact that a calculation is not feasible. The addition is not directly reversible, i.e., one cannot derive the summands from the sum alone, therefore the relation Dep i (b, {a}) will not exist. It should be noted that, although this type of relation deals with implicit data flow, the relations have to be modeled explicitly, that is, for each equation one has to specify whether it should be reversible. Also, all dependencies of a variable have to be made explicit in this way, as well.

Table 1. Syntax of architecture language

A ::= {R} R ::= Hasi(X) | Reci,j({X}) | Comp i (X = T ) | Dep i (Y, { X 1 , ..., X n })
A formal software system is defined as a set of relations. To make the description more readable for humans, we will provide a graphical representation of the system. The graphical form shows the components as boxes containing the corresponding relations and arrows between component boxes show the intercomponent relations.

The verification process of CAPVerDE works as follows: The modeling of the architecture has to be done by the user. One follows the syntax of the language to describe the data flow. Also, the dependencies between the data has to be expressed. CAPVerDE provides a GUI to aid the modeling process. The tool then automatically checks the architecture for consistency, i.e., whether the data flow is possible. For instance, receiving data from a component that does not possess the data would be inconsistent. If the architecture is consistent, the privacy goals have to be expressed as properties in the provided language. This can also be done using the GUI. The verification of selected properties is done automatically by the tool. It returns "property holds" or "property does not hold" for the given architecture. One can view the verification trace of a property to inspect the applied rules necessary for the verification.

Privacy Goal Formalization

The only type of property we consider for this paper is the one regarding the access of data. A property φ = HasAcc i (X) states that component C i is able to access variable X. An architecture satisfies this property (denoted by A φ) iff the component can use its explicit and implicit data flow to derive X. Also negation of properties is possible. A negated property ¬φ holds iff the property φ does not hold.

Table 2 shows the rules of inference that are used by the automatic verification process. The first three rules (H1-H3) regard the explicit data-flow and

H1 Hasi(X) ∈ A A HasAcci(X) H2 Reci,j(E) ∈ A X ∈ {E} A HasAcci(X) H3 Comp i (X = T ) ∈ A A HasAcci(X) H4 Dep i (X, {X 1 , ..., X n }) ∀l ∈ [1, n], A HasAcci(X l ) A HasAcci(X)
hold if the component somehow processes the data. The fourth rule (H4) is based on the dependence relation and hence on the computational ability of the component to derive the data.

Privacy-Preserving Aggregation

This section presents the integrated concept for managing the aggregation of data in Fog Computing based Smart Grid networks. We first introduce the prerequisites for our scheme, namely a central trusted party which manages the participants' keys. Then we describe the enrollment phase of devices in the network, before we discuss the actual steps the nodes in the network have to perform. We present two protocols: One for disguising the individual readings of a smart meter and another for blinding even the aggregate value to unauthorized aggregators.

Our scheme follows the privacy enhancing architecture of smart metering proposed by Molina-Markham et al [START_REF] Molina-Markham | Private memoirs of a smart meter[END_REF]. More precisely, we use zero knowledge protocols for communication assume secure communication between the utility service provider and the nodes in the Smart Grid topology. Additionally, we make use of a secret-sharing-based approach as has been proposed multiple times in the past [START_REF] Kursawe | Privacy-Friendly Aggregation for the Smart-Grid[END_REF][START_REF] Danezis | Smart meter aggregation via secret-sharing[END_REF][START_REF] Dimitriou | Secure and scalable aggregation in the smart grid resilient against malicious entities[END_REF].

Our scheme consists of two procedures. The first procedure, explained in section 5.1, is to hide individual readings from the aggregators. The second procedure is to hide the actual aggregation result from the aggregators themselves, such that only the final recipients, the respective Smart Grid Service Providers, can decrypt the value. This second procedure is described in section 5.2

Procedure 1 -Hiding individual readings from the aggregators

We refer to fig. 2 for a description of the actors. Let each of the n home-level IEDs be IED n . Each IED j creates secret-shares of its reading m j : m j = n i=0 m j i and disseminates each share m j i to IED i where i = j. Each IED j then accumulates all the shares they have received x j = n i=0 m j i and sends the result on to the data aggregator DA. The DA can calculate n j=0 x j . This is equal to the accumulated readings:

n j=0 m j .
This procedure is elegant, because it works locally without a centralized key management authority. It protects the confidentiality of all participating child nodes resp. their contribution. On the other hand, it does not protect the confidentiality of the sum since the aggregator learns the aggregated value. Also, if all child nodes but one collude, they can unveil the measurements of the remaining node. Therefore, we improve the scheme in order to hide both the sum and all the participating nodes.

Procedure 2 -Hiding the sum from the aggregators

We extend the previous protocol by further blinding the values with a secretshare from the intended receiver of the aggregated values. That is, we keep the structure of the participants, but rely on a central key management authority (cf. section 3.2) to generate and distribute keys.

First, the key management authority (KMA) generates a key k and generates shares for each of the n IEDs: k = n j=0 k j . Those shares are then sent to each IED j , which then re-shares that key: k j = n i=0 k i j where i = j. Each IED j then blinds the shares of its reading (m j i ) with the re-shared key, before sending it on to the other IEDs: m j i + k j i . Each IED j accumulates their received shares:

x j = n i=0 m j i + k j i
and sends the result on to the aggregator DA, who then calculates the sum as n j=0 x j . This is equal to the accumulated readings with the blinding: n j=0 m j +k j = k+ n j=0 m j . Now, only the holder of k can remove the blinding and thus obtain the actual aggregate.

Proof of Privacy-Preserving Data Dissemination

In this section we describe the actual proof of privacy preservingness. The highlevel security threats as described in section 3.3 are examined and it is discussed how data confidentiality is achieved.

Procedure 1

Figure 3 shows the architecture of Procedure 1 (cf. section 5.1). Three aggregators for the street-level and one for the district-level are considered. Also for the sake of readability only the rightmost components of each level report to a service provider. The variables m a , m b , m c are measurements of the Intelligent Energy Devices and value0 a , value0 b , value0 c are the aggregated measurements on the smart meter level. value1 a , value1 b , value1 c are the aggregated measurements of the street level and value2 is the aggregated measurement of the district level. All other variables are intermediate results and shared secrets between the components of one level. c0 1 ad c0 2 , for instance, are the shared secrets of SM GW c , which are distributed between SM GW a and SM GW b .

The depicted graph describes the explicit data flow of the architecture. As mentioned above, the implicit data flow is described via the dependence relations. For this architecture we assume that all components have the computational power to build sums and differences. Therefore, all components have the same set of dependence relations. All components can perform the explicit calculations done by other components on the system as well as all corresponding equivalent calculations. This is based on the assumption that the protocol is known. For example the set of dependence relations of SMGW A includes the dependence relations dep SM GW a (c0 2 , {value0 c , c0 1 }) and also the derived dep SM GW a (value0 c , {c0 1 , c0 2 }) and dep SM GW a (c0 1 , {value0 c , c0 2 }).

Non-Colluding Attackers

In this setting we want to verify that no single actor can access measurements that are not intended for them. To to do so we can verify that components on the same level cannot access each others' values and components on a higher level cannot read individual readings from the lower level. An example of a property of the first type in the syntax of CAPVerDE is φ 1n = ¬HasAcc SM GW a (value0 c ). The SMGW A should not be able to read the measurement of SMGW C (same level). An example of a property of the second type is φ 2n = ¬HasAcc AG1c (value1 c ). The Aggregator C (street level) should not be able to read the summed measurements of the SMGWs. We refer to appendix A for the discussion of the verification trace.

Colluding Attackers Here we consider colluding attackers in the model, that is two components sharing their knowledge to obtain data that both cannot access individually. To model such behavior we have to manually add "attackrelations" to the architecture. To stay with the existing example, we let the SMGWs A and B collude to access the readings of C. Therefore, we add the relation Rec SM GW a,SM GW b ({c0 2 }) to the architecture to model that B shares its secret with A.

The property φ 1c = ¬HasAcc SM GW a (value0 c ) fails for the architecture with the additional collusion-relation added. The verification trace looks identical to the one for the non-colluding attacker (depicted in fig. 5) up to line 37. Rule H2 for property HasAcc SM GW a (c0 2 ) now applies, which makes the dependence relation dep SM GW a (value0 c , {c0 1 , c0 2 }) possible. Hence, the property HasAcc SM GW a (value0 c ) holds and the negation consequently does not. Consequently, the privacy goal is not achieved in this model if one considers colluding attackers.

Procedure 2

Figure 4 shows the architecture of Procedure 2 (cf. section 5.2). The same simplifications as before apply. As described above, in this version a Key Management Authority distributes the keys. Due to the introduced encryption we now have more variables. The variables k0 a , k0 b , k0 c denote the keys for the smart meters and k0 the corresponding key of the service provider of the next level.

Accordingly, k1 a , k1 b , k1 c are the keys for the street level aggregators and k1 is given to the service provider on the district level. In addition, the variables ac1 a , ac1 b , ac1 c , ac2 are introduced for intermediate results.

In this procedure our privacy goal is to guarantee that the aggregators cannot read the aggregated values which are intended for the respective service providers.

Non-Colluding Attackers

The property φ 2n = ¬HasAcc AG1c (value1 c ) can be verified and produces a long trace. Essentially, the component AG1c has no explicit data flow relations to obtain value1 c and thus can only access it based on dependence relations. The only possible one is dep AG1c (value1 c , {ac1 c , k0}). Rule H3 applies for the aggregated encrypted sum ac1 c . For the service provider key k0, however, the component AG1c can only use implicit data flow. Only two dependence relations exist, of which the first is a circle as dep AG1c (k0, {ac1 c , value1 c }) is derived from the same equation as the previously used one. The second one is dep AG1c (k0, {k0 c , k0 b , k0 a }), which needs the three keys from the SMGWs. k0 c cannot be obtained via explicit data flow and thus AG1c has to use dependence relations, again. Apart from the circle, only dep AG1c (k0 c , {k0 c 1, k0 c 2}) is applicable. The only non-circular dependence relation for

k0 c 1 is dep AG1c (k0 c 1, {c0 1 , k0 c 2}). dep AG1c (c0 1 , {value0 c , c0 1
}) is the only applicable implicit relation. However, we have shown above that HasAcc SM GW a (value0 c ) does not hold for the architecture of Procedure 1. The same holds for this architecture as no non-circular dependence relation exists that allows AG1c to obtain value0 c ). Thus, the verification of the property HasAcc AG1c (value1 c ) fails and the negation can be successfully verified.

Colluding Attackers Here we consider an additional collusion relation of the form Rec SM GW a,SM GW b ({ck0 2 }). The property φ 1c = ¬HasAcc SM GW a (value0 c ) is again to be verified. We omit the verification steps that are identical to the ones of Procedure 1. We start at dependence relation dep SM GW a (value0 c , {c0 1 , c0 2 }). c0 1 can only be obtained via dep SM GW a (c0 1 , {ck0 1 , k0 c 1}). While Rule H2 applies for ck0 1 , k0 c 1 can only be derived via accessing c0 1 , which we are currently trying to obtain, or k0 c . The latter cannot be obtained by SM GW a without having k0 (circle again). Therefore, the verification of HasAcc SM GW a (value0 c ) fails and we successfully verify φ 1c .

The proposed architecture achieves data confidentiality in the presence of honest-but-curious data aggregators as well as non-colluding Nano-Grid neighbors. It is not possible to deduce individual supply or demand from the aggregated values.

Conclusion

The proposed scheme describes an architecture for data aggregation and dissemination in Smart Grids supported by distributed Fog nodes. The architecture is topology-aware in such that a grid-based hierarchy is built. A comprehensive modeling approach has been selected and demonstrated, which aims to enable Smart Grid services to transparently explain and proof their confidentiality. On the infrastructural layer, the scheme describes machine-to-machine communication which assures privacy-preserving aggregation of smart meter readings and dissemination between the SMGWs and possible Smart Grid Service Providers. We model the proposed scheme and show that its integrity and privacy-properties can be validated by CAPVerDE, an automated prover.

Future Work

In this paper, we have shown that general privacy properties of data flows in the context of Smart Grids can be modeled and proven using automated solvers. As a next step, we elaborate a systematization of aspects that need to be protected from unauthorized access and curious actors. We want to especially look into the protection of meta data and how communication relationships between the different actors can be obfuscated. Examples for such meta data could be the origin of readings, the recipient Smart Grid Service Providers, but also the type of reading (topic) and frequency of transmission.

The proposed scheme is efficient regarding the computational effort, but can still be optimized with respect to the messaging overhead. Anonymity and privacy-respecting data flows always have to evaluate between the price in efficiency and the return in privacy. For future work, we will evaluate other hierarchical aggregation schemes to conduct a systematic network load comparison.

Regarding the key management, for privacy reasons it might be beneficial to investigate distributed schemes instead of using a centralized key management authority. This challenging research field is another follow-up activity in the journey towards the development of truly privacy-respecting Smart Grids. Rule H3 applies and consequently HasAcc AG1c (value1 c ) holds. Therefore, the negation does not hold and our data protection goal of protecting the aggregated measures against the next level aggregators is not satisfied in this architecture.

Fig. 1 .

 1 Fig. 1. The edge computing schematic.

Fig. 2 .

 2 Fig. 2. The lowest layer includes the consumers and generators. In the higher layers, only aggregators reside. Aggregators can not decrypt the aggregated values. Possible data flow is indicated by the arrows. Only the Smart Grid Service Providers have access to the aggregated data. Authorization is based on the aggregation level.

Fig. 5 .

 5 Fig. 5. Verification trace explaining the rule application of CAPVerDE.

Table 2 .

 2 Rules of inference for the architecture logic in CAPVerDE

IED a

Has IEDa (m a )

DA2

IED c

Has IEDc (m c )

SMG c

Has SMGc (c0 1 )

IED a

Has IEDa (m a ) KMA Has KMA (k0 a , k0 b , k0 c ) Has KMA (k1 a , k1 b , k1 c ) Comp KMA (k0 = k0 a + k0 b + k0 c ) Comp KMA (k1 = 3*k0 + k1 a + k1 b + k1 c )

Rec SMGc,SMGb ({bk0 2 }) Rec SMGb,SMGc ({ck0 2 })

SMG a

Has SMGa (a0 1 , k0 a1 , value0 a ) Comp SMGa (a0 2 = value0 a -a0 1 ) Comp SMGa (k0 a2 = k0 a -k0 a1 ) Comp SMGa (ak0 1 = a0 1 + k0 a1 ) Comp SMGa (ak0 2 = a0 2 + k0 a2 ) Comp SMGa (x0 a = bk0 1 + ck0 1 )

DA1 a

Has DA1a (a1 1 , k1 a1 , ac1 a ) Comp DA1a (a1 2 = ac1 a -a1 1 ) Comp DA1a (k1 a2 = k1 a -k1 a1 ) Comp DA1a (ak1 1 = a1 1 + k1 a1 ) Comp DA1a (ak1 2 = a1 2 + k1 a2 ) Comp DA1a (x1 a = bk1 1 + ck1 1 )


Rudolf Mayer
email: rmayer@sba-research.org

Markus Hittmeir
email: mhittmeir@sba-research.org

Andreas Ekelhart
email: aekelhart@sba-research.org

Privacy-preserving Anomaly Detection using Synthetic Data

Keywords: Synthetic Data, Anomaly Detection, Machine Learning

With ever increasing capacity for collecting, storing, and processing of data, there is also a high demand for intelligent knowledge discovery and data analysis methods. While there have been impressive advances in machine learning and similar domains in recent years, this also gives rise to concerns regarding the protection of personal and otherwise sensitive data, especially if it is to be analysed by third parties, e.g. in collaborative settings, where it shall be exchanged for the benefit of training more powerful models. One scenario is anomaly detection, which aims at identifying rare items, events or observations, differing from the majority of the data. Such anomalous items, also referred to as outliers, often correspond to problematic cases, e.g. bank fraud, rare medical diseases, or intrusions , e.g. attacks on IT systems. Besides anonymisation, which becomes difficult to achieve especially with high dimensional data, one approach for privacy-preserving data mining lies in the usage of synthetic data. Synthetic data comes with the promise of protecting the users' data and producing analysis results close to those achieved by using real data. However, since most synthetisation methods aim at preserving rather global properties and not characteristics of individual records to protect sensitive data, this form of data might be inadequate due to a lack of realistic outliers. In this paper, we therefore analyse a number of different approaches for creating synthetic data. We study the utility of the created datasets for anomaly detection in supervised, semi-supervised and unsupervised settings, and compare it to the baseline of the original data.

Introduction

The demand for and practice of data sharing and exchange between different data collecting parties is increasing, often because different data sets complement each other, or because the processing and analysis of data is outsourced. Many interesting knowledge discovery tasks are dependent on large, high quality amounts of data being available. However, when data is sensitive, e.g. when it concerns individuals or is business related, there are certain regulatory and other barriers for data sharing. Still, collaborative analysis of data can be very beneficial, e.g. when learning from misuse patterns that other parties have been exposed to, such as network intrusions or fraud. Thus, means to enable such data exchange are required. In some cases, anonymisation techniques such as k-anonymity [START_REF] Sweeney | K-anonymity: A model for protecting privacy[END_REF] or Differential Privacy [START_REF] Dwork | Differential privacy[END_REF] can be successfully employed. For a detailed overview on privacy-preserving data publishing methods, see [START_REF] Chen | Privacy-preserving data publishing[END_REF]. However, k-anonymity has been shown to be still prone to linkage attacks when adversaries have background knowledge and access to other data sources. Differential privacy, when applied to the model or the output of the model, on the other hand is not applicable for all types of analysis techniques. Both approaches distort the data records to some extend, which, due to the information loss, has potentially negative effects on the utility of the data and the models subsequently trained upon.

Synthetic data is generally considered as data obtained not from direct measurement. In the context of data analysis efforts, it is often considered to be data generated (or synthesised) from a real dataset that, e.g. for privacy considerations, can not be shared. Its aim is to provide a dataset containing records that are similar to the original ones, and that preserve the high-level relationships within the data, without actually disclosing real, single data points. While preserving global properties is often possible, there is generally still a certain loss in data utility, albeit for many settings, this might be acceptable -and potentially be of higher utility than anonymisation techniques.

Approaches can be distinguished on how the model for generating data is obtained. If the original data is not directly used, synthetic data can be generated based on rules and constraints describing the characteristics, requiring an indepth knowledge on the original data and expert knowledge on the domain. Approaches that can access the original data can learn models from that data, and use these models to generate new data. These methods have the advantage of being applicable to virtually any kind of domain and being scalable to large amounts of attributes. They can also capture correlations that are not easily understood by human experts and for which rule-based methods would fail. The complexity of the employed models for representing the original data is of varying degree, which has implications on how well the synthetic data resembles the original. In this paper, we focus on approaches that utilise the original data.

Another differentiation is whether the dataset is partially or fully synthetic. The former means that only a subset of the available attributes (or sometimes a subset of the samples), especially those identified to be sensitive, are replaced by synthetically generated ones. For partially synthetic data, data utility is expected to be higher, even though disclosure risks may be higher as well.

Anomaly detection methods typically utilise general outlier detection techniques, but differ when there is a need to distinguish harmless noise from malicious intentions, actions or attacks. Many sub-types of anomaly detection can be distinguished, depending on the type of data, the required output, and whether labels that distinguish between normal and abnormal cases are available [START_REF] Chandola | Anomaly detection: A survey[END_REF]. The latter decides whether the detection is to be performed in a supervised, unsupervised or semi-supervised setting. The general settings is that the anomaly (or outlier, minority) type of data points are very few compared to the "normal" data points -sometimes well below 1% of the whole dataset.

Anomaly detection can be considered a rather peculiar use case for employing synthetic data, as this approach tries to specifically preserve the global characteristics, but not of single data points -which could result in information leakage. Anomaly detection techniques, however, try to specifically find the unusual data points among a large amount, and the synthetically created dataset is not expected to produce outlier data points similar to the original ones, as they do not represent global properties. In this paper, we thus evaluate the utility of synthetic data for this task. We consider multiple scenarios, utilising three different approaches for generating synthetic data, and address supervised, semi-supervised and unsupervised methods for anomaly detection based on the generated data. We compare the results to a baseline of methods run on the original dataset.

Section 2 describes related work in synthetic data and anomaly detection. Section 3 describes the setup for our experiments, before we present our evaluation in Section 4. Finally, we present conclusions and future work in Section 5.

Related Work

One of the earliest applications of generic synthetic data generation in a data mining context is described by Rubin in [START_REF] Rubin | Multiple Imputation for Nonresponse in Surveys[END_REF], where multiple imputation is used to synthetically generate certain columns of datasets. This is thus a setting for partially synthetic data. Specific applications include e.g. the generation of timeseries or log data, as they would be encountered for intrusion-detection scenarios [START_REF] Lundin | A Synthetic Fraud Data Generation Methodology[END_REF][START_REF] Barse | Synthesizing test data for fraud detection systems[END_REF]. More recently, several efforts have been made to generate synthetic images, e.g. in the medical domain for MR images [START_REF] Han | GAN-based synthetic brain MR image generation[END_REF].

In general, most approaches to synthetic data generation that are based on existing data, and not (only) on rules, consist of the following steps:

-Learning a representation (model) of the original data, with a certain statistical approach (such as estimating a probability density function) -Synthesising the data, i.e. generating new data based on the model -Optionally, a module to ensure privacy of the generated, synthetic data samples, e.g. by means of applying Differential Privacy A major difference in existing approaches lies in the complexity of the learned model from the data. This can range from relative simple models learning (independent) probability density functions for each attribute, and more sophisticated models that preserve the correlation between attributes e.g. via co-variances, to models that capture very complex correlations e.g. via auto-encoders, or approaches utilising generative adversarial networks (GANs) [START_REF] Goodfellow | Generative Adversarial Nets[END_REF]. In the synthetisation step, some approaches apply further methods for data protection, e.g. by applying Differential Privacy before publishing the synthetic dataset.

While synthetic data has shown to be of utility for multiple data analysis tasks including regression or classification (see e.g. [START_REF] Hittmeir | On the Utility of Synthetic Data: An Empirical Evaluation on Machine Learning Tasks[END_REF]), another major aspect is whether it is actually able to fulfil the promises towards preserving the privacy of the individuals in the original dataset. Regarding disclosure risks for synthetic data, usually a distinction is made between fully and partially synthetic data. Reiter and Mitra [START_REF] Reiter | Estimating risks of identification disclosure in partially synthetic data[END_REF] proposed identification disclosure risk estimations for partially synthetic data. They also note that the notion of identification disclosure is not meaningful for fully synthetic data, since fully synthetic records do not relate to original records in terms of a 1-to-1 correspondence. From an intruder's perspective, the approach to gain information by linking certain synthetic records to individuals is thus not promising, as such links generally do not exist, and matches that have been established with real individuals are only spurious, and not real. Other disclosure risks exist, e.g. attribute disclosure, where it is assumed that the intruder knows the values of certain attributes of their victim and wants to learn the value of some sensitive attribute (called the target variable). For a detailed discussion of the latter, see e.g. [START_REF] Hittmeir | A Baseline for Attribute Disclosure Risk in Synthetic Data[END_REF].

For our evaluation in this paper, we employ the following three data synthesizers: The Synthetic Data Vault (DV) [START_REF] Patki | The synthetic data vault[END_REF] builds a model based on estimates for the distributions of each column. In order to preserve the correlation between attributes, the synthesizer applies a multivariate version of the Gaussian copula and computes the covariance matrix. The DataSynthesizer (DS) [START_REF] Ping | Datasynthesizer: Privacy-preserving synthetic datasets[END_REF] provides, among simpler techniques, the 'correlated attribute mode', in which dependencies between attributes are represented by a Bayesian network, learned on the original data. The DataSynthesizer further uses the framework of Differential Privacy, and offers the possibility to inject noise in the model and thus the generated data, by a parameter controlling the magnitude. Finally, synthpop (SP) [START_REF] Nowok | synthpop: Bespoke creation of synthetic data in r[END_REF] uses as the default method a CART (Classification and Regression Trees) algorithm. The user is able to specify a large number of parameters and may apply a built-in function for disclosure control to the resulting synthetic dataset.

When preparing a dataset for publishing via anonymisation, synthetisation or similar approaches, some information at the level of individual records is invariably removed [START_REF] Chen | Privacy-preserving data publishing[END_REF] in order to achieve the desired level of protecting the sensitive information that cannot be shared. Utility evaluation of such datasets can generally be done by two methods. One is to measure certain properties on the sanitised dataset and compare it to the original dataset, such as mean or standard deviation or the data distributions. This evaluation has the advantage of being independent of the final task being carried out on the dataset, but is also generally more difficult to put into an application context. Another approach is to measure the utility on a task, e.g. a supervised classification task, or outlier detection. In this approach, the metric measures the differences in effectiveness of the models on the original dataset compared to the sanitised one. In this paper, we focus on the second approach, as it is more suited for anomaly detection.

Anomaly detection typically utilises general outlier detection techniques [START_REF] Agyemang | A Comprehensive Survey of Numeric and Symbolic Outlier Mining Techniques[END_REF][START_REF] Zimek | Ensembles for unsupervised outlier detection: challenges and research questions a position paper[END_REF], but differ when there is a need to distinguish harmless noise, fluctuations or various forms of novelty, from malicious intentions, actions or attacks. There are many forms of anomaly detection depending on the nature of the input data, the form of output and the presence or absence of data labels [START_REF] Chandola | Anomaly detection: A survey[END_REF]. Point anomalies denote individual data instances that are different from the normal data points. In a contextual anomaly, data instances are anomalous within a specific context, such as temporal or spatial. Collective anomalies identify a collection of instances as anomalous with respect to the entire dataset. In this paper, we focus primarily on point anomalies. Techniques for this type of anomalies are often the building blocks for contextual and collective anomalies, thus some of our conclusions transfer also to these scenarios.

Anomaly detection is used in a wide variety of contexts, such as fraud detection [START_REF] Abdallah | Fraud detection system: A survey[END_REF], intrusion detection [START_REF] Zuech | Intrusion detection and Big Heterogeneous Data: a Survey[END_REF], video surveillance systems, or forensic investigations in general. Specific techniques include unsupervised learning to find structures or patterns in data in the absence of any labels. Recent techniques include Generative Adversarial Networks (GANs), such as f-AnoGAN [START_REF] Schlegl | f-AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks[END_REF]. When labels are present only for the "normal" data points, a semi-supervised methods such as a one-class support vector machine (SVM) [START_REF] Smola | A tutorial on support vector regression[END_REF] or autoencoders are employed. Supervised methods require labels for both the normal and the anomaly cases, and can be addressed by utilising standard classification techniques, such as Logistic Regression, Random Forests, or Support Vector Machines.

Experiment Setup

In our experiments, we utilise the "credit card fraud" dataset by the ML group at ULB, which was provided publicly on Kaggle1 . It contains data based on real samples from transactions made by credit cards in September 2013 by European cardholders, gathered over two days. Out of the total of 284,807 transactions, only 492 are identified as fraudulent. The dataset is thus highly unbalanced, as frauds account for only 0.172% of all transactions. The dataset contains 30 input variables. Due to confidentiality reasons, most of the original features have been transformed with Principal Component Analysis (PCA). A projection of the dataset to two dimensions can be seen in Figure 1, created using t-Distributed Stochastic Neighbour Embedding (t-SNE) [START_REF] Maaten | Visualizing data using t-SNE[END_REF]. It can be observed that some of the anomaly data points are separable, especially the ones seen towards the the top-right of the visualisation. However, also a large number of points seem to be mixed with normal (legit) records, and thus likely difficult to separate.

For the generation of synthetic data, we used the Synthetic Data Vault, the DataSynthesizer and the synthpop package (cf. Section 2). Our primary goal is an unbiased evaluation, and not an optimisation towards a specific synthesizer or target evaluation, but we performed a limited parameter search starting with the standard settings of each synthesizer. We performed the following procedure in order to synthesise and prepare the data for the utility evaluation.

1. We deleted columns in the context of standard feature cleaning, e.g. purely identifying attributes like the 'Time' column in the original dataset. 2. We performed a holdout method, i.e. we randomly split the data into training and test data, such that the size of the latter is 20% of the original table. With this procedure, we obtain datasets that we utilise for training a supervised anomaly detection system (e.g. a classification algorithm). Moreover, we want to separately investigate the behaviour of semi-supervised and unsupervised methods as well. We simulate semi-supervised data by repeating the above outlined procedure for just the data samples that are present in the training set (i.e. after splitting the data), and are labelled as "normal" cases. Finally, for the unsupervised approach, we simply remove the class label before synthesising the data. We thus obtain another five datasets for each of these cases, as for the supervised case, and therefore utilise in total 15 different synthetic training sets for each of the splits generated in Step 2. In addition, we also utilise the test dataset, which is used to estimate the results of the machine learning models on all the training sets. This dataset is not modified in any way, i.e. the synthetisation is performed only to generate the synthetic training sets. This setup will be used for our experiments for anomaly detection.

In Figure 2, we show the same projection as in Figure 1, but for the synthetic data generated by the DataSynthesizer and synthpop. We observe that Data-Synthesizer is generating the fraudulent records much closer to the legit ones, while for synthpop separate clusters of fraudulent data appear. This can be an indication for the utility of the datasets for the anomaly detection task. For the anomaly detection, we utilise the implementation provided in the Python scikit-learn framework2 . We specifically use these methods, a detailed description of which can be found at the scikit-learn documentation3 :

-Supervised: Logistic Regression, k-nearest Neighbours, Random Forests, Support Vector Machines (SVMs), Naïve Bayes -Semi-supervised: One-Class SVMs, Gaussian Mixture Models, Auto Encoder -Unsupervised: Isolation Forests, Local Outlier Factors

Evaluation

Following the generation of synthetic training data, the final step of our experiment is to train machine learning models on the real and the synthesised training datasets and to evaluate these models by comparing their prediction scores on the test data. For a binary task like anomaly detection, which either detects an anomaly or normal behaviour, there can be four different outcomes, depending on the label associated to the data point and the prediction of the machine learning model. Assuming that the fraudulent class is considered the "positive" class, we can distinguish two successful outcomes: true positives are data points that are frauds and detected as such, while true negatives are data points that are normal (legit) behaviour and identified as such. Two types of errors can occur: false positives are legit cases that the model predicts as frauds, while false negatives are fraud cases that have not been detected as such, but are considered legit. These are depicted in so-called confusion matrices, which for a binary classification task are 2 × 2 matrices, where the columns represent the two classes 0 ('legit') and 1 ('fraud'). The first row shows the number of elements in the class that have been predicted to be legit, the second row shows the number of elements that have been predicted to be fraudulent.

Based on this representation, we report the following measures. Precision is defined as the ratio of true positives (correctly identified frauds) to all data points predicted as frauds, i.e. how many of the cases the model has predicted to be frauds are actually such. Recall indicates how many of the frauds have been identified by the model, given as the ratio of the true positives to all cases that have been labelled as fraud. Each of these measures alone is not representative, as it is rather easy to optimise one of them, but difficult to have both take high values in conjunction. The F1 score provides a unified score, by computing the harmonic mean. The F2 score weighs recall higher than precision, which makes it suitable in our application, where it's likely more important to identify most of the anomalies, and a certain amount of false positives can be tolerated. In the following tables, we report these scores in percent. It is important to note that classification accuracy is less meaningful in our setting. It is defined as the ratio of true positives and true negatives to the size of the dataset. With highly imbalanced data, even models which trivially predict all instances as legit (normal) cases, and do not detect frauds at all, achieve high accuracy scores.

For each of the anomaly detection methods, we performed a random grid search over a number of parameters to optimise the results on the original training set. We then applied these settings, without any further optimisation, also to the models learned on the synthetically created datasets.

Table 1 shows the confusion matrices for the supervised anomaly detection algorithms, the metrics derived thereof are shown in Table 2. We can observe that the anomaly detection is a difficult task already on the original dataset, where Naïve Bayes is achieving a good recall, but has very low precision, and seems thus not usable for that task. The other algorithms perform significantly better, with k-Nearest Neighbours scoring best on precision, recall and F2 score.

On the synthetic datasets, we can see that the Synthetic Data Vault is not able to create datasets that can be learned by the supervised methods. The confusion matrices show that SVMs, k-NN and Random Forest cannot identify any of the anomaly data points, but classify them all as the "non-fraud" class, therefore achieving zero precision, recall, and F2 score. Only a very small number of frauds are correctly identified by Naïve Bayes and Logistic Regression, and while there are relatively few non-fraud cases wrongly predicted as fraud, the overall F2 score stays low -it is in the range of approximately 10%, and thus not useful. We can observe similar patterns for the DataSynthesizer when using Differential Privacy -even though larger values of recall are achieved, the more relevant F2 scores are still rather low, being in the range of 35% to 40%.

The models trained on the data synthesised by the DataSynthesizer without Differential Privacy do a better job at detecting frauds than the models trained on the data synthesised by the Synthetic Data Vault. Precision stays relatively high for most settings, but recall drops compared to the original data. Thus, the overall F2 scores drop by 15% to more than 20%, except for Naïve Bayes, which drops only marginally, but from a very low baseline of 24%. This trend of lower recall is slightly inverted for Logistic Regression, where it drops only marginally, by 1.8%, and thus achieves the best results for this synthetic dataset.

The best results on synthetic data are achieved with synthpop. For Logistic Regression, recall stays the same, and precision drops only marginally, and thus the overall F2 score is very close to the original dataset, lower just by 1.8%. For the other classifiers, the degradation is a bit larger, mostly with a lower recall, which results in an F2 score lower by approximately 10%. In overall, however, the scores are still relatively close to the original dataset.

In Table 3, we can see the confusion matrix for the unsupervised (Isolation Forest, Local Outlier Factor) and semi-supervised (One-Class SVM, Gaussian Mixture Model, Auto Encoder) methods. The scores are given in Table 4. We can observe that the unsupervised task, and to some extent also the semi-supervised task, is a much harder one. The major impact is on precision, where many methods struggle to achieve high values without reducing recall too much.

For the unsupervised methods, the Isolation Forest obtains the best precision values, and also high recall values. Thus, also in regards to the F2 score, Isolation Forest achieves by far the highest scores in the range of around 20%. It is interesting to note that the synthetic datasets are almost on par with the original data for Isolation Forests. Contrary to before, in this setting, the Synthetic Data Vault and DataSynthesizer with Differential Privacy are not much different than the other approaches -the latter is even the best of the synthetic datasets, achieving an F2 score of 20.4%. Results for the Local Outlier Factor are generally not satisfying. An interesting observation, however, is that all the synthetic datasets outperform the model on the original dataset -albeit still at a low overall score. Only synthpop is closer to the results of the Isolation Forest.

For the One-Class SVM as a semi-supervised method, the results are in between the two unsupervised other approaches. Interestingly, also for this setting, the synthetic dataset generated by synthpop performs better than the original one. Overall, however, the One-Class SVM shows a low precision. We obtain much better results with the Auto Encoder, which is mostly due to a much lower false-positive rate. The number of actual frauds correctly detected is 62 (55%) on the real data, and only marginally lower on most synthesizers, which achieve 59 or 60 correctly identified fraud cases. Precision and thus F2 score are the highest on the dataset created with synthpop, which is overall very close to the real dataset, with a just 0.4% lower score. Similarly to the unsupervised methods, also on this method the Synthetic Data Vault achieves better results than the DataSynthesizer. Finally, the best results on the semi-supervised task are achieved by the Gaussian Mixture Model. On the original dataset, 71.4% of the fraud cases are identified, with a low false-positive rate of only 12 records. The results on synthpop match the recall, but have a lower precision, thus resulting in an overall rather large drop of F2 score by approximately 14%. Again, the Synthetic Data Vault is the best of the synthesising methods, having actually a marginally higher recall, and a still acceptable precision; thus, the overall F2 score drops only around 9% from 74.1% on the original dataset.

While these are still below the results of the supervised approach, the semisupervised setting is a more difficult task, as it can rely only on labels for the "normal" cases. It is however a setting of practical application value, as it is generally easier to obtain these labels, and more difficult to label anomaly cases.

Conclusions and Future Work

In this paper, we evaluated the utility of synthetic data for the task of anomaly detection, on the example of fraud detection. We considered the setting where the data required to build the models can not easily be shared with the people responsible for the training due to its sensitive nature. In settings where multiple parties would like to collaborate to obtain a more powerful model, facilitating such exchange can be crucial. Synthetic data can be used to create a counterpart of the data that does not represent individual records, but still preserves important characteristics. Thus, the disclosure of sensitive data can be reduced.

In our evaluation, we considered the cases of supervised, semi-supervised and unsupervised anomaly detection. We have assumed that synthetic data might not be usable for such a task, as anomaly detection deals with outliers, and synthetic data generally preserves global characteristics -thus rather the ones of the legit, normal cases. However, the evaluation showed that while anomaly detection is generally a hard task, in specific settings, synthetic data can reach similar effectiveness as the models trained on the original data. It can thus be a viable alternative when the original data cannot be shared, and other forms of data sanitisation, such as anonymisation via k-anonymity, are not feasible.

Future work will extending our evaluation to additional anomaly detection data sets, e.g. on network intrusion, and will include additional detection methods, especially recent approaches like generative adversarial networks (GANs). Further, we will investigate how well synthetic data compares in terms of utility to datasets that need to be anonymised or otherwise treated before they could be shared. We will investigate further inference attacks on synthetic data, especially in the context of trying to infer information on the outliers.

Fig. 1 :

 1 Fig. 1: Projection of the original dataset, generated using t-SNE

Fig. 2 :

 2 Fig. 2: Projection of the synthetic datasets generated by DataSynthesizer (left) and synthpop (right), generated using t-SNE

Table 1 :

 1 Supervised anomaly detection: confusion matrix

		Naïve Bayes SVM	K-NN Random Forest Log Regression
	Real	55626 1224 26 86	56840 10 33 79	56845 5 25 87	56848 2 32 80	56836 14 26 86
	SDV	56838 12 100 12	56850 0 112 0	56850 0 112 0	56850 0 112 0	56845 5 102 10
	DS	55243 1607 17 95	56848 2 62 50	56848 2 58 54	56843 7 49 63	56776 74 28 84
	DSP	56498 352 41 71	56850 0 112 0	56850 0 112 0	56849 1 110 2	56745 105 70 42
	SP	55424 1426 24 88	56848 2 51 61	56845 5 39 73	56849 1 41 71	56824 26 26 86

Table 2 :

 2 Supervised anomaly detection: scores

	Naïve Bayes	SVM	K-NN	Random Forest Log Regression
	Pr Re F2 Pr Re F2 Pr Re F2 Pr Re F2 Pr Re F2
	Real 6.6 76.8 24.5 88.8 70.5 73.6 94.6 77.7 80.6 97.6 71.4 75.5 86.0 76.8 78.5
	SDV 50.0 10.7 12.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 66.7 8.9 10.8
	DS 5.6 84.8 22.1 96.2 44.6 50.0 96.4 48.2 53.6 90.0 56.3 60.8 53.2 75.0 69.3
	DSP 16.8 63.4 40.8 0.0 0.0 0.0 0.0 0.0 0.0 66.7 1.8 2.2 28.6 37.5 35.3
	SP 5.8 78.6 22.4 96.8 54.5 59.7 93.6 65.2 69.4 98.6 63.4 68.3 76.8 76.8 76.8

Table 3 :

 3 Unsupervised & semi-supervised anomaly detection: confusion matrix

		Isol. Forest	LOF	1-Class SVM GMM AutoEncoder
	Real	55335 1515 26 86	50756 6094 77 35	53655 3195 19 93	56838 12 32 80	56591 259 50 62
	SDV	55268 1582 34 78	51190 5660 13 99	51718 5132 13 99	56759 91 31 81	56582 268 52 60
	DS	55185 1665 28 84	51279 5571 25 87	52465 4385 14 98	56309 541 49 63	56533 317 53 59
	DSP	55445 1405 33 79	50755 6095 13 99	52126 4724 14 98	56643 207 53 59	56531 319 53 59
	SP	55128 1722 29 83	54283 2567 20 92	54110 2740 21 91	56719 131 32 80	56607 243 52 60

Table 4 :

 4 Unsupervised & semi-supervised anomaly detection: scores

	Isol. Forest	LOF	1-Class SVM	GMM	AutoEncoder
	Pr Re F2 Pr Re F2 Pr Re F2 Pr Re F2 Pr Re F2
	Real 5.4 76.8 21.0 0.6 31.3 2.7 2.8 83.0 12.4 87.0 71.4 74.1 19.3 55.4 40.3
	SDV 4.7 69.6 18.5 1.7 88.4 8.0 1.9 88.4 8.7 47.1 72.3 65.3 18.3 53.6 38.7
	DS 4.8 75.0 19.1 1.5 77.7 7.1 2.2 87.5 9.9 10.4 56.3 29.9 15.7 52.7 35.8
	DSP 5.3 70.5 20.4 1.6 88.4 7.5 2.0 87.5 9.3 22.2 52.7 41.3 15.6 52.7 35.7
	SP 4.6 74.1 18.4 3.5 82.1 14.8 3.2 81.3 13.9 37.9 71.4 60.7 19.8 53.6 39.9

Privacy-preserving Anomaly Detection using Synthetic Data

https://www.kaggle.com/mlg-ulb/creditcardfraud

Specifically, version 0.22, available at https://scikit-learn.org/0.22

https://scikit-learn.org/stable/modules/outlier detection.html

Acknowledgement

This work was partially funded by the BRIDGE 1 programme (No 871267, "Well-Fort") of the Austrian Research Promotion Agency (FFG), and the EU Horizon 2020 programme under grant agreement No 732907 ("MyHealth-MyData"). SBA Research (SBA-K1) is funded within the framework of COMET -Competence Centers for Excellent Technologies by BMVIT, BMDW, and the federal state of Vienna, managed by the FFG.