Liwei Chen
email: chenliwei@iie.ac.cn

Mengyu Ma
email: mamengyu@iie.ac.cn

Gang Shi Wenhao Zhang
email: zhangwenhao@iie.ac.cn

Dan Meng
email: mengdan@iie.ac.cn

Gadget Weighted Tagging: A Flexible Framework to Protect Against Code Reuse Attacks

The code reuse attack (CRA) has become one of the most common attack methods. In this paper, we propose gadget weighted tagging (GWT), a flexible framework to protect against CRAs. In GWT, we firstly find all possible gadgets, which can be used in CRAs. Then, we attach weighted tags to these gadgets based on the lengths and types of the gadgets, and the weighted values are configurable. At last, GWT monitors the weighted tag information at runtime to detect and prevent CRAs. Furthermore, combining with the rule-based CFI, GWT+CFI can precisely confirm the gadget start and greatly reduce the number of possible gadgets, compared to the baseline GWT. We implement a hardware/software co-design framework to support GWT and GWT+CFI. The results show that the performance overheads of GWT and GWT+CFI are 2.31% and 3.55% respectively, and GWT can defeat variants of CRAs, especially those generated by automated tools. Gadget Gadget Normal Code Normal Code Normal Code Normal Code Gadget Gadget Gadget Gadget Gadget Gadget Normal

Introduction

Code reuse attack (CRA) has become the primary attack vector nowadays. Attackers find special code snippets called gadgets, ending with an indirect branch instruction, and then manipulate program control flow to chain gadgets together to construct a malicious program. The CRA is proved to be Turing-complete and can be generated by automated tools [START_REF] Schwartz | Q: Exploit Hardening Made Easy[END_REF]. Therefore, the CRA is an attack approach which is easy-to-use, hard-to-detect and can be applied to any purpose.

Many defense mechanisms have been proposed to protect against CRAs, and control flow integrity (CFI) [3][15] is thought to be one of the most promising ways. However, CFI has high complexity and low efficiency [START_REF] Burow | Control-Flow Integrity: Precision, Security, and Performance[END_REF]. In order to design a practicable CFI, some researches [START_REF] Zhang | Practical Control Flow Integrity and Randomization for Binary Executables[END_REF] [START_REF] Zhang | Control Flow Integrity for COTS Binaries[END_REF] [START_REF] Cheng | ROPecker: A Generic and Practical Approach For Defending Against ROP Attack[END_REF] [START_REF] Pappas | Transparent ROP Exploit Mitigation Using Indirect Branch Tracing[END_REF] propose the coarse-grained CFI to realize a looser notion of control flow integrity without the static control-flow graph (CFG). Abandoning CFG to reduce complexity, the coarse-grained CFI is a tradeoff between security and efficiency. Hence, the security of the coarse-grained CFI is lower than the CFI based on CFG. Some recent papers [START_REF] Göktas | Out of Control: Overcoming Control-Flow Integrity[END_REF][13][8] [START_REF] Davi | Stitching the Gadgets: On the Ineffectiveness of Coarse-grained Control-flow Integrity Protection[END_REF] have proved that the coarse-grained CFI could be bypassed by CRAs with special gadgets.

We observe that all possible gadgets, which can be used in CRAs, only take a part of the whole code space. As shown in Figure 1, gadgets in normal programs are discontinuous usually, whereas the CRA is composed of a contiguous sequence of gadgets. Additionally, with some special gadgets inserted, such as long-NOP gadgets and callpreceded gadgets [START_REF] Davi | Stitching the Gadgets: On the Ineffectiveness of Coarse-grained Control-flow Integrity Protection[END_REF] [START_REF] Kayaalp | SCRAP: Architecture for Signature-based Protection from Code Reuse Attacks[END_REF], CRAs may pretend to be normal programs. Therefore, we can distinguish CRAs from normal programs by monitoring both normal gadgets and special gadgets at runtime.

In this paper, we propose gadget weighted tagging (GWT), a flexible framework to detect and prevent CRAs. In GWT, we classify all possible gadgets into three major types on the basis of their effects on CRAs: functional gadgets, NOP gadgets and normal codes. Then, we configure different weighted scores to these gadget types, and attach weighted tags to the gadgets in binary files. Finally, GWT computes the probability of CRA occurrence by dynamically monitoring the weighted tags at runtime. Moreover, we propose GWT+CFI, since we discover that the control-flow rules in coarse-grained CFI are good supplements to the GWT. GWT+CFI can accurately identify the gadget start, and significantly reduce the number of potential gadgets. Hence, it is more precise to find possible gadgets in GWT+CFI, compared to the baseline GWT.

Several important parameters used in GWT, such as weighted scores for different gadget types, are configurable. Thus, users can modify the parameters as needed. Furthermore, if users discover some special gadgets, they can add the new gadget types into the GWT, and allocate different weighted scores for the gadget types. As a result, GWT can detect variants of CRAs with different gadgets, and the security of GWT can be further improved with adding more gadget types.

When a new CRA attack method or a new gadget type is proposed, the ways to construct the attack or to find the gadget should also be introduced. Hence, in GWT, we can reuse the ways to find the potential gadgets in programs, according to the descriptions in previous papers. Additionally, we can find and mark possible gadgets in GWT by reusing automated tools for CRA generation [START_REF] Schwartz | Q: Exploit Hardening Made Easy[END_REF].

We apply GWT and GWT+CFI in a hardware/software co-design framework. The results show that the performance overheads of GWT and GWT+CFI are 2.31% and 3.55% respectively. Moreover, GWT can detect all CRAs generated by Q [START_REF] Schwartz | Q: Exploit Hardening Made Easy[END_REF], and variants of CRAs with special gadgets.

The key contributions of this paper are summarized as follows:

1) We propose gadget weighted tagging (GWT), a novel framework to prevent CRAs, which is of high-flexibility, high-security, low-overhead and easy to apply.

2) We propose GWT+CFI, GWT combining with the rule-based CFI, which can find possible gadgets more precisely, compared to the baseline GWT.

3) We implement a hardware/software co-design to support GWT and GWT+CFI, and evaluate its security effectiveness and performance overhead.

2 Background and Related Work

Code Reuse Attacks (CRAs)

The basic idea of CRAs is to reuse instructions from the existing code space to implement malicious operations. In order to achieve attack purposes, attackers should firstly find a set of instruction sequences (called gadgets) from the entire code space, and then link the selected gadgets into a gadget chain to construct a malicious program. In short, the CRA is composed of a contiguous sequence of gadgets.

A gadget usually has several normal instructions for computation and an indirect branch instruction to change control flow to link gadgets. For simplicity, we focus on three most common indirect branch instructions in this paper: call, return and indirectjump. Furthermore, the gadgets used in CRAs have two important features as follows:

1) Sparse Distribution. Every gadget should end with an indirect branch instruction, however, the indirect branch instructions only take a small part of the whole program. Moreover, attackers need gadgets containing special operations to construct a malicious program, or to bypass defense mechanisms, such as CFI. Thus, many existing gadgets may not meet for the special needs of attackers. As a result, the gadgets, which are useful for CRA construction, are quite rare in normal programs.

2) Small Size. The gadgets with more instructions can perform more operations, however, they also inevitably lead to more side effects and some of them may conflict with each other. Hence, attackers usually prefer to discover the short and simple gadgets only with the intended operations [START_REF] Schwartz | Q: Exploit Hardening Made Easy[END_REF], instead of using long and complex gadgets. In fact, the gadgets in real CRAs usually have only 2 to 6 instructions [START_REF] Cheng | ROPecker: A Generic and Practical Approach For Defending Against ROP Attack[END_REF].

According to the difference of the indirect branch instructions, CRAs can be classified into return-based ROP (return-oriented programming) and jump-based JOP (jumporiented programming). In ROP, the stack pointer esp is used as the program counter in a return-oriented program. Instead, JOP uses a dispatch table to manage gadget addresses. The program counter of JOP is the register pointing into the dispatcher table [START_REF] Tyler | Zhenkai: Jumporiented Programming: A New Class of Code-reuse Attack[END_REF], and an special dispatcher gadget is used to drive control flow. Thus, the dispatcher gadget is a key sign of JOP, which usually contains an self-modification operation and an indirect jump instruction. Generally, any gadget ending with an indirect jump instruction, that carries out the following algorithm, can be considered as a candidate of dispatcher gadget [START_REF] Tyler | Zhenkai: Jumporiented Programming: A New Class of Code-reuse Attack[END_REF].

pc ← -f(pc); goto *pc;

Control Flow Integrity (CFI)

There are many CFI implementations that have been proposed [START_REF] Abadi | Control-flow Integrity Principles, Implementations, and Applications[END_REF][4] [START_REF] Davi | Hardware-assisted fine-grained control-flow integrity: Towards efficient protection of embedded systems against software exploitation[END_REF][10][6] [START_REF] Pappas | Transparent ROP Exploit Mitigation Using Indirect Branch Tracing[END_REF]. We divide these CFI methods into two main kinds: CFG-based CFI and rule-based CFI. CFG-based CFI, also is known as fine-grained CFI, enforces the control flow to adhere to control-flow graph (CFG) generated by static program analysis. CFG-based CFI is high security, and can detect any illegal control flow transfers. However, it is almost impossible to generate an ideal CFG containing all possible control-flow paths, and the performance overhead of CFG-based CFI is high [START_REF] Burow | Control-Flow Integrity: Precision, Security, and Performance[END_REF]. Furthermore, some recent works [START_REF] Evans | Control Jujutsu: On the Weaknesses of Fine-Grained Control Flow Integrity[END_REF] [START_REF] Conti | Losing Control: On the Effectiveness of Control-Flow Integrity Under Stack Attacks[END_REF] have proposed CRA attack methods to bypass the CFG-based CFI with unideal CFG. In addition, even the ideal CFG-based CFI could be also bypassed by control flow bending [START_REF] Carlini | Control-Flow Bending: On the Effectiveness of Control-Flow Integrity[END_REF] combining with the non-control-data attack.

Rule-based CFI, that is also called coarse-grained CFI, uses several control-flow rules to defend against CRAs, instead of CFG. Therefore, the rule-based CFI is lowoverhead and easy to implement. The control flow rules can be classified into two types as following.

1) Control-flow transfer rules. These rules stipulate the destinations of different indirect branch instructions [11][9].

1.1) RETURN: a return instruction should point to an instruction right after a call instruction.

1.2) CALL: a call instruction should start execution at the entry point of a function.

1.3) Indirect-JUMP: an indirect-jump instruction should point to either a position inside the same function, or an entry point of another function.

2) Code length rules. These rules stipulate the code length of gadgets, and the chain length of CRA attacks [START_REF] Pappas | Transparent ROP Exploit Mitigation Using Indirect Branch Tracing[END_REF] [START_REF] Cheng | ROPecker: A Generic and Practical Approach For Defending Against ROP Attack[END_REF].

2.1) Gadget length: a short code snippet (e.g. 2-6 instructions) ending with an indirect branch instruction is a gadget, whereas a longer code snippet is not a gadget.

2.2) CRA length: a long gadget chain (e.g. 6 gadgets) is a CRA, whereas a shorter gadget chain is not a CRA.

Legal Gadgets

The rule-based CFI has several definite rules to recognize gadgets. However, attackers can find some special gadgets that obey these rules, called legal gadgets in this paper. As a result, legal gadgets are mistaken for normal codes by the rule-based CFI, so that CRAs consisting of legal gadgets can bypass the rule-based CFI. There are several kinds of legal gadgets to bypass different control-flow rules.

1) Call-Preceded (CP) Gadget. A call-preceded instruction is any instruction in the address space of the application that immediately follows a call instruction [START_REF] Davi | Stitching the Gadgets: On the Ineffectiveness of Coarse-grained Control-flow Integrity Protection[END_REF]. A call-preceded gadget is a gadget that its first instruction is a call-preceded instruction. Thus, CP gadgets are legal destinations of any return instruction.

2) Entry-Point (EP) Gadget. An entry-point gadget is a gadget starting at the entry point of a function. Hence, EP gadgets are legal destinations of any call or indirect-jump instruction.

3) Long-NOP gadget. A long-NOP gadget [START_REF] Davi | Stitching the Gadgets: On the Ineffectiveness of Coarse-grained Control-flow Integrity Protection[END_REF][7] is a gadget that contains enough instructions to obey the gadget length rule of rule-based CFI, and does not induce any side effects, i.e., the content of all registers and memory area used by the CRA is preserved. Therefore, attackers can use short and useful gadgets to perform malicious operations, and use long-NOP gadgets to bypass the rule-based CFI.

Threat Model

In this paper, we focus on defending against code reuse attacks. We assume that attackers have full control over data and stack/heap memory regions. In addition, we also assume that the attackers can modify all key registers of processors.

We assume that the DEP technology has already been realized, thus it forces the attackers to use CRA attacks. Moreover, we do not consider the gadgets consisting of a sequence of unintended instructions, such as in x86 platform with variable instruction sizes, since they can be detected by CFI [START_REF] Kayaalp | Branch regulation: Low-overhead protection from code reuse attacks[END_REF].

Furthermore, we will introduce our approach, GWT, in two different systems.

1) Baseline System. The baseline system does not have any special security mechanism, and is only protected by the DEP technology.

2) Protected System. The protected system supports the DEP technology and the rule-based CFI with control-flow transfer rules as described in Section 2.2.

4 Gadget Weighted Tagging 4.1 Finding Gadgets 1) Finding Gadget End.

The only strict limit of gadgets is ending with an indirect branch instruction (return, call and indirect-jump). Thus, all code snippets ending with an indirect branch instruction can be considered as gadgets in theory. Therefore, we should discover all indirect branch instructions in program codes, and take these instructions as the gadget end.

2) Defining Gadget Types.

Q [START_REF] Schwartz | Q: Exploit Hardening Made Easy[END_REF], the state-of-the-art automated tool for CRA generation, uses functional kinds to identify gadgets. Q defines nine kinds of gadgets, including NoOp, Jump, MoveReg, LoadConst, Arithmetic, LoadMem, StoreMem, ArithmeticLoad, and ArithmeticStore [START_REF] Schwartz | Q: Exploit Hardening Made Easy[END_REF]. In Q, every gadget should have and only have one kind to define its function, and Q proposes a discovery algorithm to recognize the functional kind of each gadget.

In GWT, we define four types of gadgets as shown in Figure 2. The functional gadget includes the eight kinds of gadgets defined in Q, except the NoOp gadget. The dispatcher gadget and the syscall gadget are two special types of functional gadgets. Because the two types of gadgets play critical roles in CRAs, we take them as separate types out from the normal functional gadgets.

2.1) Functional Gadget. The functional gadget is a gadget that can perform a fixed and useful operation without any side effect, such as arithmetic calculation, branching, or loading data from memory. We use a method similar as Q [START_REF] Schwartz | Q: Exploit Hardening Made Easy[END_REF] to identify functional gadgets. Details of the algorithm to identify the functional gadgets can be found in [START_REF] Schwartz | Q: Exploit Hardening Made Easy[END_REF]. However, the original algorithm of Q can only be used to discover return-based gadgets and construct ROP chains. Hence, we have added new features, such as discovering indirect-jump instructions and syscall instructions, to find jump-based gadgets to construct JOP chains, and to discover syscall gadgets and legal gadgets in GWT.

Program Codes

Functional

Gadgets Dispatcher Gadgets

Normal Codes

NOP Gadgets

Gagets

Syscall Gadgets

Normal Functional Gadgets Fig. 2. The Gadget Types in GWT 2.2) Dispatcher Gadget. The dispatcher gadget is an special functional gadget, which ends with an indirect jump instruction and contains an self-modification operation to the jump destination. Furthermore, the dispatcher gadget plays a key role in the JOP [START_REF] Tyler | Zhenkai: Jumporiented Programming: A New Class of Code-reuse Attack[END_REF]. It essentially maintains an virtual program counter and executes the JOP program by advancing it through one gadget after another.

2.3) Syscall Gadget. The syscall gadget is an special functional gadget ending with an system call instruction. To make CRA attacks simpler, attackers can carefully construct attacks that consist of a small number of gadgets, and then inject code. For example, attackers can use the syscall gadget to invoke an system call to alter the execute bit on an attacker-controlled buffer (i.e. destroy the DEP protection), and then redirect control flow to it [START_REF] Göktas | Out of Control: Overcoming Control-Flow Integrity[END_REF].

2.4) NOP Gadget. The NOP gadget is a gadget which has neither useful operations nor side effects to CRAs. It includes the NoOp gadget in Q and long-NOP gadgets described in Section 2. To ensure that the NOP gadget does not break the semantics of the CRA chain, the NOP gadget should make use of only a small set of registers [START_REF] Davi | Stitching the Gadgets: On the Ineffectiveness of Coarse-grained Control-flow Integrity Protection[END_REF]. Generally, the NOP gadget can be used to confuse defense mechanisms, such as the rule-based CFI [START_REF] Cheng | ROPecker: A Generic and Practical Approach For Defending Against ROP Attack[END_REF] [START_REF] Pappas | Transparent ROP Exploit Mitigation Using Indirect Branch Tracing[END_REF].

We define a parameter in GWT, MaxRegMod, which means the maximum number of registers modified by the NOP gadget. If a gadget is not a functional gadget and the registers modified by this gadget are not larger than MaxRegMod, then the gadget is a NOP gadget. Else, the gadget is not a NOP gadget (i.e. normal code).

3) Finding Gadget Start (Maximum Gadget Length).

It is simple to find the gadget end (i.e. indirect branch instructions), whereas to identify the gadget start is difficult. In previous papers, the code length is always used to recognize gadgets, since longer gadgets lead to more side effects and may conflict with other gadgets. Usually, the maximum length of a meaningful gadget is 6 instructions [START_REF] Cheng | ROPecker: A Generic and Practical Approach For Defending Against ROP Attack[END_REF]. However, attackers can still find some special gadgets (e.g. long-NOP gadgets), which are long enough and have few side effects to CRAs, to break the limits.

In GWT, the classified gadget type still depends on code length, however, we add a new gadget type (i.e. NOP gadget) to mitigate the above problem. A long enough gadget, which has few side effects, could be recognized as a NOP gadget in GWT, instead of normal code in previous papers. A gadget will be identified as normal code, only if the gadget contains too many instructions leading to significant side effects.

3.1) The Maximum Length of the Functional Gadget. From the gadget end of a functional gadget, walks backwards one by one (instruction), until the gadget is not a functional gadget. Then, the instruction number is the maximum length of the functional gadget, and the first instruction is the start of the functional gadget. Because the dispatcher gadget and the syscall gadget both belong to the functional gadget, the maximum lengths of the two types of gadgets are the same as the functional gadget.

3.

2) The Maximum Length of the NOP Gadget. It is similar as the functional gadget. Increases the code length of a NOP gadget, until the gadget is recognized as normal code. Then, the code length is the maximum length of this NOP gadget.

Weighted Tagging

In GWT, we attach the weighted tag to each gadget end (i.e. indirect branch instruction). The structure of weighted tag is shown in Figure 3, and it has 32 bits and contains 3 parameters. The gadget type has 3 bits, and the meaning of this parameter is shown in Table 1. The maximum length of functional gadget has 14 bits, and the maximum length of NOP gadget has 15 bits.

We define five different gadget types, including normal code, functional gadget, dispatcher gadget, syscall gadget and NOP gadget. Normal code means the gadget can not be used in any CRA, and other four gadget types are introduced in Section 4.1. Different gadgets have different effects on CRAs, hence, we propose weighted scores to mark different gadget types. The foundation of weighted scores is the dangerousness and importance of each gadget type. With a higher score, the gadget type is more dangerous and important, and is also more possible to be used in CRAs.

The specific weighted score of each gadget type is shown in Table 1, and it is configurable. (i) The NOP gadget does not contain any useful operation for CRAs, and thus it has only 0 point. (ii) The functional gadget contains an useful and constant operation for CRAs, so it gets 1 point. (iii) The dispatcher gadget is a key sign of the JOP, and it is appeared every other gadget in the JOP, hence it has 2 points. (iv) System call is the start point of many important kernel functions of OS, and using the system call can significantly reduce the difficulty of CRA construction. Consequently, one common purpose of CRAs is to perform an special system call, and the syscall gadget is more important than other gadget types, which has 4 points. (v) Since the CRA is composed of a contiguous sequence of gadgets, if the current gadget is normal code, there is no CRA at present. As a result, the weighted score of the normal code is zero clearing.

We add the gadget tag information into the program executable file by annotation. The prior of every indirect branch instruction is a weighted tag annotation. Moreover, tag annotation starts with a prefetch instruction to retain binary compatibility, which is the similar as other methods [3][9]. The prefetch instruction is followed by the weighted tag information. Because we assume that the system supports the DEP technology, the weighted tag annotation can not be modified by attackers.

Gadget Type

The Maximum Length of Functional Gadget

The Maximum Length of NOP Gadget 0 2 3 16 31 17

Fig. 3. Weighted Tag Information for Gadget End

Monitoring Gadget Tags

At runtime, we assume that codes between two indirect branch instructions construct a potential gadget, and we identify the type of the potential gadget by its code length and the weighted tag information attached to the second indirect branch instruction. Furthermore, we define four simple intermediate variables to calculate the probability of CRA occurrence: current gadget type (CGT), current gadget length (CGL), real gadget type (RGT) and CRA occurrence index (COI). CGL is used to record the code length of the current potential gadget, and CGT is used to record the gadget type of the current gadget tag. When an instruction is executed, the CGL adds one to itself. When an indirect branch instruction is executed, it is the boundary between two potential gadgets. Therefore, CGT records the gadget type of the corresponding tag, while CGL records the current gadget length for RGT computation, and then starts a new count of code length for the next gadget.

RGT, which is computed based on the CGT and CGL, records the real type of the current gadget at runtime. The detailed algorithm process of RGT is described in Figure 4. The MaxFunc is the maximum length of the functional gadget, and the MaxNOP is the maximum length of the NOP gadget.

We use COI to denote the possibility of the CRA occurrence. If the COI is larger, then it is more likely CRA will occur. Furthermore, we define MaxCOI, the maximum value of COI. If the COI is larger than MaxCOI, then we assume that a CRA occurs. The COI is calculated based on RGT and weighted scores of gadget types, and the detailed algorithm process of COI is described in Figure 4. CRA detecting module (CDM). In addition, we assume that the rule-based CFI has already been implemented.

Hardware Implementation

The CFM is the main module for weighted information monitoring. It records the number of instructions executed (CGL) and gadget type (CGT) of current gadget at runtime. When an indirect branch instruction is executed, the CFM computes real gadget type (RGT) based on CGL and CGT, and then sends the RGT to the CDM. The CDM computes COI, and compare it to the MaxCOI. If the COI is larger than MaxCOI, then a CRA occurs, and thus CDM should send an exception to report an attack. The detailed computation algorithms of these parameters are described in Figure 4.

The WTC stores the parameters used by CFM and CDM, such as MaxCOI and weighted scores. Additionally, these parameters are configurable. Users can modify these parameters as needed. For example, if users want to improve system security, they can reduce the MaxCOI, or increase the weighted scores of gadgets. Furthermore, we assume that attackers can not directly change these parameters stored in WTC. One main shortage of GWT is that GWT can not precisely confirm the gadget start.

Only using the gadget length to identify the gadget start is not clear enough. Therefore, we propose GWT+CFI, combining GWT and the rule-based CFI together. The rulebased CFI fits our approach very well, and introduces two benefits into the GWT. 1) Gadget Start. In order to bypass the rule-based CFI, attackers can only use some special gadgets, such as call-preceded (CP) gadgets and entry-point (EP) gadgets. Fortunately, the two types of gadgets both have fixed start points: call-preceded instruction and the function entry. Consequently, we can precisely identify the gadget start in GWT+CFI.

2) Gadget Number. Except legal gadgets, other illegal gadgets can be detected by the rule-based CFI. As a result, the number of gadgets that can be used in CRAs is reduced significantly in GWT+CFI. Therefore, we can locate useful gadgets in code space more accurately.

Finding Legal Gadgets

The major difference between GWT and GWT+CFI is the gadget discovery. In GWT+CFI, we only need to discover legal gadgets. On the other stages, such as gadget weighted tagging and monitoring, the GWT+CFI is almost the same as the GWT.

In GWT+CFI, we should find all indirect branch instructions firstly, and take them as the gadget end. Then, we should also find the gadget start. The start point of a CP gadget is a CP instruction, and the start point of an EP gadget is the function entry. Consequently, we can identify all legal gadgets based on the gadget start and end. At last, we should select useful gadgets from these legal gadgets. It is almost the same as the Section 4.1, to distinguish functional gadgets, dispatcher gadgets, syscall gadget, NOP gadgets and normal codes, as shown in Figure 6.

Note that one indirect jump instruction can jump to any position inside the same function, that does not violate the control-flow transfer rules of CFI. Hence, attackers may find legal gadgets without CP instruction or function entry by indirect jump instructions. Therefore, if we find that a function contains indirect jump instructions, the gadget discovery inside this function should follow the baseline GWT, instead of GWT+CFI.

Program Codes

Legal Gagets

Normal Codes ( and Illegal Gadgets)

NOP Gadgets

Functional

Gadgets Dispatcher Gadgets

Syscall Gadgets

Normal Functional Gadgets Fig. 6. The Gadget Types in GWT+CFI

Security Analysis

Because the hardware architecture of GWT is very simple, it is only used to record gadget information and compute COI. As a result, for simplicity, we implement a software method based on pintools [START_REF] Luk | Pin: Building Customized Program Analysis Tools with Dynamic Instrumentation[END_REF] to simulate hardware functionality of GWT for security analysis. Pintools [START_REF] Luk | Pin: Building Customized Program Analysis Tools with Dynamic Instrumentation[END_REF] are dynamic program analysis tools, which can manage the execution of every instruction. In addition, we select widely-used programs in Ubuntu /bin and /usr/bin/. In order to keep the balance between security and stability, we set the MaxRegMod is 6, MaxCOI is 6, and the weighted scores are the same as the Table 1.

Since the main difference between GWT and GWT+CFI is the gadget discovery, the CRA detection of GWT+CFI is almost the same as the GWT. Hence, we mainly focus on the security analysis of GWT in this section.

Gadget Discovery

In GWT, the average number of all possible gadgets is about 1072 in each program with average 40KB code size, and thus the gadgets take only a small part of the whole program codes. The average numbers of functional gadgets, dispatcher gadgets, syscall gadgets and NOP gadgets are 452, 7, 1.5 and 612 respectively. Furthermore, the average length of all possible gadgets is about 9.9 in each program. The average length of functional gadgets, including the dispatcher gadgets and syscall gadgets, is 5.3, and the average length of NOP gadgets is 13.3, as shown in Figure 7.

In GWT+CFI, the average number of legal gadgets is about 471, which decreases significantly compared to the baseline GWT, especially the legal functional gadgets. The average numbers of functional gadgets, dispatcher gadgets, syscall gadgets and NOP gadgets are 45, 0.9, 0.15 and 425 respectively. Conversely, the average length of legal gadgets is about 13.5, which is longer than the GWT. The average lengths of functional gadgets and NOP gadgets are 7.7 and 14.1, as shown in Figure 7.

MaxRegMod is the maximum number of registers modified, and defines the boundary between NOP gadgets and normal codes. Obviously, increasing the MaxRegMod will increase the maximum lengths of potential NOP gadgets, and include more normal codes into NOP gadgets. Thus, it may also increase the false positive rates of GWT. Figure 8 shows the changes of average lengths of NOP gadgets in GWT and GWT+CFI, with the MaxRegMod increasing. Although a processor has 32 registers, many registers have special purposes, and only 8 registers can be used for general-purpose computation usually. As a result, the average length of NOP gadgets has little change, after MaxReg-Mod is larger than 8. Additionally, because NOP gadgets have few effects on CRAs, and the weighted score of NOP gadget is zero, appropriately increasing MaxRegMod will not affect the CRA detection rate of GWT usually.

Practical Attacks

We test GWT against CRAs generated by Q [START_REF] Schwartz | Q: Exploit Hardening Made Easy[END_REF], which can automatically generate ROP payloads for given programs. Since we have already marked all possible gadgets in these programs, GWT can easily detect the attacks consisting of gadgets, and count the weighted scores of gadgets in the CRA chains. Moreover, since Q can only generate return-based gadgets, the gadgets are normal functional gadgets in GWT, thus their weighted scores are 1 point. As a result, if the gadget number is larger than the MaxCOI, a CRA is detected. At last, the GWT can successfully detect 100% all the payloads generated from more than 100 applications under the directory /bin and /usr/bin/. Then, we try to build some CRAs manually to bypass GWT. A practical way is to structure a short and simple gadget chain to close DEP, and then to inject malicious codes [START_REF] Göktas | Out of Control: Overcoming Control-Flow Integrity[END_REF]. At last, we build a few CRAs to close DEP in a well-designed targeted program, which contain only a syscall gadget, a functional gadget and several NOP gadgets. The COI of these CRAs is 5, which is smaller than MaxCOI [START_REF] Cheng | ROPecker: A Generic and Practical Approach For Defending Against ROP Attack[END_REF]. However, it is very difficult to construct practical attacks to bypass GWT, since we can not find such satisfactory gadgets in real-world programs.

The weighted score of gadget types and MaxCOI are two critical parameters for CRA detection. Larger weighted scores and smaller MaxCOI can offer higher security, but may also increase the false positive ratio. Thus, we should define proper values to balance the security and stability. Figure 9 demonstrates the CRA detection rate and false positive rate of GWT with different MaxCOI. With the MaxCOI increasing, GWT needs more gadgets to identify the CRAs. CRAs generated by Q usually consist of 20-40 gadgets, and thus to set MaxCOI less than 20 can detect most CRAs generated by Q. On the other hand, with the MaxCOI decreasing, GWT may mistake some normal programs for CRAs. For example, a normal program invokes a system call, and this system call may be recognize as a syscall gadget. If we set MaxCOI less than 4, GWT will mistake the system call invoked by this normal program for a CRA.

Performance Analysis

In this section, we implement hardware architecture of GWT and GWT+CFI in RTL (Register Transfer Level) using Verilog on a Loongson processor. Moreover, we select workloads from the SPEC CPU2006 benchmark [START_REF] Henning | SPEC CPU2006 Benchmark Descriptions[END_REF], and these benchmarks are compiled using the GNU version of GCC at O3 optimization level. We compare the performance overheads of GWT, RBCFI (rule-based CFI) and GWT+CFI on the target applications and the system. Note that we only perform RBCFI with three control-flow rules as described in Section 3 (i.e. protected system). We perform assembly-level instrumentation of the binaries for both GWT and RBCFI to insert the additional information needed to perform the checks for both methods.

As shown in Figure 10, the performance overhead of GWT is just about 2.31% on the average and it is less than 6% for all benchmarks. Furthermore, the performance overhead of GWT+CFI is about 3.55% on the average and it is less than 9% for all benchmarks, which includes the overhead of GWT as well as the overhead of rulebased CFI.

Conclusions and Future Work

In order to defend against CRAs, we propose GWT, a flexible, low-overhead and highsecurity framework. GWT discovers and marks all potential gadgets with static program analysis, and then dynamically monitors the weighted gadget tags at runtime to calculate the probability of CRA occurrence. Furthermore, we propose GWT+CFI, GWT combining with the rule-based CFI, can more precisely find possible gadgets in code space, compared to the baseline GWT. We implement GWT and GWT+CFI in a hardware/software co-design system. The results show that the performance overheads of GWT and GWT+CFI are 2.31% and 3.55% respectively, and GWT can defeat variants of CRAs with different gadgets, especially those generated by automated tools. Moreover, GWT can be further optimized by adding more special gadget types and better gadget type classification.

The identification of the functional gadget, the NOP gadget and the normal code in GWT is not perfect. Attackers may manually find some useful gadgets which are identified as NOP gadgets in GWT, or find some NOP gadgets which are recognized as normal codes in GWT. Therefore, we plan to propose more precise methods to distinguish the gadget types. Additionally, it is our future work to insert discovery algorithms of special gadgets to improve the security of GWT, such as function-based gadgets.

Acknowledgement

This work is partially supported by the National Natural Science Foundation of China (No. 61602469).

Fig. 1 .

 1 Fig. 1. The Gadgets in Normal Program and CRA

Figure 5

 5 Figure 5 shows the hardware implementation for GWT. It contains three important hardware modules: control-flow monitor (CFM), weighted tag configuration (WTC) and

Fig. 4 .

 4 Fig. 4. The algorithm process of RGT and COI

Fig. 5 .

 5 Fig. 5. Hardware Implementation of GWT

Fig. 7 .Fig. 8 .

 78 Fig. 7. The Average Numbers (left) and the Average Lengths (right) of Different Gadgets in GWT and GWT+CFI

Table 1 .

 1 Weighted Scores of Different Gadget Types

	Gadget Type	Gadget	Weighted
	Value (3 bits)	Type	Score
	000	Normal Code zero clearing
	001	Functional Gadget	1
	010	Dispatcher Gadget	2
	011	Syscall Gadget	4
	100	NOP gadget	0
	others	undefined	-

  The CRA Detection Rate (left) and The False Positive Rate (right) of GWT

		120%		120%
		100%		100%
	Detection Rate Detection Rate	60% 80% 20% 40%	False Positive Rate False Positive Rate	60% 80% 20% 40%
		0%		0%
		0 5 10 15 20 25 30 35 40 45 50 55 60		0	1	2	3	4	5	6	7	8	9	10
		MaxCOI							MaxCOI
	5.00% 6.00% 7.00% 8.00% 9.00% Fig. 9. 4.00% 4.00%											GWT
													RBCFI
	3.00%
	2.00%
	1.00%
	0.00%

GWT+CFI Fig. 10. Performance Overheads of Different Methods (Normalized to the Baseline System)


Rene Meis
email: rene.meis@paluno.uni-due.de

Maritta Heisel
email: maritta.heisel@paluno.uni-due.de

Towards Systematic Privacy and Operability (PRIOP) Studies

The assessment of privacy properties of software systems gains more and more importance nowadays. This is, on the one hand because of increasing privacy concerns of end-users due to numerous reported privacy breaches, and on the other hand due to stricter data protection regulations, e.g., the EU General Data Protection Regulation that prescribes an assessment of the privacy implications that a project possibly has. The lack of systematic methods to assist a comprehensive and detailed privacy analysis makes it hard for analysts to address the end-users' and legal requirements. In this paper, we adopt the principles of the hazard and operability (HAZOP) studies, which have successfully been used for safety analyses, to privacy to provide a systematic method to identify the relevant privacy threats for a software to be developed. We propose a method called privacy and operability (PRIOP) studies that allows to systematically analyze the potential privacy issues that a software to be developed might raise, based on the software's functionality at the requirements level.

1 We consider any information that is related to a natural person as personal information. We call this natural person data subject.

Introduction

Privacy is a software quality that gains more and more attention these days. On the one hand end-users are more concerned about privacy and call for more transparency on how their personal information 1 (PI) is processed [START_REF]MOBILE PRIVACY: Consumer research insights and considerations for policymakers[END_REF]. On the other hand different legislators prescribe that data protection/privacy impact assessments ((D)PIAs) are performed, e.g., the European Union in the new EU General Data Protection Regulation. A (D)PIA has to be performed for all kinds of projects that involve the processing of PI. Its goal is to assess the implications of the project on the data subjects' privacy.

A central element of a (D)PIA is the identification and evaluation of privacy threats to estimate the privacy risks implied by the considered project. In this paper, we focus on software projects and want to assist analysts to identify and evaluate the privacy threats of a software project as early as possible during the development process, namely in the requirements engineering phase. To do so, we adopt the Hazard and Operability (HAZOP) [START_REF]2001 Hazard and Operability Studies (HAZOP Studies) -Application Guide[END_REF] studies, which have successfully been used to assess the safety implications of a system, to a systematic methodology called Privacy and Operability (PRIOP) studies. We illustrate how PRIOP can be applied based on artifacts produced by the Problem-based Privacy Analysis (ProPAn) method [START_REF] Meis | Computer-aided identification and validation of privacy requirements[END_REF].

The rest of the paper is structured as follows. Section 2 introduces a small eHealth scenario as running example, and HAZOP and ProPAn as background of this work. PRIOP is introduced in Section 3. Section 4 discusses related work and Section 5 concludes the paper.

Background

Running Example We illustrate how a PRIOP study is performed using an electronic health system (EHS) scenario provided by the industrial partners of the EU project Network of Excellence (NoE) on Engineering Secure Future Internet Software Services and Systems (NESSoS ). This scenario is based on the German health care system which uses health insurance schemes for the accounting of treatments. The functionalities of the considered system cover the management of electronic health records (EHRs) (functional requirements R1 and R2), the interaction with mobile devices of patients (R5 and R6), the accounting and billing of patients (R3 and R4), and providing anonymized medical data for clinical research (R7).

In this paper, we focus on the functional requirement R3. R3 is concerned with the problem that doctors shall be able to perform the accounting of treatments that patients received from them. For this, the treatments, diagnoses, and insurance number of the patient are passed to an external insurance application that provides the connection to the patient's insurance company. This insurance application then returns the information which treatments are beared by the patient's insurance contract and the software-to-be shall create an invoice for the treatments that are not covered by the patient's insurance contract. For this, the doctor additionally enters the costs for the treatments.

Hazard and Operability Studies The international standard IEC 61882 [START_REF]2001 Hazard and Operability Studies (HAZOP Studies) -Application Guide[END_REF] defines what a Hazard and Operability (HAZOP) study is and a process to perform a HAZOP study. HAZOP aims at identifying potential hazards and operability problems. A hazard is defined as the potential source of "physical injury or damage to the health of people or damage to property or the environment" [START_REF]2001 Hazard and Operability Studies (HAZOP Studies) -Application Guide[END_REF] and an operability problem is any deviation from the intended behavior of the system that leads to non-conformance with its (functional) requirements. During a HAZOP study small parts of a system are analyzed in isolation. To systematically identify the potential hazards or operability problems of these parts, HAZOP proposes the eleven guide words NO, MORE, LESS, AS WELL AS, PART OF, REVERSE, OTHER THAN, EARLY, LATE, BEFORE, and AFTER. These guide words are interpreted in the context of the behavioral characteristics of the part under consideration and lead to deviations of the intended behavior. The derived deviations for a part are documented together with the possible causes of the described situation, its consequences, and safeguards that shall prevent the occurrence of this situation, or reduce the consequences the deviation may have in a template.

In this paper, we adapt HAZOP to be used in the context of a privacy threat analysis. Next, we introduce the Problem-based Privacy Analysis (ProPAn) method that can be used as a starting point for a PRIOP study.

Problem-based Privacy Analysis To perform a privacy threat analysis, first, the system, consisting of the machine (software to be developed) and the environment it shall be integrated in (cf. [START_REF] Jackson | Problem Frames. Analyzing and structuring software development problems[END_REF]), has to be analyzed. To be more precise, it has to be known 1) which PI of which data subjects is processed by the machine, 2) how is this PI collected by the machine, 3) where and how is the PI stored, and 4) to which other entities the machine provides the PI it processes.

In this paper, we demonstrate PRIOP based on inputs provided by the Problem-based Privacy Analysis (ProPAn). ProPAn is a systematic and toolsupported2 method to perform a privacy analysis starting with a set of functional requirements. The functional requirements represent a decomposition of the overall problem of building the machine and they have to be modeled as problem diagrams following Jackson's problem frame approach [START_REF] Jackson | Problem Frames. Analyzing and structuring software development problems[END_REF]. Figure 1 shows the problem diagram for requirement R3 of the EHS example.

It shows on the left the machine EHS (box on the left) and its interfaces (lines between the boxes) to the environment (boxes in the middle). The environment of the machine consists of domains. Jackson distinguishes three types of domains. Biddable domains (B) are usually people, lexical domains (X) are physical representations of data, and causal domains (C) are objects that behave according to a given specification. The relevant environment for R3 consists of the lexical domains EHR (representing the electronic health records) and Invoice (representing the invoices for treatments that the patient's insurance contracts do not cover), the causal domain InsuranceApplication (which is the interface to the patients' insurances to perform the accounting of treatments patients received), and the biddable domain Doctor (who initiates the accounting).

On the right, the problem diagram shows the functional requirement R3 (dashed oval on the right) and its references to the environment. Jackson distinguishes two kinds of references from functional requirements to domains. First, a requirement can refer to (dashed line) an event, action, or state of a domain due to which the environment shall behave in the desired way. This desired behavior of the environment is expressed using the second kind of reference. That is, a requirement can constrain (dashed line with filled arrow head) events, actions, or states of a domain. R3 refers to the event that the Doctor initiates the accounting and to the EHRs of the involved patients, which contain the PI treatments, diagnoses, insurance number, and the patients billing information. Additionally, R3 constrains that the InsuranceApplication provides the feedback which treatments are covered by the patients' insurance contracts and that a corresponding invoice is created for treatments not beared by the patient's insurance.

ProPAn helps an analysis team that incorporates expertise in requirements engineering, privacy, and the application domain to 1) elicit privacy-relevant domain knowledge [START_REF] Meis | Problem-Based Consideration of Privacy-Relevant Domain Knowledge[END_REF], 2) identify the PI processed by the system and how it flows through it [START_REF] Meis | Supporting privacy impact assessments using problem-based privacy analysis[END_REF], and 3) derive the relevant privacy requirements for the machine [START_REF] Meis | Computer-aided identification and validation of privacy requirements[END_REF]. Figure 2 shows an excerpt of a graph that visualizes the flow of patient's PI due to the functional requirements. This graph is a result of ProPAn's steps to identify the PI that is processed by the system and how it is processed by the system. The gray highlighted part of the graph shows the information flows that were elicited due to R3. The flows outside of the gray part originate from other functional requirements or domain knowledge. The gray printed domains represent designed domains, and the white domains represent given domains. According to Jackson, designed domains are part of the machine and hence, part of the development problem. In contrast, given domains are the parts of the machine's environment that have to be considered as they are, i.e., their specified behavior is not under the control of the development team and cannot be changed. The graph shows that due to R3 treatmentCosts are collected from Doctors (flow from a given domain to a designed domain) and stored in an Invoice together with pa-tientBillingDetails and treatments (flows to a designed domain). In other words, during the privacy analysis with ProPAn, we identified and documented that an invoice contains the previously mentioned PI. Furthermore, insuranceNumber, treatments, and diagnoses flow to the InsuranceApplication (flow from a designed domain to a given domain), due to the machine because of R3. Due to requirement R4 and elicited privacy-relevant domain knowledge, the information provided to the InsuranceApplication flows further to InsuranceEmployees and the patientBillingDetails and treatmentCosts are sent to a FinancialApplication (R4) and further to its employees to perform the billing.

The privacy requirements considered by ProPAn are based on the six protection goals for privacy proposed by Hansen et al. [START_REF] Hansen | Protection goals for privacy engineering[END_REF]. These consists of the classical

Categorization

Guide word consideration security requirements confidentiality (SC), integrity (SI), and availability (SA) and the privacy goals unlinkability, transparency, and intervenability. We refined the privacy goal unlinkability based on the work of Pfitzmann and Hansen [START_REF] Pfitzmann | A terminology for talking about privacy by data minimization: Anonymity, unlinkability, undetectability, unobservability, pseudonymity, and identity management[END_REF] to the privacy requirements anonymity (UA), pseudonymity (UP), undetectability (UU), and data unlinkability (UD). The privacy goal transparency was refined by us in [START_REF] Meis | A taxonomy of requirements for the privacy goal transparency[END_REF] into information requirements for the collection (TC), storage (TS), and flow (TF) of PI, and informing about exceptional cases (TE) concerning the processing of PI. In [START_REF] Meis | Understanding the privacy goal intervenability[END_REF] we refined the privacy goal intervenability into intervention requirements for data subjects (ID) and authorities (IA).

In this paper, we consider the artifacts produced by ProPAn as input for PRIOP, but any method supporting points 1)-3) (mentioned above) can be used.

Privacy and Operability Studies

PRIOP aims at a systematic privacy and operability analysis of a software project. Figure 3 visualizes the central steps (arrows) of a PRIOP study and the created artifacts (boxes). First, the software project has to be decomposed into subproblems. PRIOP does not prescribe how the decomposition is achieved. For example, Jackson's problem frame approach can be used to derive the project's subproblems. For each of these subproblems, we create a table for further analysis. This table should contain a short summary of the subproblem that is considered and should mention who is involved in the PRIOP study of the subproblem. Then each subproblem is categorized based on its functionality, as discussed later in this section. For each identified category of the subproblem a block is added to the subproblem's table. Finally, the PRIOP guide words have to be considered for every combination of subproblem and category. The consideration of a guide word results in a row in the block of the considered category in the table of the considered subproblem. In the following, we provide more details on the categorization of subproblems and the consideration of guide words.

PRIOP Operation Categories During the analysis of the identified subproblems, we distinguish four categories of how PI can be processed by the machine. These categories are collection, storage, flow, and deduction of PI. An operation is in the category collection, if it describes that information is collected by the machine from a given domain. Operations in the category storage are concerned with the storage of PI at designed domains. If an operation causes a flow of PI from the machine to a given domain, then it is in the category flow. Operations in the category deduction are concerned with the deduction or computation of PI based on other information.

An operation can be in none (i.e., it does not process PI) or multiple of these categories, depending on the characteristics of the operation. This differentiation of operation categories helps to systematically assess the characteristics of a subproblem in order to identify privacy threats that it possibly causes. To refer to all of these categories simultaneously, we will use the term processing.

If the PRIOP study is performed based on ProPAn, then we consider the functional requirements as subproblems and can perform their categorization automatically based on the artifacts created by this method. For requirement R3 of the EHS example, we can see from Figure 2 that it is concerned with the collection of PI from Doctors, the storage of PI at the domain Invoice, and sending PI to the InsuranceApplication. Figure 2 also shows which PI is collected, stored, and flows. Furthermore, the ProPAn model documents (not shown in this paper) that the PI treatmentCosts is derived from the PI treatments, diagnoses, and insuranceContracts by doctors based on the feedback provided by the insurance application. Hence, R3 belongs to all operation categories.

PRIOP Guide Words

We consider all HAZOP guide words as useful to identify privacy threats, because these guide words describe in general the deviations that can occur in all kinds of operations a subproblem may be concerned with. We add one additional guide word, namely INCORRECT. This guide word shall cover the cases in which operations are performed incorrectly or with incorrect information as an input. Table 1 shows all PRIOP guide words and our deviation patterns for the four previously introduced operation categories. If these deviation patterns are used for a concrete subproblem, then the terms in angle brackets (< >) have to be instantiated for the subproblem (cf. column deviations in Table 2). The term <PI> is instantiated with the PI that is collected/stored/flown/deduced due to the subproblem. If a subproblem is in the category flow, then the term <target> has to be instantiated with the given domains to which the PI flows. Furthermore, there are some terms in italics. While the other terms can be instantiated based on the combination of operation category and subproblem, the italic terms have to be instantiated under consideration of the concrete deviations the guide words imply. The terms <other PI> and <additional PI> have to be instantiated with the PI that is considered to be unintendedly collected/stored/flown/deduced. <other domain>, <other target>, and <additional target> have to be instantiated with the domains to which information flows unintendedly.

The PRIOP Template The previously introduced guide words shall help to identify deviations of the intended behavior of the operations a subproblem is concerned with. These deviations can lead to violations of privacy requirements and the subproblem's operability. In the case that such an identified deviation leads to a violation of a privacy requirement, the deviation is a privacy threat. We developed a template that is based on the templates proposed to be used in HAZOP studies in [START_REF]2001 Hazard and Operability Studies (HAZOP Studies) -Application Guide[END_REF], but enhanced with additional fields to allow to elicit and document attributes that are needed for a later risk evaluation of the identified privacy threats. An excerpt of the PRIOP template instance for R3 of the EHS example is shown in Table 2. We omit the general information about the subproblem and the people involved in the PRIOP study. In Table 2, we see for each operation category a block (introduced with a row with black background). We selected one or two guide words for each operation category block to illustrate how the proposed template could be filled. For each operation category it is documented from, at, to, or by which domain which PI is collected, stored, flows, or is deduced, respectively. The columns are separated into three areas.

The first two columns show the considered guide word for the row and the deviations it can lead to for the operation category and the considered subproblem. Our deviation patterns shown in Table 1 can be used as starting point for the derivation of the deviations implied by a guide word. The terms <PI> and <target> can be instantiated with the corresponding information provided in the operation category block. Nevertheless, the deviation pattern instances need to be modified to fit into the context of the subproblem. The deviations possibly represent privacy threats or operability issues.

The second area consists of the third and fourth column. In this area, the analysis team has to document the identified causes that possibly lead to the deviations. Additionally, the likelihood of each cause shall be documented. The analysis team should agree on a common likelihood scale, be it qualitative or quantitative. A common scale will make it easier to homogeneously evaluate the risks implied by the identified privacy threats.

The third area consists of the last three columns. This area is concerned with the consequences the deviations may have on the privacy requirements or the operability of the subproblem. The consequences are first documented as free text, then the harmed privacy requirements are explicitly listed, and it is documented to which degree the described consequences impact the listed privacy requirements. Similar to the likelihood scale, the analysis team has also to agree on a consequence scale.

If the analysis team identified possible causes for a guide word and consequences that harm privacy requirements, then the deviation represents a privacy threat. Whether and how this threat has to be further assessed is in most cases determined using a risk matrix that defines which combinations of likelihood and consequence of a threat are acceptable and which are not. Our template already provides this information such that a risk matrix can easily be filled based on an instantiated template.

We only discuss the last row of the template instance for R3 in Table 2. The row is concerned with the deduction of the treatment costs for treatments not covered by the patients' insurance contracts which is performed by doctors based on the result of the accounting provided by the insurance application. The deviation that is derived for the guide word REVERSE is that the patient's PI healthInsurances, diagnoses, and treatments can be deduced from treatment-Costs. This deviation could be possible if the treatment costs are observed over a longer time, e.g., because specific diagnosed illnesses could imply a series of treatments that lead to specific treatment costs allowing to conclude from the treatment costs the diagnosed illness and received treatments. The analysis team decided that this is rarely possible. As a consequence the financial employees who are able to observe the treatment costs over a longer time might be able to deduce the treatments, diagnoses, or insuranceContracts of specific patients. The analysis team identified that this consequence harms a confidentiality requirement saying that the deducable PI shall not be disclosed to financial employees. Furthermore, the transparency requirements that are concerned with informing the patient about the flow of and exceptional cases for the deducable PI and the related intervenability requirements for the patient and authorities are harmed. From the documented consequence a major impact on all listed privacy requirements is expected.

The shown template can be enriched with further columns. For example, it can be helpful to provide additional columns to document rationales, e.g., why a specific likelihood was selected for a possible cause, why a possible cause has a documented consequence, or why a consequence impacts the stated privacy requirements in the defined way. Furthermore, already existing safeguards or possible treatments could be documented that shall either reduce the likelihood of a possible cause or the consequence on a privacy requirement.

Relation of Guide Words to Privacy Requirements If the taxonomy of privacy requirements used by ProPAn (see Section 2) is used, we can provide additional support to instantiate the template. Based on the deviation patterns (see Table 1), we identified the privacy requirements that are expected to be harmed by a deviation. Table 3 shows the relations that we identified. An "X" in the table means that a deviation implied by the guide word for the operation category, could harm the respective privacy requirement. If a cell is empty or a privacy requirement is not mentioned, then we do not expect a violation of this privacy requirement for deviations implied by the respective guide word and operation category. In Table 3, we use the abbreviations for the privacy requirements that were introduced in Section 2 and we introduce three groups (Gn) of privacy requirements that share the combinations of guide words and operation categories for which they are relevant. This mapping of combinations of guide words and operation categories to privacy requirements shall help to identify the privacy requirements that are harmed by an identified deviation, but it could also serve as a starting point to elicit scenarios that violate the privacy requirements under consideration of the guide word and operation category.

We identified that for all combinations of guide words and operation categories the privacy requirements integrity (SI), availability (SA), exceptional information (TE), data subject intervention (ID), and authority intervention (IA), which all belong to group G1, might be harmed. This is, because every change in the behavior of an operation could damage the integrity and availability of the processed information, and every change of the way that the PI is processed by the machine could lead to exceptional cases about which the data subject has to be informed and that could violate intervention options the data subject or authorities have. Additionally, all modifications of how PI is collected, stored, or flows can lead to a violation of the transparency requirements collection, storage, and flow information, respectively. A change in the deduction of information might affect collection, storage, and flow information requirements (G2).

Group G3 consists of the privacy requirements confidentiality (SC), undetectability (UU), anonymity (UA), data unlinkability (UD), and pseudonymity (UP). These requirements might be relevant for the guide words MORE, AS WELL AS, OTHER THAN, EARLY, LATE, BEFORE, and AFTER in all operation categories, because the guide words imply either that more, additional or other information is processed by the machine, or in a different order, earlier, or later as expected, which could lead to a violation of these requirements. Note that for the guide words MORE, EARLY, LATE, BEFORE, and AFTER, the requirements about the PI that is processed are affected. In contrast, for the guide words AS WELL AS and OTHER THAN, the requirements about the additional or other PI that is processed in addition to or instead of the PI that originally should be processed could be harmed. The guide words NO, LESS, PART OF, and INCORRECT are not implying a violation of the privacy requirements in group G3, because they only concern that fewer or incorrect PI is processed, which does not harm the privacy requirements contained in G3. The guide word REVERSE is interpreted differently depending on the operation category (cf. Table 1). Hence, for the categories collection, flow, and deduction it might harm the requirements in G3, but for the category storage it does not.

The transparency requirement flow information (TF) might be harmed by deviations for the guide words REVERSE and OTHER THAN in the operation category collection, because they possibly imply a flow from the machine to another domain that is not intended. Similarly, collection information requirements (TC) might be harmed by deviations for the guide word REVERSE in the operation category flow. This is, because these scenarios would consider that instead of sending information to other domains, the machine would receive (collect) this or even other information which might be unintended.

Discussion The procedure described in [START_REF]2001 Hazard and Operability Studies (HAZOP Studies) -Application Guide[END_REF] to perform a HAZOP study stresses that for an analysis, the team has to carefully select the guide words that are considered for the system under consideration. Similarly, it can be the case that only a subset of the proposed guide words is relevant for a PRIOP study of a specific software project and that even additional guide words are identified as important. Hence, we do not claim that our selection of guide words represents a complete set of guide words relevant for the identification of privacy threats of a software project, but expect that it provides a good foundation.

Similarly, the operation categories could be extended. For example, Gürses [START_REF] Gürses | Multilateral Privacy Requirements Analysis in Online Social Network Services[END_REF] mentions that information can be collected, used, processed, distributed, or deleted. Collection and distribution (flow to other domains) are covered by our proposed categories. Usage contains from our point-of-view deduction and storage, but other kinds of usage might be identified for a concrete system as additional operation categories. Processing is considered by us as a high-level term describing that something is done with the PI, be that collection, storage, etc. Deletion is an additional category that is worth to analyze in future work, because it is only partly covered by PRIOP. The HAZOP standard does not categorize operations in a way that we propose in this work, but we think that making these operation categories explicit can help analysts to identify scenarios that lead to a harm of privacy requirements. Nevertheless, it can also be valuable to consider the guide words for a given subproblem without considering the operation categories, because this could prevent that the scope of the considered deviations is unnecessarily limited to the operation categories.

Anyway, no method for the identification of any kind of threats can guarantee to elicit a complete set of relevant threats [START_REF] Young | An integrated approach to safety and security based on systems theory[END_REF]. Nevertheless, we think that our proposed systematic analysis will help analysts to identify, evaluate, and document the privacy threats relevant for their software projects.

An important point that always needs to be assessed critically is the scaleability of a proposed analysis method. If we perform a PRIOP study, then we have to fill in a template for every subproblem. For each operation category a subproblem is assigned to, we have to consider the 12 guide words. That means that in the worst case, we have to fill in 48 rows of the proposed template for each subproblem. Our observation is that this maximum is rarely reached. If it is reached, this is an indicator that the subproblem could be further decomposed into simpler subproblems, because it includes collection, storage, flow, and deduction of PI. Overall, we expect that the effort that has to be spent to perform a PRIOP study scales linearly with the complexity of the software project. The central attributes describing the complexity of the software project for a PRIOP study are the number of subproblems, data subjects, and PI that shall be processed by the machine. For the EHS example, we filled out 168 rows in total. This took us 28 hours in total and 10 minutes per row in average. Limitations of PRIOP are that 1) the analysis of the subproblems in isolation may not be sufficient if threats arise from the combination of different functionalities, 2) the analysis is limited to the documented subproblems and hence, PRIOP will not help in identifying privacy threats if subproblems are missing or lack important details, and 3) the success of a PRIOP study depends on the analysis team. To address limitation 3), we encourage that the analysis team has to incorporate expertise in requirements engineering, privacy, and the application domain.

Related Work

Deng et al. [START_REF] Deng | A privacy threat analysis framework: supporting the elicitation and fulfillment of privacy requirements[END_REF] propose a privacy threat analysis framework called LINDDUN. LINDDUN considers the high-level privacy threats linkability, identifiability, non-repudiation, detectability, information disclosure, content unawareness, and policy/consent noncompliance, which are negations of popular privacy goals. For the considered system, a DFD (data flow diagram) is created. For each combination of privacy threat and DFD element kind, a threat graph is provided. These are used to derive the possible concrete privacy threats that have to be handled. Based on the high-level privacy threats, the authors also suggest PETs (privacy enhancing technologies) that shall help to mitigate the concrete threats. In comparison to our work, the threat graphs of LINDDUN provide more detailed information that may help to identify whether a high-level privacy threat is relevant or not. But it is possible that the usage of these threat graphs unnecessarily limits the scope of the privacy threat analysis. In future work, we want to elaborate how LINDDUN and PRIOP could be combined to provide better support for the identification of privacy threats.

Several authors investigated the needs of (D)PIAs and methodologies that can be followed in order to perform a (D)PIA. Wright [START_REF] Wright | The state of the art in privacy impact assessment[END_REF] gives an overview of the state of the art in PIA. Oetzel and Spiekermann [START_REF] Oetzel | A systematic methodology for privacy impact assessments: A design science approach[END_REF] describe a methodology to support a complete process for a PIA, and Bieker et al. [START_REF] Bieker | A process for data protection impact assessment under the european general data protection regulation[END_REF] describe a methodology for a DPIA under the EU General Data Protection Regulation. The proposed methodologies describe which steps have to be performed in which order to perform a (D)PIA, but they do not describe concrete techniques that can be used to systematically identify privacy threats. PRIOP can be used to realize the threat identification and risk evaluation steps of the proposed methods. Alnemr et al. [START_REF] Alnemr | A data protection impact assessment methodology for cloud[END_REF] propose a DPIA methodology for clouds. They support the identification of privacy threats based on an exhaustive questionnaire. This questionnaire is complementary to PRIOP, and we want to investigate in future work how the questionnaire can be integrated into PRIOP.

Conclusions

In this paper, we present with PRIOP a systematic method to identify and document privacy threats and operability issues of software projects. During a PRIOP study, possible deviations of the software project's subproblems are examined under consideration of the four operation categories collection, storage, flow, and deduction. The deviations of a subproblem in the context of the relevant operation categories are derived using the twelve proposed PRIOP guide words. Deviation patterns are provided by PRIOP for all combinations of guide words and operation categories to support an analysis team. The identified deviations for the guide words then have to be further analyzed for possible causes and consequences they might have on the privacy requirements of the software project or the operability of the subproblem. To further support the execution of a PRIOP study, we provide a mapping that shows which privacy requirements could be harmed by a deviation for a combination of guide word and operation category. The documentation created using PRIOP can be used to further assess the risks implied by the identified privacy threats. We illustrated PRIOP using an EHS example and artifacts produced with the ProPAn method.

In future research, we will integrate PRIOP into the ProPAn tool to benefit from the artifacts created using the ProPAn method. Furthermore, we want to investigate how generic threats, e.g., in the form of threat patterns as introduced by Uzunov and Fernandez [START_REF] Uzunov | An extensible pattern-based library and taxonomy of security threats for distributed systems[END_REF] for security, can be related to the operation categories and guide words to further assist the identification of privacy threats. The evaluation of PRIOP using a real case study is also future work.

Fig. 1 .

 1 Fig. 1. Problem diagram for functional requirement R3

Fig. 3 .

 3 Fig. 3. Steps and artifacts of a PRIOP study

Table 1 .

 1 Deviation patterns for the all combinations of proposed guide words and operation categories

	Guide word	Collection	Deviation patterns for operation category Storage Flow	Deduction
	NO	<PI> is not collected	<PI> is not stored	<PI> does not flow to	<PI> is not deduced
				<target>.
	MORE	More <PI> is collected	More <PI> is stored	More <PI> flows to	More <PI> is deduced
		than intended, including collection of <PI> with additional methods, with higher linkability, in higher amount, or with higher availability.	than intended, including storage of <PI> with higher linkability, in higher amount, with higher availability, or with longer duration.	<target> than intended, including flow of <PI> with higher linkability, in higher amount, or with higher availability.	than necessary, including deduction of <PI> with higher linkability, in higher amount, or with higher availability.
	LESS	Less <PI> is collected	Less <PI> is stored than	Less <PI> flows to	Less <PI> is deduced
		than intended, including collection of <PI> with less methods, with lower linkability, in lower amount, or with lower availability.	intended, including storage of <PI> with lower linkability, in lower amount, with lower availability, or with shorter duration.	<target> than intended, including flow of <PI> with lower linkability, in lower amount, or with lower availability.	than necessary, including deduction of <PI> with lower linkability, in lower amount, or with lower availability.
	AS WELL AS	In addition to <PI>	In addition to <PI>	In addition to <PI>	In addition to <PI>
		<additional PI> is	<additional PI> is	<additional PI>	<additional PI> is
		collected or in addition to collect the <PI>. <other domains> the software-to-be	stored.	target>. <additional or<PI> flows to an flows to <target>,	deduced.
	PART OF	Only a part of <PI> is	Only a part of <PI> is	Only a part of <PI>	Only a part of <PI> is
		collected.	stored.	flows to <target>, or	deduced.
				<PI> flows to fewer
				targets.
	INCORRECT	The collected <PI> is	The stored <PI> is	The <PI> flowing to	The deduced <PI> is
		incorrect.	incorrect.	<target> is incorrect.	incorrect.
	REVERSE	<PI> flows from	<PI> is deleted.	<PI> or <other PI>	<original PI> is or
		machine to source of collection.		flows from <target> to the machine.	can be deduced from <PI>.
	OTHER THAN	<other PI> is	<other PI> is stored	<other PI> flows to	<other PI> is or can
		collected instead of <PI>.	instead of <PI>.	<target> instead of <PI> or <PI> flows to	be deduced instead of <PI>.
				<other target>.
	EARLY	<PI> is collected earlier	<PI> is stored earlier	<PI> flows earlier than	<PI> is deduced earlier
		than intended relative to clock time.	than intended relative to clock time.	intended to <target> relative to clock time.	than intended relative to clock time.
	LATE	<PI> is collected later	<PI> is stored later than	<PI> flows later than	<PI> is deduced later
		than intended relative to clock time.	intended relative to clock time.	intended to <target> relative to clock time.	than intended relative to clock time.
	BEFORE	<PI> is collected before	<PI> is stored before	<PI> flows before	<PI> is or can be
		another prior operation. E.g., collection before gaining consent.	another prior subsequent operation. E.g., storing before gaining consent, or before anonymization.	another prior operation to <target>. E.g., sending before gaining consent, or before anonymization.	deduced before another prior operation. E.g., deduction before gaining consent, or before anonymization.
	AFTER	<PI> is collected after	<PI> is stored after	<PI> flows after another	<PI> is or can be
		another subsequent	another subsequent	subsequent operation to	deduced after another
		operation. E.g., collection of up-to-date information after it was needed or the data subject withdrew consent.	operation. E.g., storage of after another operation would have needed <PI>, or data subject has withdrawn consent.	<target>. E.g., operation on <target> is performed before the up-to-date <PI> was provided, or data subject has withdrawn consent.	subsequent operation. E.g., deduction after another operation would have needed <PI>, or data subject has withdrawn consent.

Table 2 .

 2 Excerpt of the instantiated template for functional requirement R3

	Impact			-			1) Major	2), 3), and 4)	Moderate		Moderate									i) Major												Major					Major
	Likelihood Consequences Harmed Privacy	Requirements	PI: treatmentCosts	Unlikely -Doctor will not get paid and Patients	not billed.		Unlikely Patients will get too high bills or are 1) SI, 2) TE, 3) ID, and 4)	billed for treatments that they did not IA for treatmentCosts	receive.	PI: treatments, patientBillingDetails, treatmentCosts	Unlikely Patients will get too high or too low 1) SI, 2) TS, 3) TE, 4) ID,	bills. and 5) IA for treatments,	patientBillingDetails,	treatmentCosts					PI: insuranceNumber, diagnoses, treatments	a) Unlikely i) If too few diagnoses are transmitted i) 1) SI, 2) TF, 3) TE, 4) ID,	b) Possible to the InsuranceApplication or if and 5) IA for treatments,	diagnoses and insuranceNumber are diagnoses and	not linkable to the treatments, then insuranceNumber	the Insurance Application is not able ii) -	to perform the accounting. This will	result in higher bills.	ii) If treatments that are not beared by	the Patient's insurance contract are	not transmitted to the	InsuranceApplication, then no invoice	will be created for them.	a) Unlikely Patients will get too high bills or are i) 1) SI, 2) TF, 3) TE, 4) ID,	b) Rare billed for treatments that they did not and 5) IA for treatments,	receive. diagnoses and	insuranceNumber	PI: treatmentCosts	Rare Financial employees that are able to 1) SC against Financial	observe the treatment costs over a Employees, 2) TF, 3) TE,	longer time might be able to deduce 4) ID, and 5) IA for	the treatments, diagnoses, or treatments, diagnoses,	insuranceContracts of specific and insuranceContracts.	Patients.
	Possible causes		from: Doctor	Doctor forgets to enter the costs of	his/her treatments or does not save	the changes made.	Doctors incidentially enter costs for	treatments not performed or too high	treatment costs	at: Invoice	a) Doctor explicitly initiates the	creation of an invoice without	knowing whether the concerned	treatments are beared by the	Patient's insurance contract	b) A software error causes the	creation of the invoice without before	having the necessary information	to: InsuranceApplication	a) Software error	b) Insurance Application is temporary	unreachable										a) Software error	b) Incorrect data in EHR			by: Doctor and InsuranceApplication	TreatmentCosts may allow to deduce	the treatments performed, diagnoses,	or insuranceContracts if observed	over a longer time especially if to	some extent the date of deduction is	known.
	Deviations Guide	word	Operation category: Collection	NO treatmentCosts are not collected			MORE Higher treatmentCosts are collected	than intended.		Operation category: Storage	BEFORE treatments, patientBillingDetails, and	treatmentCosts are stored in Invoice	before it is known which treatments	are beared by the Patient's insurance	contract.				Operation category: Flow	LESS Less diagnoses and treatments flow	than the patients received, or it is not	possible for the insurance application	to link the diagnoses and treatments	to the Patient's insuranceNumber.								INCOR-The insuranceNumber, diagnoses, or	RECT treatments flowing to	InsuranceApplication are incorrect.		Operation category: Deduction	RE-healthInsurances, diagnoses, and	VERSE treatments can be deduced from	treatmentCosts.

http://www.uml4pf.org/ext-propan