Olgierd Pieczul
email: olgierdp@ie.ibm.com

Simon N Foley
email: s.foley@cs.ucc.ie

Runtime detection of zero-day vulnerability exploits in contemporary software systems

It is argued that runtime verification techniques can be used to identify unknown application security vulnerabilities that are a consequence of unexpected execution paths in software. A methodology is proposed that can be used to build a model of expected application execution paths during the software development cycle. This model is used at runtime to detect exploitation of unknown security vulnerabilities using anomaly detection style techniques. The approach is evaluated by considering its effectiveness in identifying 19 vulnerabilities across 26 versions of Apache Struts over a 5 year period.

Introduction

Contemporary software is routinely constructed from a myriad of components and frameworks. With the emphasis on rapid construction and reuse comes a tacit acceptance that it is neither practical nor expected that a programmer should fully understand the minutiae of every component included with their application. As a consequence, its not unusual for the programmer to focus on understanding their application at the level of the business logic, while hoping that abstraction neatly deals with the complex interoperation of its underlying components. However, these underlying components often require particular configuration and usage patterns in order to operate securely. These component requirements may be ignored by the programmer, through ignorance or as a result of coding error in the application. A programmer, while focused on using an external software component to implement one task may overlook that the application is implicitly enabled to perform another, unexpected task. Flaws can also occur within the components themselves, for similar reasons; the component developer may not have anticipated all possible use-cases. A consequence of what can be described as "dark code" [START_REF] Holzmann | Code inflation[END_REF] or "the dark side of the code" [START_REF] Pieczul | The dark side of the code[END_REF], is that while the programmer expects certain program execution paths, other, unexpected paths may be possible and give rise to a security vulnerability.

The software industry's approach to software vulnerabilities includes security quality assurance processes such as code reviews, static analysis and penetration testing. However, it is rarely possible to cover the subtleties of all of the numerous components and their inter-operation. In addition, during application development, third-party components are not routinely reviewed or tested for vulnerabilities. Some of the security vulnerabilities that gained notoriety in recent years, such as Heartbleed and Shellshock, were deployed for many years before they were discovered, despite being used by large number of consumers.

Our position is that many software vulnerabilities can be attributed to programming errors that enable unexpected software behavior. Runtime verification [START_REF] Delgado | A taxonomy and catalog of runtime softwarefault monitoring tools[END_REF] of software execution against a specified expected behavior can help identify unexpected behavior in the software. For small applications, limited requirements on expected behavior can be specified a priori, for example as a temporal proposition [START_REF] Delgado | A taxonomy and catalog of runtime softwarefault monitoring tools[END_REF]; however, this does not scale when the requirement is to constrain emergent behavior across large complex systems of interoperating components. An alternative strategy, used by anomaly detection techniques [START_REF] Creech | A semantic approach to host-based intrusion detection systems using contiguousand discontiguous system call patterns[END_REF][START_REF] Forrest | The evolution of system-call monitoring[END_REF][START_REF] Forrest | A sense of self for unix processes[END_REF][START_REF] Raman | JaSPIn: JavaScript based Anomaly Detection of Cross-site scripting attacks[END_REF], is to learn a behavioral reference profile from system logs of past/normal behavior and use this profile at runtime to check behavior. However, practical application of this approach has tended to generate profiles with limited expressiveness [START_REF] Tan | Undermining an anomaly-based intrusion detection system using common exploits[END_REF] for relatively small and uncomplicated software components [START_REF] Patcha | An overview of anomaly detection techniques: Existing solutions and latest technological trends[END_REF], such as sendmail or lpd, observed though system calls [START_REF] Forrest | A sense of self for unix processes[END_REF]. The practical challenges of applying these techniques to contemporary enterprise software, such as dealing with scale and alignment with software release processes have not been considered.

The contributions of the paper are as follows. The anomaly-detection model proposed in [START_REF] Pieczul | Discovering emergent norms in security logs[END_REF] is extended to incorporate the notion of scope which provides an effective way of dealing with scale in contemporary application systems. Based on this model, an implementation framework is developed whereby existing software testing techniques are adapted to generate profiles of expected behavior that can be used for runtime verification. The approach has been evaluated by considering its effectiveness in identifying code vulnerabilities across the 26 versions of Apache Struts over the past 5 years. Of significance is the result that anomaly-detection/run-time verification techniques would have been quite effective in identifying exploits of these zero-day vulnerabilities, prior to their discovery. To the best of our knowledge this is the first successful construction of profiles used to detect vulnerabilities in large-scale enterprise software.

The paper is organized as follows. Section 2 uses an example of contemporary software development-a microblog application implemented with Struts-to illustrate how easy it is for the programmer to unwittingly introduce a programming flaw/security vulnerability. Section 3 reviews and considers the challenges of applying techniques for anomaly detection to contemporary software systems and describes our proposed approach. Section 4 discusses the evaluation of the approach and Section 5 provides the conclusion.

Contemporary software development

Contemporary software is implemented using a variety of high-level programming languages, software frameworks and third party components. While applicationlevel code may appear straightforward, enabling rapid and low-cost application development, its underlying system is a complex arrangement of inter-dependent Fig. 1. Micro-blog application code and trace software whose behavior can be difficult for a programmer to fully comprehend. There are a great many examples whereby misunderstanding of contemporary software systems leads to errors in the application code that can be exploited as security vulnerabilities. We consider this problem using a simple example that runs through the paper.

A micro-blogging application

A micro-blogging social networking application provides an online facility for users to post short text messages and share them with others. The application provides a web interface and REST API for integration with various types of clients, including desktop browsers and mobile devices. The application is built using Apache Struts, a popular Model-View-Controller framework for J2EE. Struts abstracts application logic from HTTP request processing flow by encapsulating it into objects called actions. For example, Figure 1 shows how the action responsible for posting a new micro-blog message is implemented in just a few lines of code. Actions can be mapped to specific URL paths, such as /api/ post. Struts can separately handle routine tasks, including parameter validation, authentication, session handling and CSRF protection, before passing parameter values to the action as associated by their setters. This facilitates the separation of business logic from HTTP processing concerns. Thus, rather than dealing with low-level operations such as accessing parameters by name, the developer need only implement a public setter. This makes the code re-usable, easier to maintain, document and test.

Application behavior can be traced as a sequence of the underlying Java method calls. For example, invoking the application using /api/post?message =Hello results in the included trace fragment. Each method includes a reference to its calling class. In the above, Struts creates an instance of PostAction, which is provided with its session and parameter data and is in turn executed, and the return value sent as response.

An unexpected vulnerability

While the implementation of the micro-blog post action appears straightforward, it can be easy for a developer to overlook subtleties in the interoperation between the application code and the underlying Struts framework. The code in Figure 1 contains one such oversight, that results from how the getUser method operates with Struts, and results in a security vulnerability. Suppose that this method was not intended to be a part of the action's interface, rather, it is implemented by the programmer as a means to provide convenient access to the user object from the session. The method returns a User object, which is a container for various user attributes, such as name and identifier and accessor methods. The combination of a public getter for the User object and public setters for the object makes it possible to manipulate user information request parameters. For example, an attacker Alice can invoke /api/post?message=Hello&user.id= frank in order to modify the user-id stored in the session to Frank and submit a message on Frank's behalf. A fragment of the execution trace in this case is: The Struts ParametersInterceptor parses and interprets the user.id attribute as an instruction to set the id property of the user property. This results in a sequence of operations equivalent to getUser().setId("frank"). This oversight by the developer illustrates how easy it can be to program an unexpected execution path that compromises security. Accessing object fields though a chain of getters and setters is a useful and documented feature of Struts. However, it can come as a surprise to a programmer, who is focused more on application-level development, that something as intuitive as implementing a getter for the current user results in a security vulnerability.

StrutsActionProxy: PostAction.
This problem is further highlighted by an almost identical vulnerability (CVE-2014-0094) that was found within Struts itself. Even though the consequences of exposing a public getter may have been clear to the Struts developers, it is easy to overlook the fact that every Java object (and therefore, every Struts action) also contains a getClass method. This results in an unintended exposure of an action's Class object, accessed through request parameters [START_REF] Ashraf | Analysis of recent struts vulnerabilities parameters and cookie interceptors, their impact and exploitation[END_REF]. Unfortunately, even after the problem was discovered, the implemented remedy was incomplete. The initial remedy black-listed the class parameter, however, it did not consider uppercase parameters (such as Class). Eventually, three more vulnerabilities were reported on incomplete remedies before the issue was believed addressed.

The micro-blogging application is one example of how programming error can result in a security vulnerability whose identification and prevention require an in-depth understanding of the underlying systems. However, given the complexity of contemporary systems, our position is that there will always be some aspect of the behavior that the programmer does not fully understand. For example, a study of developers by Oliveira et. al. [START_REF] Oliveira | It's the psychology stupid: How heuristics explain software vulnerabilities and how priming can illuminate developer's blind spots[END_REF] found that 53% of its participants knew about a particular coding vulnerability, however they did not correlate it with their own programming activity unless it was explicitly highlighted.

The use of anomaly detection in software execution has tended to focus on relatively small-scale homogeneous applications. However, the microblogging example illustrates that even very simple contemporary applications are in fact large systems of interconnected components. This section considers the key challenges encountered in applying anomaly detection techniques to contemporary application software. For reasons of space, but without loss of generality, the discussion is focussed on a common enterprise scenario of a web application/service built on a high-level software platform (Java) and an MVC framework (Struts).

Abstraction and scope

The activity of an application is observed as a sequence of events, such as a system log. These observations can be made at different levels of abstraction. For example, the micro-blogging application activity could be observed as a series of high-level user actions, such as posting or viewing a message. At a lower level of abstraction the actions can be observed as HTTP calls to the application server. Further lower levels describe the activity of the application code and its libraries, Java virtual machine, operating system, and so forth. The objective is to use the observed activity/log of the system to build a behavior reference model for use in anomaly detection. The challenge is determining a level of abstraction that enables anomalous execution paths to identify security vulnerabilities.

Security vulnerabilities occur at different levels of abstraction. Building reference models from observations of low-level interactions, such as operating system calls [START_REF] Creech | A semantic approach to host-based intrusion detection systems using contiguousand discontiguous system call patterns[END_REF][START_REF] Forrest | A sense of self for unix processes[END_REF][START_REF] Maggi | Detecting intrusions through system call sequence and argument analysis[END_REF] or Javascript [START_REF] Raman | JaSPIn: JavaScript based Anomaly Detection of Cross-site scripting attacks[END_REF] calls has been shown to be quite effective in detecting specific exploits on the application, such as buffer overflow vulnerabilities or cross-site scripting. However, the anomalous execution path in the micro-blog example cannot be detected at a level of abstraction comparable with [START_REF] Creech | A semantic approach to host-based intrusion detection systems using contiguousand discontiguous system call patterns[END_REF][START_REF] Forrest | A sense of self for unix processes[END_REF][START_REF] Maggi | Detecting intrusions through system call sequence and argument analysis[END_REF][START_REF] Raman | JaSPIn: JavaScript based Anomaly Detection of Cross-site scripting attacks[END_REF] as observing the sequence of calls to the database looks the same as valid behavior. While one may think of this anomaly as occurring at a higher level of abstraction, making observations based on HTTP requests is not sufficient in this case (although we note that there are other anomalies that can be characterized at this high-level). An analysis algorithm is proposed [START_REF] Pieczul | Discovering emergent norms in security logs[END_REF] that discovers a level of event abstraction that provides good anomaly-detection accuracy in transaction-like behaviors; it has proven quite effective in discriminating actions that remain static from the parameters that can change within a transaction.

In our experiments we found that observing (Java) application behavior in terms of its method calls and permission checks is the most effective in distinguishing the un-expected execution paths that lead to software vulnerabilities. However, in practice it is unrealistic to base anomaly detection on every underlying method call invoked as a consequence of executing an application. For example, the few lines of the microblog application Java code in Figure 1 generate tens of thousands of method calls. Therefore, the scope of observation is restricted to methods that belong to the specific component(s) of interest. Section 2 considers detection of vulnerabilities in the microblog application code and, therefore, restricts the observed scope to just those methods that belong to this application's Java package. In the experiment described in Section 4 the focus is on detecting vulnerabilities in the code that implements Apache Struts and therefore the scope is the 2400 methods defined within Struts packages.

Generating baseline activity

A baseline of observed activity must be obtained in order to build the behavior reference model for the software component. In order to be applicable to an enterprise environment, the method of establishing baseline activity has to be systematic, repeatable and aligned with application development/deployment lifecycle. Most importantly, the baseline generation must provide sufficient coverage for the normal/expected behavior of the application.

A common position is that a satisfactory baseline can be obtained as a recording of "normal" activity obtained though monitoring application operation in its target environment [START_REF] Forrest | The evolution of system-call monitoring[END_REF][START_REF] Forrest | A sense of self for unix processes[END_REF][START_REF] Raman | JaSPIn: JavaScript based Anomaly Detection of Cross-site scripting attacks[END_REF]. However this has practical limitations for enterprise software; in particular, the system must run over a period of time in order to generate a baseline with sufficient coverage. As the software industry adopts faster and more automated delivery approaches and software releases are as frequent as on daily basis, building a (normal) baseline based on observation of application in production is impractical. It would not only require the application to be active for a significant length of time without anomaly detection, it also may be impossible to obtain a sufficient baseline before it becomes obsolete.

We argue that is more appropriate to obtain the baseline and corresponding behavioral model during the software testing phase. In [START_REF] Pieczul | The dark side of the code[END_REF] it is suggested that unit tests might be used to generate the necessary system logs. Such a baseline is useful if the required scope and level of abstraction is consistent with the unit tests. Notwithstanding coverage, unit tests are typically constructed for a specific component, in isolation, and may only be suitable for relatively simple components or utility libraries An alternative approach is using functional and/or integration tests to also generate the baseline. Enterprise software is often subjected to extensive testing, usually highly automated and with controlled coverage. Traces resulting from such software tests may provide a reliable and systematic baseline model. However, as with unit tests, the functional test coverage is usually built only to a certain satisfactory level but is rarely complete. Additional coverage may be obtained using fuzz testing or application scanners that automatically exercise application in order to perform non-functional types of testing such as security or performance. The application scanners often use functional test execution as its starting point and then extend the coverage by exploring the application further in automated fashion.

In our experiments we found that using an application scanner allows the application system to be explored in a comprehensive, un-biased and repeatable manner and was the most effective for our task. The application scanner explores the application extensively to discover its structure, parameters, cookies and so forth, and also engages in a series of tests in an attempt to discover security vulnerabilities. While the scanner interacts via high-level requests to the application's external interfaces, a system log/trace of the consequent low-level actions is generated. If the scanner does not discover any security vulnerabilities then it is this log that provides the baseline of expected activity.

Behavioral reference model

A number of approaches can be used to infer a reference model of system behavior based on observed past activity, with varying complexity and precision. Statistical models [START_REF] Helman | Statistical foundations of audit trail analysis for the detection of computer misuse[END_REF] probe various system characteristics periodically to establish a baseline of metrics. Such models, while useful for simple intrusion detection, are not sufficiently rich to detect anomalous software execution. Sequence-based techniques have been designed to infer acceptable behavior/policies for anomaly detection [START_REF] Forrest | A sense of self for unix processes[END_REF][START_REF] Pieczul | Discovering emergent norms in security logs[END_REF] and process mining [START_REF] Van Der Aalst | Workflow mining: Discovering process models from event logs[END_REF]. Strengths and weaknesses of existing techniques have been studied [START_REF] Chandola | Anomaly detection for discrete sequences: A survey[END_REF][START_REF] Tan | Undermining an anomaly-based intrusion detection system using common exploits[END_REF] and expected properties of the reference model discussed [START_REF] Forrest | The evolution of system-call monitoring[END_REF][START_REF] Patcha | An overview of anomaly detection techniques: Existing solutions and latest technological trends[END_REF].

In many applications, activity may be contained in finite and separate units of work, where a sequence of operations that have clear beginning and end and, in general, follow some repeatable pattern is performed. Application behavior may contain multiple such transaction-like work units. For example, observing the micro-blogging application at a method call level, can be modeled as a collection of different kinds of transaction behaviors, such as posting a message or reading a message. Activity observations related to invocations of the same transaction are similar and activity related to invocations of different transactions substantially different. For this type of activity, rather than merging all transactions into a single set of short-range correlations [START_REF] Forrest | A sense of self for unix processes[END_REF], the reference model should be able to express units of activity separately from each other [START_REF] Pieczul | Discovering emergent norms in security logs[END_REF], thus at increased precision and reduced chance for mimicry attack [START_REF] Wagner | Mimicry attacks on host-based intrusion detection systems[END_REF].

Correlations between sequences of operations and their target values can be used to discover repeating patterns of behavior and [START_REF] Pieczul | Discovering emergent norms in security logs[END_REF] uses this to develop an automated technique to partition a system trace into a collection of behavioral norms. In our experiments we use these behavioral norms [START_REF] Pieczul | Discovering emergent norms in security logs[END_REF] to provide a reference model for anomaly detection. A collection of behavioral norms is generated, from baseline activity log of Java events, for a given scope and level of abstraction, each corresponding to a sequence of method calls parameterized by common target attributes. The approach is not unlike process mining [START_REF] Van Der Aalst | Workflow mining: Discovering process models from event logs[END_REF] or sequence call monitoring [START_REF] Forrest | The evolution of system-call monitoring[END_REF], however, behavioral norms can be used to provide a more precise model for system-level behavior.

Runtime verification and Application Integration

While traditional logging may be useful for retrospective detection of anomalous behavior, integrating runtime verification/anomaly detection with the application provides the ability to interrupt application execution before the anomalous operations are executed. Contemporary software platforms provide integration techniques such as Aspects that may be suitable for easy and non-disruptive enablement of execution monitoring and runtime verification.

The trapping and monitoring of application execution is implemented using Runtime Verifier, a customized Java Security Manager. Its purpose is to intercept permission check requests, such as the permission to open a file, connect to server or execute a process. According to its mode of operation, this manager may log the events, check that event execution is compliant with a norm-model provided, and if required, prevent the execution of anomalous event sequences.

While the Java Security Manager is a natural integration point for monitoring and controlling application execution, it is limited to those methods that have explicit permission checks such as input/output operations or security related activity such as access to cryptographic keys. An arbitrary off-the-shelf component might not have been programmed with its own permissions. In this case it is insufficient to rely on the Security Manager, and therefore, a Java Aspect was developed that can intercept specific method calls that match programmerspecified pattern. For example, in the case of the micro-blog application, the aspect was configured to intercept method calls that are provided by classes of the application. The aspect, on intercepting a method call, invokes the Security Manager, which in turn forwards the call details as an event to the Runtime Verifier runtime. This is depicted in Figure 2. Using a security manager in the implementation of the Runtime Verifier permits control over application execution. Permissions and identified method calls are checked prior to execution, and thus can be validated for compliance with the behavioral model at runtime, and prevented, if considered anomalous.

Experimental evaluation

The previous section outlined our implementation of Java application-level anomaly detection based on behavioral norms: checking that the actual execution of an application is compliant with a model of expected behavior that was generated during software development. Our thesis is that unknown security vulnerabilities in software components can be identified as runtime anomalies arising from unexpected execution paths.

Testing this thesis using a catalog of vulnerabilities hand-crafted for the purpose, may provide insight, but their design can be contrived/cherry-picked and is not an effective evaluation of whether the approach would work 'in the wild'. We therefore decided to test the thesis against an application that used a well-established and popular enterprise-scale software component that has a history of security vulnerabilities. In particular, the objective is to test whether vulnerabilities reported against earlier versions of software can be identified as anomalies, while those same anomalies are not reported against later versions of the software in which the corresponding vulnerability has been remedied.

In the following we considered the vulnerability history of 26 versions of Apache Struts over a five year period, starting with version 2.3.1, released in December 2011, to version 2.3.24.1, released in September 2015.

Experiment setup

Figure 3 outlines the key elements of the experimental setup. In order to evaluate Struts behavior in its typical environment, we developed a small Struts-based web application based on the micro-blogging program described in Section 2. The application makes conventional use of Struts, with a standard configuration including default interceptor stack and properties.

The application system is built automatically and the experiment is carried out separately for each version of Struts. Experiment characteristics, such as execution times and sizes, were comparable for the different versions of Struts. Experiments were orchestrated by Apache Maven and each iteration for a different Struts version comprised of two phases. In the first phase, a trace of the application system's execution is generated and from which the behavioral model is built for the given version of Struts. In the second phase the effectiveness of anomaly checking based on the generated behavioral model is verified.

Building behavioral models

A commercial application security scanner was run against the micro-blogging web application. The scanner configuration was standard and not tailored in any particular way for Struts. The same scanner configuration was repeatedly used against each deployment of the application with a different version of Struts. In each experiment, the scanner interacts with the micro-blogging web application, black-box testing an extensive collection of known vulnerabilities and misconfigurations. The Runtime Verifier was deployed with the application and used to build/check the behavior models in each experiment.

In our experiment, we consider the behavior of Struts, in terms of how it is used by the application. Therefore, the Runtime Verifier (Section 3.3) was configured to intercept all method calls within the Struts packages scope, that is org. apache.struts2. * ., corresponding to 460 distinct methods that are used in the context of the micro-blogging web application. A single scan experiment resulted in 8963 HTTP requests to the micro-blogging web application URLs with a range of inputs, taking 8 minutes to complete. In monitoring the execution of the application during this scan, the Runtime Verifier generates a 237 megabyte baseline trace containing 2.76 million Struts events (in scope), which is analyzed, and a set of behavioral norms is generated within approximately a further 5 seconds, running on a mid-range computer. The norms are sequences of method calls, such as ..., UrlHelper.encode, UrlRenderer UrlProvider.getAnchor, UrlRenderer UrlProvider.isPutInContext, ... , represented as tri-grams. Figure 4 plots an example of the number of distinct norms (transactional behavior patterns) generated, against the number of HTTP requests made by the scanner to the application deployed with Struts version 2.3.1. ¿From the start of the scan, as the number of requests increase, the number of norms resulting from the requests rapidly increase initially, and then appear to stabilize after a period. This graph suggests that our scan size of 8963 requests is adequate, after which no new norms are identified. Similar results were achieved for the other versions of Struts. Having generated a behavioral norm model for a given version of Struts and included it with the Runtime Verifier in the application deployment, the second phase of the experiment involves testing the effectiveness of using the model to detect vulnerabilities for that version of Struts.

Vulnerability tests

At the time of writing and based on the Common Vulnerabilities and Exposures (CVE) entries in the National Vulnerability Database, there are 19 vulnerabilities known to the general public for the 26 versions of Struts under study. For each vulnerability, the CVE advisory was studied alongside the vulnerable Struts code and the remediated version of the code, and an attack vector exploiting the vulnerability was developed. Of the 19 vulnerabilities, attacks for the 18 listed in Table 1 were developed; we could not find enough information to reproduce the vulnerability identified in CVE-2012-4386. For each vulnerability, we implemented an automated test case which attempts to exploit the vulnerability and verify that the exploitation was successful.

For example, CVE-2013-2115 is a vulnerability that allows a remote attacker to execute arbitrary OGNL code via a crafted request. It affects Struts JSP tags for rendering URLs. Using the tag is convenient and relieves the developer from having to manually map actions to URLs and passing parameters, thus further separating application logic from low-level details. In order to render a URL for a search action, including the current page's parameters, the developer can use: <s:url action="SearchAction" includeParams="all">. The tag is evaluated to /api/search?name=Frank. However, the code for processing the tag suffers from a security vulnerability. An attacker may add a request parameter by including OGNL code and that code will be evaluated when processing the tag. For example, the official security advisory for this vulnerability (CVE-2013-2115) describes that an attacker may append x=${@java.lang.Runtime@getRuntime ().exec('cmd')} to the page parameters. The OGNL code, enclosed in ${} is evaluated and custom Java code is executed by the application. However, the attack results in activity caused not only by an unexpected Struts execution path, but also by the injected code involved in creating a process and accessing a file binary. Rather than injecting malicious code that results in significantly different behavior, each test case attempts to inject benign code that simply sets a variable that can be subsequently checked to determine the success of the attack. The objective of each test is to generate the minimal behavior needed to explore the path of the attack, but it does not engage in subsequent behavior that might be easily recognized as anomalous in its own right. In this way the test case is intended to represent a worst-case scenario for anomaly detection.

In many cases developing the vulnerability test cases required some effort. A number of the vulnerabilities are not clearly described in their respective advisories and sometimes the details are intentionally undisclosed. For example, CVE-2013-4310 (discussed below in this paper) is described as allowing "to bypass security constraints under certain conditions". In order to prepare the test case for this vulnerability it was necessary to understand its nature though analysis of source changes of Struts.

Vulnerability test results.

2015-1831 ------------ --------------- --- ---+++ ------ 2014-7809 ++-++-++-++- ++-++-++-++-++- ++- ++---------- 2014-0116 +++ +++ +++ +++ +++ +++ +++ +++ +++ +++ --+ --+ --+ --+ 2014-0113 +++ +++ +++ +++ +++ +++ +++ +++ +++ --+ --+ --+ --+ --+ 2014-0112 +++ +++ +++ +++ +++ +++ +++ +++ +++ --- ------------ 2014-0094 +++ +++ +++ +++ +++ +++ +++ +++ --- --- ------------ 2013-4316 +++ +++ +++ +++ +++ +++ +++ ------ --- ------------ 2013-4310 ++-++-++-++- ++-++-++------- --- ------------ 2013-2251 +++ +++ +++ +++ +++ +++ --------- --- ------------ 2013-2248 +++ +++ +++ +++ +++ +++ --------- --- ------------ 2013-2135 +++ +++ +++ +++ +++ ------------ --- ------------ 2013-2134 +++ +++ +++ +++ +++ ------------ --- ------------ 2013-2115 +++ +++ +++ +++ --------------- --- ------------ 2013-1966 +++ +++ +++ -++ --------------- --- ------------ 2013-1965 +++ +++ +++ -++ --------------- --- ------------ 2012-4387 +++ +++ ------ --------------- --- ------------ 2012-0393 +++ --------- --------------- --- ------------

vulnerable attack successful anomalies not vulnerable attack failed no anomalies

Note that some outcomes were identical for different versions of Struts, and in the interest of saving space.

Table 1. Attack Outcomes on different versions of Struts.

table cell contains three outcomes. The first indicates whether it was reported that the particular version of Struts was indeed affected (+), or not (-) by the vulnerability. The second outcome specifies whether the execution of the attack for that vulnerability was successful (+), or not (-). The third outcome, specifies whether th Runtime Verifier detected anomalous behavior during execution of the test case (+), or not (-). For example, the outcome +++ means that the version had the reported vulnerability, that the attack test case successfully executed and that anomalies were detected (true positive).

Considering CVE-2013-2115, the URL tag vulnerability described above, we see from Table 1, that the attack test-case successfully exploited this vulnerability, and was detected as an anomaly, for all versions 2. identifies the anomaly as OGNL code used in rendering the URL. Examining execution traces may help identify that part of the code that is responsible for the vulnerability. Indeed, a study of subsequent versions of the Struts source code that repair the vulnerability reveals that the issue was attributed to translateVariable that invoked OGNL processing. In the repaired code, the method was removed as unnecessary. Carrying out the same test-case on subsequent non-vulnerable versions results in an unsuccessful attack and no anomalous behavior (---), which is as expected. The corresponding trace fragment for those versions shows that no OGNL code execution-the root cause of the vulnerability in previous versions-was observed in the context of URL rendering: 1 reports that the anomaly detection identified successful attacks on vulnerable versions, while unsuccessful attacks on non-vulnerable versions were not identified as anomalous. The table, however, reports two vulnerabilities with a false negative result (++-), that is, a successful attack on a vulnerable version for which anomalous behavior was not observed.

One false negative arises from CVE-2014-7809: a CSRF vulnerability caused by a predictable token generated using a weak generator. It allows an attacker, knowing a previous value of a token, to predict the value of the next token and to use it to perform an attack. Although the attack is caused by a simple coding error, we argue that it does not arise from an unexpected path of code execution. A CSRF attack as a request using a token generated by the attacker results in exactly the same behavior as a legitimate request made by the user. As such, it can not be detected as an anomaly in the execution path.

Another false negative arises from CVE-2013-4310, which reports an ability to bypass security constraints. Our investigation discovered that this vulnerability is only applicable to applications that have a somewhat unusual security mechanism. As mentioned in Section 2, actions in Struts are the basic unit of an application's business logic and are normally mapped to specific URL paths (such as /api/post). Struts offers an alternative addressing through URL parameters with action prefix, such as /api/other?action:post. The vulnerability describes a scenario when a security control, implemented outside struts, based on specific URL pattern is bypassed using the alternative addressing. It could be argued such scenario does not really describe a Struts vulnerability, but rather a faulty security control that documented feature of Struts allows to facilitate. In our experiment, the attack exploiting CVE-2013-4310 was undetected because the application we implemented included use of action: prefixes. Addressing actions though parameters was considered a normal behavior of the application and, when executed during the attack, did not cause anomalous behavior.

False positives In two instances, the test cases result in false positives, that is, anomalous behavior is detected for an unsuccessful attempt to exploit nonvulnerable version (--+). This outcome is observed for CVE-2014-0114 and CVE-2014-0116 in versions where these vulnerabilities are repaired. They are variants of the problem discussed in Section 2 where the internal state of an application can be modified through a chain of getters and setters, and in this case, though crafted cookies. A study of the attack test-cases reveals that the anomalous behavior is related to the special treatment that Struts gives to particular cookie names. The original vulnerabilities were repaired by adding a blacklist of disallowed cookie names (such as starting with class). Thus, processing a normal cookie results in a behavior that is different to processing a cookie with blacklisted name. However, in its standard configuration, the application scanner does not generate a request involving a Struts black-listed cookie and, therefore, the generated model of expected behavior does not include an execution path corresponding to the security processing of a blacklisted cookie. Thus, the testcase, while not an attack, is flagged as an anomaly.

Overall, this part of the experiment indicates that all vulnerabilities that could be attributed to unintended code execution path in struts were successfully detected. It also shows that, with one exception of black-listed cookies, in non-vulnerable versions, where malicious request is properly handled by the application no anomalous behavior is reported.

Functional tests

On generating an expected model of behavior (for a given version of Struts) we check that its norms are sufficiently complete by engaging a further standard application scan and confirm that no anomalies are identified. This confirms that 'normal' expected behavior is properly recognized by the Runtime Verifier. However, for the purposes of the evaluation, we are interested in confirming that the anomaly detection can also discriminate between attacking behavior that exploits a vulnerability versus other behavior that executes code in the region of the vulnerability, but does not actually exploit the vulnerability. To this end, a suite of functional tests were developed to check this ability to discriminate.

For example, the URL tag vulnerability CVE-2013-2115 described in Section 4.3 involves passing a crafted value through a URL parameter. The test calls an application using an additional parameter but without OGNL code. Some vulnerabilities require more advanced test cases. For example, the attack test-case for CVE-2013-2251 involves passing a crafted string through a Strutsspecific request parameter, allowing indirect action addressing. We developed two further functional test-cases that check this particular functionality. First uses the indirect addressing, but with a correct action name, and checks that the requested action was called. Second has an incorrect action but checks whether the application replied with expected/corresponding error. The purpose of these tests is to check whether the anomaly detection actually reacts to a genuine vulnerable path of execution or whether the path from related valid functionality is flagged as an anomaly. By making sure that the functionality is exercised we can distinguish between these two cases, exercising particular functionality normally (even if in error scenario) and during the attack.

Overall, we developed 19 test cases that explore non-attacking behavior in the region of the 18 vulnerabilities. The outcome-score of each test is similar to the vulnerability tests and represented using two values. The first outcome value reports whether the test was successful, that is whether the tested functionality worked correctly. Note, that because we have also tested an application's response to an incorrect request, a successful outcome may mean that the application correctly responded to an incorrect value, such presentation of an error page. The second outcome value reports whether an anomaly was detected during execution of the test. Most of the test outcomes are indicating a successful test with no anomalies. However, in two cases the outcome indicates test failure with no anomalies is observed. These are the test cases for indirect action addressing using action: prefix. The test fails for all versions from 2.3.15.2. This is because, as a response to CVE-2013-4310, this functionality was disabled.

Experiment insights

The application scanner may trigger an unexpected behavior, which if unattended to, becomes part of the model used in runtime verification. However, this phenomenon was not observed during our experiments. Furthermore, we assume that any vulnerability identified during scanning will be remediated by modifying the source code and the scan repeated.

Anomaly detection/prevention adds a performance overhead that should be considered. In order to integrate anomaly detection with the application we used a Java Security Manager and AspectJ. These tools are routinely used for implementing security mechanisms and their performance impact has been investigated [START_REF] Herzog | Performance of the java security manager[END_REF][START_REF] Hilsdale | Advice weaving in AspectJ[END_REF]. During the the Struts experiment, the average time to process an HTTP request from the scanner to the application took 4950µs without instrumentation. With the Anomaly Manger enabled for runtime verification, the average time increased by 3.85% to 5140µs. However, the increase depends on how much of application activity is covered by runtime verification: in the experiment this was limited to the Struts library.

Part of our experimental setup involved inspecting the codebases of different versions of Struts in order to identify the vulnerable code and implement the attack tests. In carrying out this detailed code-level review we observed a number of programming phenomena across the different versions. In particular, the phenomena that some generic functionality of Struts allows for a specific execution scenario that compromised security. The otherwise harmless features, such as addressing actions, setting their parameters and evaluating expressions, when used in a particular way allowed unintended operations. For example, five vulnerabilities exploit the feature that allows setting action properties through HTTP parameters and accessing sensitive objects such as session or class loader.

Overall, we observed that the majority of the programming issues relate to rather simple programming errors. In particular, while a general functionality was implemented, a specific, unexpected execution pattern was not considered nor handled by the code. In some cases, such as accessing the class loader via parameter (CVE-2014-0094), we conjecture that the developers were surely aware that accessing properties though parameters can be a security risk. Some specific parameters, such as session object, were black listed but others that are less obvious, such as class, were not. In other cases, when a vulnerability was identified in one part of the framework it was not immediately correlated to another part that was also vulnerable [START_REF] Ashraf | Analysis of recent struts vulnerabilities parameters and cookie interceptors, their impact and exploitation[END_REF]. For example, the fix for CVE-2014-0094 addressed class loader manipulation though request parameters but did not provide the fix for the same attack using cookies. Our analysis of Struts vulnerabilities would seem to confirm other studies [START_REF] Oliveira | It's the psychology stupid: How heuristics explain software vulnerabilities and how priming can illuminate developer's blind spots[END_REF] that indicate that developers tend to repeat security errors even when they are aware of particular vulnerability.

Conclusion

We argue that many of the security vulnerabilities that are caused by programing error can lead to unexpected execution paths that can be detected using anomaly detection techniques. However, applying such techniques to contemporary, enterprise software is not trivial and poses a set of challenges such as selecting the scope and abstraction level of software monitoring, establishing a base line behavior and choosing suitable behavioral model.

We propose a methodology for putting anomaly detection into use for contemporary software components and apply it to Apache Struts as a part of large scale application. Our experiments demonstrate that it is possible to learn a sufficiently rich model of the application's expected use of Struts such that it can be used to detect anomalies in its subsequent use of Struts. Indeed, results indicate that all 16 execution path-related vulnerabilities identified over 26 versions of Struts over 5 years are effectively identified as anomalies. While the experiments were comprehensive, they are limited to Struts vulnerabilities. Nevertheless, we believe the results point to the potential of using anomaly detection techniques in contemporary software and that further research on this topic is worthwhile.

Fig. 2 .

 2 Fig. 2. Java Runtime Verifier Fig. 3. Experiment setup

Fig. 4 .

 4 Fig. 4. Growth in behavioral norms (transactional patterns).

ParametersInterceptor: PostAction.getUser() PostAction: ApiAction.getSession() ParametersInterceptor: User.setId("frank")

	<init>
	ServletConfigInterceptor: ApiAction.setSession()
	ParametersInterceptor: PostAction.setMessage("Hello")
	DeprecationInterceptor: PostAction.execute()
	PostAction: PostAction.getUser()
	PostAction: ApiAction.getSession()
	PostAction: User.getId()
	PostAction: Datastore.add("frank", "Hello")

  Table 1 contains 3 the outcome of testing each of the 18 reported vulnerabilities against each of the 26 versions of Struts. Each

	CVE ID	2.3.1 2.3.4 2.3.14 2.3.14.1 2.3.14.2 2.3.15 2.3.15.1 2.3.16 2.3.16.1 2.3.16.2 2.3.16.3 2.3.20 2.3.24 2.3.24.1
	2015-5209 +++ +++ +++ +++	+++ +++ +++ +++ +++	+++	+++ +++ +++ ---

  3.1 -2.3.14.1 publicly announced to be vulnerable (+++). A closer examination of the execution trace fragment, generated from the attack test case,

	DefaultUrlHelper DefaultUrlHelper.translateAndEncode
	DefaultUrlHelper DefaultUrlHelper.translateVariable
	OgnlInvoke invoke
	ServletUrlRenderer ComponentUrlProvider.getAnchor
	ServletUrlRenderer ComponentUrlProvider.isPutInContext

  Some tests executed with unanticipated outcomes. A surprising result -++, indicates a successful attack, with the anomaly detected, for Struts version 2.3.14.1 for which no vulnerability was reported in CVE-2013-1965 and CVE-2013-1966. A closer examination of these two vulnerabilities confirms that, contrary to publicly available information, version 2.3.14.1 is indeed vulnerable.False negatives In most cases, Table

	DefaultUrlHelper DefaultUrlHelper.encode
	ServletUrlRenderer ComponentUrlProvider.getAnchor
	ServletUrlRenderer ComponentUrlProvider.isPutInContext

Presented as a table, keeping in mind Edward R. Tufte's (2004) observation that "small non-comparative highly labeled data sets usually belong in tables".

Acknowledgement. This work was supported, in part, by Science Foundation Ireland under grant SFI/12/RC/2289.


Georg Merzdovnik
email: gmerzdovnik@sba-research.org

Klaus Falb

Martin Schmiedecker
email: mschmiedecker@sba-research.org

Artemios G Voyiatzis
email: avoyiatzis@sba-research.org

Edgar Weippl

Whom You Gonna Trust? A Longitudinal Study on TLS Notary Services

TLS is currently the most widely-used protocol on the Internet to facilitate secure communications, in particular secure web browsing. TLS relies on X.509 certificates as a major building block to establish a secure communication channel. Certificate Authorities (CAs) are trusted third parties that validate the TLS certificates and establish trust relationships between communication entities. To counter prevalent attack vectors -like compromised CAs issuing fraudulent certificates and active man-in-the-middle (MitM) attacks -TLS notary services were proposed as a solution to verify the legitimacy of certificates using alternative communication channels. In this paper, we are the first to present a long-term study on the operation of TLS notary services. We evaluated the services using active performance measurements over a timespan of one year and discuss the effectiveness of TLS notary services in practice. Based on our findings, we propose the usage of multiple notary services in conjunction with a semi-trusted centralized proxy approach, so as to protect arbitrarily-sized networks on the network level without the need to install any software on the client machines. Lastly, we identify multiple issues that prevent the widespread use of TLS notary services in practice and propose steps to overcome them.

Introduction

Secure communication is a key part of today's Internet applications. The majority of online applications, ranging from e-mail to VPN and browsing the web, rely on SSL and TLS 3 to provide secure communication mechansims such as authenticity, confidentiality, and integrity. TLS 1.2 is, at the time of writing, the most recent version [START_REF] Dierks | The Transport Layer Security (TLS) Protocol Version 1.2. RFC 5246[END_REF], with TLS 1.3 currently in the making. Trust in the TLS ecosystem is distributed over software vendors and an underlying public key infrastructure (PKI) composed of various certificate authorities (CAs). To establish a secure connection, a client verifies the signature of a server's certificate.

If the server's certificate is signed by a trusted certificate authority, the certificate is accepted, otherwise it is rejected. To determine if a CA is to be trusted, the client relies on a so called "trust store", i.e., a list of certificate authorities that it can trust. These trust stores are usually shipped with the application or are included in the operating system. If an attacker gets her hands on one of the private keys of one of these certificate authorities, she is able to issue valid (trusted) certificates for arbitrary-named servers, since the signatures can only be validated against the local trust store. This allows for effective Man-in-the-Middle (MitM) attacks against any kind of targets.

Recent incidents have shown that the subversion of the chain of trust is a viable scenario. Examples include the infamously hacked certificate authorities DigiNotar and Comodo [START_REF] Leavitt | Internet security under attack: The undermining of digital certificates[END_REF], during which their private keys were stolen. Incidents such as the case of Superfish [START_REF]Superfish vulnerability[END_REF] and the Dell eDellroot certificate [START_REF] Thomas | Response to concerns regarding eDellroot certificate[END_REF] demonstrate that sometimes even system vendors, like Lenovo or Dell, accidentally introduce vulnerabilities. In these cases, trusted certificate authorities were included in the local trust store of the operating system, which also included the private keys to provide extended functionality, allowing everyone to extract the CA private key and launch unnoticed MitM attacks. For affected users, there is nearly no possibility to distinguish between valid server certificates and those signed by fraudulent CAs, since there are no visible distinction marks and the client's software marks them as trusted.

To solve the problem of multiple valid and trusted certificate chains, several solutions have been proposed recently. These solutions include DANE [START_REF] Hoffman | The DNS-Based Authentication of Named Entities (DANE) Transport Layer Security (TLS) Protocol: TLSA. RFC 6698 (Proposed Standard)[END_REF], public key or certificate pinning using HPKP [START_REF] Evans | Public key pinning extension for HTTP (HPKP)[END_REF], and TLS notary services. The latter are based on the principle of multi-path probing. Figure 1 depicts the usual workflow of such notary services. The idea is to query different "notary" servers if they are presented with the same certificate for a certain communication entity as the client. Therefore, to launch an undetected MitM attack, an attacker would need to intercept as well all the connections to the entity that originate from all the queried notary servers. Since these notary servers are usually spread in different networks around the globe, the risk of an effective, unnoticed MitM attack is highly reduced, even if the certificate is trusted by the local trust store. On the other hand, such a system could reduce the dependability on certificate authorities, since the validation does not have to depend on trusted certificate authorities, but could rely solely on the quorum of a set of notary servers.

DANE is far from being usable in practice as it relies on DNSSEC which is still not widely deployed. Certificate and public key pinning are still supported only by selected applications (e.g., Chrome, Firefox, and some mobile apps [START_REF] Fahl | Why Eve and Mallory love Android: An analysis of Android SSL (in) security[END_REF][START_REF] Oltrogge | To pin or not to pin-helping app developers bullet proof their TLS connections[END_REF]). On the other hand, TLS notaries are already implemented as browser extensions, thus being usable in practice. However, there is still no complete study on the long-term usage of notary services and how they react to changes in a real-world setting. In this paper we therefore implement a modular system to evaluate notary services in the long term and on a daily basis, independently of the used browsers.

The contributions of this paper are as follows: -We present a longitudinal study on the effectiveness of three well-known notary services over a one-year period.

-We describe a concept of mapping multiple TLS notaries for transparent end-user protection and an implementation of it as a proxy service.

-We identify problems of combining these services, including lack of widespread adoption and the problem of view inconsistencies.

The rest of the paper is organized as follows: In Section 2, we provide the necessary background on TLS and notary services, as well as the related work. In Section 3, we describe our concept of a proxy notary. In Section 4, we describe our methodology for evaluating the proxy as well as the three TLS notary services independently, whereas our results are described in Section 5. We discuss these results in Section 6. Finally, Section 7 presents the conclusions of this paper and discusses future directions of work.

Related Work

Large-Scale TLS Protocol Studies

The problems with SSL and trusted certificate authorities have been studied for several years, and several large-scale studies that focused on the TLS ecosystem have been conducted lately. One of the first large-scale studies targeting SSL certificates is the Electronic Frontier Foundation's SSL Observatory [START_REF] Eckersley | An observatory for the SSLiverse[END_REF]. Its dataset includes publicly visible SSL certificates available through IPv4.

Holz and Durumeric [START_REF] Holz | The SSL landscape: a thorough analysis of the X.509 PKI using active and passive measurements[END_REF][START_REF] Durumeric | Analysis of the HTTPS certificate ecosystem[END_REF] focused on the IPv4-wide analysis of TLS in the context of HTTPS. Mayer et al. [START_REF] Mayer | No need for black chambers: Testing TLS in the e-mail ecosystem at large[END_REF] and Holz et al. [START_REF] Holz | TLS in the wild: an Internet-wide analysis of TLS-based protocols for electronic communication[END_REF] mainly focused on TLS in other application domains, like the e-mail ecosystem. In particular, the recently proposed improvements to port-scanning as well as the open-source release of tools like zmap [START_REF] Durumeric | ZMap: Fast Internet-wide scanning and its security applications[END_REF] and masscan [START_REF] Graham | Masscan: Mass IP port scanner[END_REF] made it easy to collect IPv4-wide information on specific questions. Durumeric et al. also set up a special search engine, Censys [START_REF] Durumeric | A search engine backed by Internet-wide scanning[END_REF], which is backed by these Internet-wide scans and allows for deeper analyses. While these studies provide an interesting and valuable view on the TLS ecosystem, they are not designed to provide further information on fraudulently issued certificates.

TLS Certificate Validation

Several approaches have been proposed in the literature to mitigate the shortcomings of a central point of trust. One method to provide protection against fraudulent certificates is Certificate Pinning which is already distributed in mobile applications [START_REF] Fahl | Why Eve and Mallory love Android: An analysis of Android SSL (in) security[END_REF][START_REF] Oltrogge | To pin or not to pin-helping app developers bullet proof their TLS connections[END_REF]. However, recent studies showed that many mobile applications incorrectly implement the validation of TLS certificates [START_REF] Fahl | Why Eve and Mallory love Android: An analysis of Android SSL (in) security[END_REF][START_REF] Fahl | Rethinking SSL development in an appified world[END_REF]. The Internet Engineering Task Force also proposed the Public Key Pinning Extension for HTTP [START_REF] Evans | Public key pinning extension for HTTP (HPKP)[END_REF]. This allows a web server to set a special header, which tells the browser to only accept a certain certificate or certificates signed by a specific CA, for the specific server and for a specific amount of time.

Other protection mechanisms have been implemented in the form of browser extensions. Soghoian et al. [START_REF] Soghoian | Certified lies: Detecting and defeating government interception attacks against SSL (short paper)[END_REF] implemented Certlock, an extension that is based on the trust-on-first-use policy to bind the CA to the CommonName of a websites certificate. This method is similar to pinning every certificate on first encounter. Winter et al. [START_REF] Winter | Spoiled onions: Exposing malicious Tor exit relays[END_REF] provide a system that uses an independent Tor circuit for certificates that issued a browser warning. However, this does not protect against valid but yet malicious certificates. Syverson and Boyce also employ Tor for page verification [START_REF] Syverson | Genuine onion: Simple, fast, flexible, and cheap website authentication[END_REF], but they do not rely on probing the same server on the same domain; instead, they host the site again on a .onion address and use this mirror to compare the keys. Holz et al. [START_REF] Holz | X.509 forensics: Detecting and localising the SSL/TLS men-in-the-middle[END_REF] implemented CrossBear, a system which employs hunter nodes to track down TLS MitM attacks.

TLS certificate notary services can be used to verify a certificate through multiple paths. Wendlandt et al. [START_REF] Wendlandt | Perspectives: Improving SSH-style host authentication with multi-path probing[END_REF] proposed Perspectives, which is based on multiple servers to observe the state of TLS certificates. Convergence [START_REF] Marlinspike | SSL and the future of authenticity[END_REF] builds on the same principles as Perspectives and provides further methods for trust management. Bates et al. [START_REF] Bates | Forced perspectives: Evaluating an SSL trust enhancement at scale[END_REF] tried to answer the question of what happens if everyone is using the notary service Convergence; Fuchs et al. extended the approach of centralized notary servers and implemented Laribus [START_REF] Fuchs | Laribus: privacypreserving detection of fake SSL certificates with a social P2P notary network[END_REF], a P2Pbased approach on notary services. Laribus is based on a social graph, which allows users to build notary groups without the need to rely on a central notary server.

Methodology and Measurement Setup

To monitor the effectiveness and behavior of different notary services, we set up an automated crawling environment. Figure 2 gives an overview of the overall design. We implemented the proxy in mitmproxy [START_REF] Cortesi | mitmproxy[END_REF], which allowed us to validate certificates through several extension modules. These extension modules implement interfaces to various notary services, which will be described in Section 3.2. We used this system to collect daily statistics of these implemented extension modules over a one-year period.

Browser

Data Collection System

The data collection system was implemented in such a way that it is extensible, reusable and can furthermore be used by the end users to evaluate their own browsing session. Therefore the overall data collection consists of three components: (1) a web browser, (2) an intercepting proxy to monitor HTTPS sessions, and (3) proxy plugins to query various notary services.

Browser To query the different webpages, we utilized wget with the proxy settings pointing to our intercepting proxy. While we used a lightweight, GUIless browser for our periodic scans, any other full-blown browser could be used as well. This makes the validation proxy described in the next paragraph easier to deploy in combination with other systems. End users can use the proxy to secure or evaluate TLS certificates against various notary services without the need to install separate plugins in their browsers.

Intercepting Proxy To conduct the certificate validation, we implemented an HTTP/HTTPS proxy server in Python 2 using the mitmproxy [START_REF] Cortesi | mitmproxy[END_REF] library as a basis. The proxy server acts as an intermediary between the client and the web server. For each encountered HTTPS certificate, the proxy server conducts the certificate validation using the configured notary services.

Proxy Plugins To make the system extensible, the communication with the notary services is implemented as plugins. This makes it easy to extend our system so as to evaluate additional notary services. The proxy in general supports two modes of operation: synchronous and asynchronous. In synchronous mode, the proxy waits for all the responses from the notary services before the original page is passed to the requesting browser. In case of a validation error, this allows to terminate the page load before the page content is rendered to the user.

The second proxy mode asynchronously collects validation information from the notary services and logs them in the file system for later inspection and analysis. In this mode, the page load cannot be interrupted or terminated, since the page is served to the user without waiting for validation responses. For the evaluation, we only look at the results from the asynchronous mode.

Notary Services

At the time of our initial analysis of the notary services ecosystem, we identified three services that were in use and also had an active and open infrastructure, namely Perspectives, Convergence, and ICSI. We give a short introduction to the inner workings of these systems in the next paragraphs.

Perspectives [START_REF]Perspectives project[END_REF] pioneered the multi-path probing approach: The system employs multiple independent servers, called notaries, which observe publicly-visible web servers and store data about their certificates. When a client contacts a server using TLS, it queries a number of notaries. The notaries reply with information about which certificate the server in question was using in which time period. Using this information, the client can make a more informed trust decision: Do the notaries see the same certificate as the client?

Convergence [START_REF]Thoughtcrime Labs[END_REF] was developed by Moxie Marlinspike and builds on the same design principles as Perspectives, but it incorporates other ideas and principles as well. Its central idea is "trust agility", i.e., the users themselves can choose whom to trust and may also revoke their trust. Similarly to Perspectives, Convergence relies on notaries to decide if a certificate is trustworthy or not. However, the decision process is somewhat different. Using a REST web service API, the client sends a request containing the host, port number, and certificate hash to each notary it wishes to query. The server sends one out of five different types of responses, which can be distinguished by the HTTP status code. The possible responses are:

-The notary could verify the certificate.

-The notary could not verify the certificate.

-The notary cannot decide whether to accept or reject the certificate; the client should ignore this notary in its trust decision. -The client sent a malformed request.

-The server could not perform the request due to an internal error. This approach makes the implementation of a client rather simple, because the client just has to count the votes collected from the notaries. The protocol is described in more detail in [START_REF]Convergence notary protocol[END_REF]. The user can decide whether decisions are based on majority voting or if an unanimous vote is mandatory in order to accept a certificate. [START_REF]ICSI certificate notary[END_REF] is a service from the University of Berkeley that monitors certificates. In contrast to the two aforementioned services, the ICSI Certificate Notary passively monitors traffic from multiple Internet sites and builds a database of certificates seen in this traffic.

ICSI Certificate Notary

The database can be queried by clients by issuing a DNS query containing the hash of the certificate. The service responds to the client whether it has observed that certificate in the past, and if it could trace this certificate to a valid root certificate through one of the following responses:

1. ICSI has seen this certificate:

(a) ICSI can establish a chain of trust to a certificate from the Mozilla root store → ICSI replies 127.0.0.2 to the request. (b) ICSI cannot establish a chain of trust → ICSI replies 127.0.0.1 2. ICSI has not seen this certificate or an error (such as a time-out) has occurred → no reply Note that it is not possible to distinguish between the cases "a query timed out" and "ICSI has not seen this certificate", therefore our proxy plugin rejects the certificate in both cases.

Data Collection

Our data collection involves periodic TLS certificate validation requests to the set of analyzed notary instances for 1,000 web pages. The scans were conducted daily, and each scan involved queries to the three different notary services for each of the encountered certificates. We conducted the evaluation of the validation proxy in two steps: First, we collected a sample set of pages served through HTTPS. Secondly, we conducted daily scans to validate the corresponding TLS certificates against different notary services and analyzed their responses.

Sample Selection

To select samples, we initially obtained the list of Alexa Top 1,000,000 sites [START_REF] Weippl | Alexa top domains[END_REF] on November 29, 2013. From this list, we then selected the top 1,000 sites that responded to an HTTPS query within 30 seconds. This selection represents the websites that attract most of the visits by users, including pages such as Facebook, Twitter, and Google. Many of the selected websites did respond to HTTPS queries, but with an immediate redirection to a (non-secure) HTTP connection. This means that while they do support HTTPS, many users will probably not use it. However, we still included these sites in the evaluation under the assumption that HTTPS is likely to be used in some parts of the website, like the login pages.

Periodic Scan

Between January 31, 2014 and January 29, 2015, for a period of one year, the collection was conducted daily. For each run of the scan, the proxy server was started and the previously selected URLs were queried, with the different notary plugins enabled. The data returned by the proxy plugins as well as the collected certificates were stored for further analysis. To get a baseline for comparison, we also queried the URLs without using a proxy server. Thus, in one evaluation run, each site from our data set was queried for a total of four times.

For each pair of URL and validation method, the following measurements were taken: Verdict: Whether the validation method accepted or rejected the site's X.509 certificate. Reason: The reason why a certificate was rejected, if it had been rejected. This metric is specific to each validation method. Validation Time: The entire time the validation process of a certificate took, including querying the notary server(s) and waiting for a response.

Results

In the following, we describe our results and findings from the collected dataset.

For each notary service, we analyzed how long it took to answer a validation request and also how long it took to react to certificate changes. Furthermore, we studied the availability of these services over the course of one year.

Certificate Changes

To analyse the functionality of notary services, it is important to observe actual certificate changes. Figure 3 depicts the number of different certificates per website we encountered during the course of our study. In 80% of the cases, the websites changed at least once their certificate; some 10% of them changed more than 9 times their certificate within the one year that our study was active.

Validation Time

An important factor concerning notary services from a usability point of view is their response time to validation requests. Therefore, we conducted an analysis of the response time of the various services. With the 1,000 webpages crawled daily for one year, we collected in total more than 350,000 response timing samples per analyzed notary service. Figure 4 summarizes the timing information for the three notary services.

The DNS-based approach of ICSI yields the fastest responses to queries, with the majority (95%) of answers received in under one second. While about half of the responses for Convergence and Perspectives are also below this mark, response times for these two services have a far higher fluctuation. This can be an issue in the case where the notary services are used to validate certificates before a page is loaded, as it could introduce noticeable page load delays for the users. We note that Convergence usually employs a client cache for fingerprints, in an effort to improve the loading times. We did not implement this caching in our proxy so as to get a comparison of the notary service based on newlyencountered pages.

Certificate Acceptance Duration

While the response time is certainly important for the general usability in dayto-day browsing, another temporal factor to take into consideration is the time a notary needs to mark new or changed certificates as valid once they are introduced or updated. Figure 5 depicts the time it took the different services to mark a new certificate as valid after it was changed on the server. Since we conducted daily crawls at a fixed time, the resolution of our scan is also on a daily basis. Therefore, a value of zero days means that the certificate was changed by a server as well as validated by a service within this 24-hour time frame. The validations are set in relation to the total amount of certificate changes that we observed during our scanning period. In the case of Convergence we only considered the server we setup ourselves and did not include the official server results, since the latter only responded in error for the majority of our scans.

As depicted in Figure 5, it takes less time to Convergence so as to adopt to newly-changed certificates, with the majority of certificates seen as valid within the first 24-hour time frame. ICSI is only able to validate about 75% of changed certificates within the same time frame. This fact could be due to the nature of ICSI, which relies on passive information collection, whereas Convergence actively probes servers itself. The relatively low validation rate of Perspectives (45%) can most likely be accredited to the fact that more and more of the servers failed; in the end, it was not possible to reach a quorum on the validity of a certain certificate. Therefore, some of the changed certificates could not be validated successfully anymore. However, even with these limitations in mind, we can still see the general trend that it takes a longer time for Perspectives to successfully validate certificates compared to the other two services. It takes one day for Convergence and at most three for ICSI to fully synchronize. Successful validations in % of total certificate changes convergence icsi perspectives Fig. 5: Time until a newly-seen certificate is marked as validated in percent of the total of observed certificate changes within the period of one year.

Service Availability

Figure 6 provides an overview on the status of certificate validation of the three notary services during the course of one year. The return state for each of the services is given as a percentage for all the collected service responses. It shows the daily average of responses to the 1,000 page request made by the respective crawler.

ICSI was constantly up and running during our scan. We experienced several problems with both Convergence and Perspectives. The analysis of Convergence was based on two servers. The first one was the official server available at notary.thoughtcrime.org and the second one was a server we hosted on an Amazon EC2 instance. The official server became unresponsive in the middle of June 2014.

We encountered a similar problem with Perspectives, where the initiallyavailable servers one after the other shut down or responded in error. As described earlier, the Perspectives validation of certificates operates with a quorumbased approach, in which at least a certain amount of servers must provide a valid response. Due to the fact that more servers answered with an error state, this requirement was no longer met and therefore, from a certain point in time, all certificates were rejected by the system, even if some of the servers still provided a valid response.

Discussion

Analyzing notary services on a longitudinal scale reveals several problems and shortcomings that limit the usability of these services. In the following we discuss the observed limitations and possible future directions for the deployment of notary services.

Response and Validation Times

One problem with notary services is the delay that these services introduce in page requests. As we described in Section 3.1, there are two approaches to verify a certificate through a notary: synchronous and asynchronous. Both approaches have positive and negative sides. Since the synchronous method waits for all notary responses before actually requesting the page, it can introduce a significant delay (as shown in Section 5.2) in page loading, especially if a notary server times out. On the other hand, the asynchronous method loads the page before it receives all notary responses, therefore leaving a window of exposure before notifying the user that something went wrong.

Another problem is the reaction to legitimate certificate changes, namely how long it takes until a service marks a newly seen certificate as valid. Our study shows that it can take up to several days until a certificate is considered as valid. Until the new certificate is validated and has been seen by all the notary services, it will appear as a MitM attack.

Adoption and Continuous Operation

For a notary service (or multi-path probing in general) to be useful for actually validating certificates, there are two important factors that need to be met: (i) Services need to be adopted by the users. This means that users have to run their own servers which others can query. For example, if there is only one official server you can query, this defeats the whole concept. (ii) It implies that one has to fully trust this service, which introduces a single point of failure. A single server could just provide wrong answers to the client's queries without a possibility to check these claims, which would be similar to a device-hosted trust store. On the other hand, even if users set up their own servers, the question is how long they can keep them up and running for other clients to use. Therefore, an important factor to consider is that the amount of available servers could fluctuate. The clients need to be informed of failing servers, since this influences the weight of still-running services in the case of majority voting.

Currently it seems that the adoption of these services by users is low. At the time of the writing, the Firefox Add-on for Perspectives has 5,334 users [START_REF] Schaefer | Perspectives Firefox Add-on[END_REF] and the plugin for Convergence only 77 users [START_REF] Kazantsev | Convergence extra Firefox Add-on[END_REF]. During our study some of the official servers seem to be discontinued, which does not help to increase the trust in this system. What our insights show is that either the incentive for the users to host their own notary services has to be increased or the system itself has to be adapted. One possible adaption is presented by tofu [START_REF] Jones | tofu:// -the web security protocol which should have been[END_REF], proposing a P2P-based system in which every client is automatically also a host. While this system may be able to solve the problem of service availability, it could still impose further risks that need to be analyzed in the future.

Privacy

Beside the technical aspects, there are others to be considered. One of them are possible privacy implications. By using a third-party service to validate certificates, it is easy for its server(s) to collect information about the pages a client visited. Therefore it is possible for the server(s) to build a browsing profile of the specific user. One solution to this problem would be for the users to host their own servers. However, this is not always an option and future research should focus on the possibilities to validate certificates without giving away too much information about the client.

While we do not have solutions to these problems (yet), we still believe that notaries are a viable alternative to increase the overall security of TLS. Thus, they should be studied further so as to improve the current limitations.

Conclusion

In this work, we presented a longitudinal study on the availability and functionality of different notary services. We conducted daily scans over a period of one year and analyzed the collected data. We explored the ecosystem of notary services and analyzed their behavior on a large scale. To conduct this study, we developed a new proxy-based system to transparently query different notary services for increased protection against MitM attacks and gave an overview on the inner workings of these notary services. Lastly, we discussed the results of our study in the context of the available notary services. We described the problems and pitfalls that can arise by using the existing systems.

Existing notary services have the problem that the initial request for an unknown page can introduce extra latency, since the notary has to query the server for the certificate. With the rise of fast, Internet-wide scanning solutions, there are several projects that analyze the TLS landscape. One of these projects is scans.io [START_REF]Scans.io -Internet-Wide scan data repository[END_REF] which hosts a regularly-collected dataset of TLS certificates. Censys.io [START_REF] Durumeric | A search engine backed by Internet-wide scanning[END_REF] provides a search engine over the scans.io datasets. Future research could evaluate the possibilities to use these data sources either as alternative initial data providers to bootstrap notaries or to wrap the data into a separate notary service.

Fig. 1 :

 1 Fig. 1: The usual flow of a request for certificate notary services.

Fig. 2 :

 2 Fig. 2: Overview of our measurement setup.

Fig. 3 :

 3 Fig. 3: Number of different certificates observed for each tracked domain

Fig. 4 :

 4 Fig. 4: Response times of notary services to validation requests. Outliers are cut off at 10 seconds.

Fig. 6 :

 6 Fig. 6: Timeline of the responses collected from the different services over the course of one year.

In this paper we use the term "TLS" to refer to all incarnations of SSL and TLS, if not specified otherwise.

Acknowledgements

This work has been carried out within the scope of "u'smile", the Josef Ressel Center for User-Friendly Secure Mobile Environments, funded by the Christian Doppler Gesellschaft, A1 Telekom Austria AG, Drei-Banken-EDV GmbH, LG Nexera Business Solutions AG, NXP Semiconductors Austria GmbH, and Österreichische Staatsdruckerei GmbH. Additionaly the research was funded by the Austrian Research Promotion Agency (FFG) through the BRIDGE 1 grant P846028 (TLSiP) and the COMET K1 program.