Daniela Oliveira

Jedidiah Crandall

Harry Kalodner

Nicole Morin

Megan Maher

Jesus Navarro

Felix Emiliano

An Information Flow-based Taxonomy to Understand the Nature of Software Vulnerabilities

Despite the emphasis on building secure software, the number of vulnerabilities found in our systems is increasing every year, and well-understood vulnerabilities continue to be exploited. A common response to vulnerabilities is patch-based mitigation, which does not completely address the flaw and is often circumvented by an adversary. The problem actually lies in a lack of understanding of the nature of vulnerabilities. Vulnerability taxonomies have been proposed, but their usability is limited because of their ambiguity and complexity. This paper presents a taxonomy that views vulnerabilities as fractures in the interpretation of information as it flows in the system. It also presents a machine learning study validating the taxonomy's unambiguity. A manually labeled set of 641 vulnerabilities trained a classifier that automatically categorized more than 70000 vulnerabilities from three distinct databases with an average success rate of 80%. Important lessons learned are discussed such as (i) approximately 12% of the studied reports provide insufficient information about vulnerabilities, and (ii) the roles of the reporter and developer are not leveraged, especially regarding information about tools used to find vulnerabilities and approaches to address them.

Introduction

Despite the security community emphasis on the importance of building secure software, the number of new vulnerabilities found in our systems is increasing with time; The 2014 Symantec Internet Security report announced that 6,787 new vulnerabilities occurred in 2013. This represents a 28% increase in the period 2013-2014, compared to a 6% increase in the period 2012-2013 [5]. Further, old and well-studied vulnerabilities, such as buffer overflows and SQL injections, are still repeatedly reported [START_REF]Security Focus Vulnerability Notes[END_REF].

A common approach to address vulnerabilities is patch-based mitigation targeting specific exploits. This approach may not completely address the vulnerability since it fails to address its essence, and does not generalize well with similar vulnerabilities exploited differently. Take the file-system TOCTTOU vulnerability as an example. Dean and Hu [START_REF] Dean | Fixing Races for Fun and Profit: How to Use access(2)[END_REF] provided a probabilistic solution for filesystem TOCTTOU that relied on decreasing the chances of an attacker to win all races. In their solution, the invocation of the access() ... open() sequence of system calls is followed by k additional calls to this pair of system calls. From the application layer viewpoint, the solution addresses the concurrency issue because the chances that the attacker will win all rounds are small. Borisov et al. [START_REF] Borisov | Fixing Races for Fun and Profit: How to Abuse atime[END_REF], however, observed that this vulnerability crosses the boundary between the application and the operating system layers, and allowed an attacker to win the race by slowing down filesystem operations. This caused the victim process to be likely suspended after a call to access().

A first step towards viewing cyber security as a science is understanding software vulnerabilities scientifically. Weber et al. [START_REF] Weber | A Software Flaw Taxonomy: Aiming Tools at Security[END_REF] also argue that a good understanding and systematization of vulnerabilities aids the development of static-analysis or model checking tools for automated discovering of security flaws.

Taxonomies decrease the complexity of understanding concepts in a particular field. Taxonomy-based vulnerability studies have been tried since the 70s [START_REF] Abbot | Security Analysis and Enhancements of Computer Operating Systems[END_REF][START_REF] Ii | Protection Analysis Project Final Report[END_REF][START_REF] Landwehr | A Taxonomy of Computer Program Security Flaws[END_REF][START_REF] Aslam | A Taxonomy of Security Faults in the UNIX Operating System[END_REF] but they were proved ambiguous by Bishop and Bailey [START_REF] Bishop | A Critical Analysis of Vulnerability Taxonomies[END_REF], who showed how the same vulnerability was put into multiple categories depending on the layer of abstraction it was being analyzed. The other problem with current taxonomies is their complexity. For example, CWE v1.9 has 668 weaknesses and 1043 pages. Ambiguous and complex taxonomies not only confuse a developer, but also hinder the widespread development of automated diagnosis tools leveraging its categories as points for checks.

This paper introduces a concise taxonomy for understanding the nature of vulnerabilities that views vulnerabilities as fractures in the interpretation of information as it flows in the system. In a seminal paper on computer viruses [START_REF] Cohen | Computer viruses: Theory and experiments[END_REF], Cohen said that "information only has meaning in that it is subject to interpretation." This fact is at the crux of vulnerabilities in systems. As information flows from one process to another and influences the receiving process' behavior, interpretations of that information can lead to the receiving process doing things on the sending process' behalf that the system designer did not intend to allow as per the security model. Information, when viewed from the different perspectives for the various levels of abstraction that make up the system (OS, application, compiler, architecture, Web scripting engine, etc.), should still basically have the same interpretation. The lack of understanding on the nature of vulnerabilities cause defense solutions to focus on only one perspective (application, compiler, OS, victim process or attacker process) and become just mitigation solutions that are rapidly circumvented by a knowledgeable adversary.

To validate the unambiguity and usefulness of this taxonomy, a machine learningbased [START_REF] Witten | Data Mining: Practical Machine Learning Tools and Techniques[END_REF] study was conducted using a training set of 641 manually classified vulnerabilities from three public databases: SecurityFocus [START_REF]Security Focus Vulnerability Notes[END_REF], National Vulnerability Database (NVD) [START_REF]National Vulnerability Database[END_REF] and Open Sourced Vulnerability Database (OSVDB) [START_REF]Open Source Vulnerability Database[END_REF]. This manually labeled set was used to train a machine learning classifier built with the Weka suite of machine learning software [START_REF] Witten | Data Mining: Practical Machine Learning Tools and Techniques[END_REF]. More than 70000 vulnerabilities from a ten year period from the three databases were automatically classified with an average success rate of 80%, demonstrating the unambiguity potential of the taxonomy.

Important lessons learned in this study are discussed. First, there are a significant number of poorly reported vulnerabilities (approximately 12% of the vulnerabilities in the manually classified set), with descriptions containing insufficient or ambiguous information. This type of report pollutes the databases and makes it hard to address vulnerabilities scientifically, and disseminate relevant information to the security community. Second, the roles of the reporter and the developer are not leveraged and important information has not been added to reports, such as tools used to find vulnerabilities and approaches taken to address them. Finally, the lack of standards on vulnerability reports and across databases adds complexity to the goal of addressing vulnerabilities scientifically, as they are viewed as dissimilar, independent and unique objects. The paper also discusses the application of such taxonomy in the context of automated diagnosis tools to assist the developer. This paper's contributions are as follows:

1. A concise taxonomy for understanding the nature of vulnerabilities based on informationflow that can be easily generalized and understood is proposed. 2. The taxonomy's categories and their information-flow nature are discussed against notorious vulnerabilities, such as buffer overflows, SQL injection, XSS, CSRF, TOCTTOU, side-channels, DoS, etc.. 3. A large scale machine learning study validating the taxonomy's unambiguity is presented. In this study a manually labeled set of 641 vulnerabilities trained a classifier that automatically categorized more than 70000 vulnerabilities from three distinct databases with an average success rate of 80%. 4. Important lessons learned are discussed such as (i) approximately 12% of the studied reports provide insufficient information about vulnerabilities, and (ii) the roles of the reporter and developer are not leveraged, especially regarding information about tools used to find vulnerabilities and approaches to address them. 5. A discussion of the application of this taxonomy in automated diagnosis tools is provided.

The rest of the paper is organized as follows. Section 2 presents the proposed taxonomy and discusses notorious vulnerabilities from the perspective of information flow. Section 3 presents the machine learning study conducted to evaluate the taxonomy. Section 4 discusses related work and section 5 concludes the paper.

The Taxonomy

This paper introduces a new vulnerability taxonomy based on information flow. The goal was to produce an unambiguous taxonomy that can be leveraged to address software vulnerabilities scientifically. Vulnerabilities are viewed as fractures in the interpretation of information as it flows in the system. Table 1 details with examples the proposed taxonomy and its categories. The following sections describe each one of these categories with some examples and how they can be viewed in terms of information flow.

Please notice that there is no design flaw category because this study understands that all vulnerabilities are ultimately caused by design flaws. Vulnerabilities are weaknesses in the design and/or implementation of a piece of software that allow an adversary to violate the system security policies regarding the three computer security pillars: confidentiality, integrity and availability.

Control-flow hijacking

These vulnerabilities allow an attacker to craft an exploit that communicates with a process in a malicious way, causing the adversary to hijack the process' control-flow.

Category Description Examples Control-flow hijacking

Vulnerabilities where information flows from the input to a process into the control flow of the process causing its execution to be hijacked.

Buffer overflows, memory corruption, SQL injection, cross-site scripts Process confusion Vulnerabilities where information flows from the security metadata of one object into a security decision about another. TOCTTOU, confused deputy, cross-site request forgery (CSRF) Sidechannels Vulnerabilities where information flows from physical or side-effects of the operation or communication channels of the system into an illegitimate authentication decision or information disclosure. Physical: timing/power and electromagnetic attacks. Communications/operation: man-in-the-middle, replay, /proc filesystem attacks Exhaustion Vulnerabilities where a significant amount of information flows into a process causing unavailability (exhaustion of resources) or an illegitimate authentication decision (exhaustion of input space).

Resources: DoS, TCP SYN flood, ICMP flood. Input space: password cracking and dictionary attacks Adversarial accessibility Vulnerabilities where information is allowed to flow to the attacker's process causing a breach of confidentiality, illegitimate authentication or interference with system functionality.

Assignment of weak permissions to system objects, access control errors, and noncontrol-data attacks [START_REF] Chen | Non-control-hijacking attacks are realistic threats[END_REF] Table 1: Taxonomy categories.

There are several vulnerabilities that fall into this category: all types of buffer overflows [START_REF] Chapin | Buffer Overflow and Format String Overflow Vulnerabilities[END_REF] (stack, heap, data, dtors, global offset table, setjmp and longjmp, double-frees, C++ table of virtual pointers, etc.), format string, SQL injection [START_REF] Su | The Essence of Command Injection Attacks in Web Applications[END_REF] and cross-site scripts (XSS) [START_REF] Wassermann | Static Detection of Cross-site Scripting Vulnerabilities[END_REF]. Code-reuse attacks [START_REF] Shacham | The Geometry of Innocent Flesh on the Bone: Return-into-libc without Function Calls (on the x86)[END_REF] are considered a capability of an attacker after leveraging a stack-based buffer overflow and not a vulnerability in itself.

In a general memory corruption attack an adversary provides a victim process with a set of bytes as input, where part of these bytes will overwrite some control information with data of the attacker's choice (usually the address of a malicious instruction). This control information contains data that will eventually be loaded into the EIP register, which contains the address of the next instruction to be executed by the CPU at the architecture level.

For these cases, the fracture in the interpretation of information occurs when user input crosses boundaries of abstractions. User input is able to influence the OS, which manages the process address space and the control memory region being abused. User input also influences the architecture layer as it is directly written into the EIP register. For buffer overflows on the heap, data, and dtors areas, an attacker overwrites a data structure holding a function pointer with a malicious address. The effect is the same in all cases: the function will be eventually called, and its address will be loaded into the EIP register.

In a SQL injection [START_REF] Su | The Essence of Command Injection Attacks in Web Applications[END_REF] user input is directly combined with a SQL command written by an application developer, and this allows an attacker to break out of the data context when she supplies input as a combination of data, control characters and her own code. This malicious combination causes a misinterpretation of data input as it is provided by the web scripting engine. The scripting engine, which processes user input, misinterprets it as data that should be concatenated with a legitimate command created by the application developer. The SQL query interpreter then parses the input provided by the scripting engine as SQL code that should be parsed and executed. The misinterpretation between the web scripting engine and the SQL query interpreter causes the vulnerability.

Process confusion

This type of vulnerability allows an attacker to confuse a process at a higher layer of abstraction where this process is usually acting as a deputy, performing some task on behalf of another lower privileged process. A fracture in the interpretation of information allows the security metadata of one object to be transferred into a security decision about another object. A classic example is TOCTTOU, one of the oldest and most wellstudied types of vulnerability [START_REF] Mcphee | Operating System Integrity in OS/VS2[END_REF]. It occurs when privileged processes are provided with some mechanism to check whether a lower-privileged process should be allowed to access an object before the privileged process does so on the lower-privileged process' behalf. If the object or its attribute can change either between this check and the actual access that the privileged process makes, attackers can exploit this fact to cause privileged processes to make accesses on their behalf that subvert security. The classic example of TOCTTOU is the sequence of system calls access() followed by open(): if (access("/home/bob/symlink", R_OK | W_OK) != -1) { // Symbolic link can change here f = fopen("/home/bob/symlink", "rw"); ... } What makes this a vulnerability is the fact that the invoker of the privileged process can cause a race condition where something about the filesystem changes in between the call to access() and the call to open(). For example, the file /home/bob/symlink can be a symbolic link that points to a file the attacker is allowed to access during the access() check (e.g., file /home/bob/bob.txt) that bob can read and write, but at a critical moment is changed to point to a different file that needs elevated privileges for access (e.g., /etc/shadow).

Consider that the security checks for /home/bob/bob.txt (including stat()ing each of the dentry's and checking the inode's access control list) get compressed into a return value for the access() system call that is stored in register EAX. This information is interpreted to mean that bob is allowed to access the file referred to by /home/bob/symlink.

The information crosses the boundary between an OS abstraction (the kernel) and a user-level abstraction into the EAX register, which contains the return value (architecture layer abstraction). Then a control flow transfer conditioned on the EAX register is now transformed into a decision to open the file pointed to by /home/bob/symlink. The interpretation of information becomes fractured in this information flow between the return value and the open() system call, which occurs at the architecture layer. To the OS, the value returned in register EAX was a security property of /home/bob/bob.txt. At the architectural level the value of the program counter (register EIP), which contains the exact same information, is implied to be a security property of /etc/shadow. The information is the same, but when viewed from different perspectives for the different layers of abstraction that make up the system the interpretation has been fractured.

TOCTTOU is a much broader class of vulnerabilities and no all cases are related to UNIX filesystem atomicity issues [START_REF] Wang | How to Shop for Free Online -Security Analysis of Cashier-as-a-Service Based Web Stores[END_REF].

Side-channels

This type of vulnerability allows an attacker to learn sensitive information about a system such as cryptographic keys, sites visited by a user, or even the options selected by the user when interacting with web applications by leveraging physical or side-effects of the system execution or communications.

Examples of such vulnerability are found in systems where the execution of certain branches is dependent on input data, causing the program to take varying amounts of time to execute. Thus, an attacker can gain information about the system by analyzing the execution time of algorithms [START_REF] Brumley | Remote timing attacks are practical[END_REF]. Other physical effects of the system can be analyzed, such as hardware electromagnetic radiation, power consumption [START_REF] Spadavecchia | A network-based asynchronous architecture for cryptographic devices[END_REF] and sound [START_REF] Zhuang | Keyboard acoustic emanations revisited[END_REF]. An attacker can also exploit weaknesses in the communication channels of a process to breach confidentiality [START_REF] Chen | Side-channel leaks in web applications: A reality today, a challenge tomorrow[END_REF][START_REF] Zhang | Peeping tom in the neighborhood: keystroke eavesdropping on multi-user systems[END_REF][START_REF] Jana | Memento: Learning secrets from process footprints[END_REF].

As example, first consider a timing attack (Physical side-channel) where an adversary attempts to break a cryptosystem by analyzing the time a cryptographic algorithm takes to execute [START_REF] Brumley | Remote timing attacks are practical[END_REF]. The cryptographic algorithm itself does not reveal cryptographic keys, but the leaking of timing information is a side-effect of its execution. This information flows from the server machine to the client machine and is interpreted in the client (the attacker's machine) as tokens of meaningful information. The combination of these tokens of information over several queries allows the attacker to succeed by making correlations among the input, the time to receive an answer, and the key value.

Another example is a Man-in-the-middle (MiM) vulnerability (Communications / Operation), which is a form of eavesdropping where the communication between two parties, Alice and Bob, is monitored by an unauthorized party, Eve. The eavesdropping characteristic of MiM vulnerabilities implies that authentication information is leaked through a channel not anticipated by the system designer (usually the network). In the classic example, Alice asks for Bob's public key, which is sent by Bob through the communication channel. Eve is able to eavesdrop the channel and intercepts Bob's response. Eve sends a message to Alice claiming to be Bob and passing Eve's public key. Eve then fabricates a bogus message to Bob claiming to be Alice and encrypts the message with Bob's public key. In this attack information flows from the communication channel between Alice's and Bob's processes into an illegitimate authentication decision established by Eve.

Exhaustion

This type of vulnerability allows an adversary to compromise the availability or confidentiality of a system by artificially increasing the amount of information the system needs to handle. This augmented information flow can leave the system unable to operate normally (attack on availability) or can allow an attacker to illegitimately authenticate herself into the system (attack on confidentiality). The Exhaustion category was subdivided into two subcategories (exhaustion of resources and exhaustion of input space) due to their differences in nature and also because they target different security pillars, respectively availability and confidentiality. They both belong to the same broader category because they leverage an artificial increase in the amount of information flowing into the system.

Exhaustion of resources vulnerabilities allow an attacker to cause a steep consumption of a system's computational resources, such as CPU power, memory, network bandwidth or disk space. A classic example is the standard DoS attack: an attacker saturates a target machine with communication requests so that the machine is left short of resources to serve legitimate requests. The victim server process does not handle the uncommon case (exploited by attackers) of a steep increase in the amount of information it has to handle.

Exhaustion of input space vulnerabilities are leveraged to allow an adversary to illegitimately authenticate herself into the system by exploiting a great portion of a vulnerable process authentication input space. For example, in a password cracking attack an adversary repeatedly attempts password strings in the hope that one of them will allow her to authenticate herself into the system. A system will be vulnerable to this type of attack depending on the strength of the password. A secure system can tolerate a steep increase in authentication information flowing into it (password guesses) without its confidentiality being compromised, or guard itself against an exhaustion attack, by for example, locking the system after a few failed attempts.

Adversarial Accessibility

These vulnerabilities occur when weaknesses in the system design and implementation allow information to flow to an adversary or her process when it should not, as per the system security policies. A classic example is when weak permissions are assigned to system objects, allowing an adversary access to sensitive information or abstractions. This illegitimate information flow to the attacker can also result in authentication breaches. For instance, a vulnerable access control mechanism that does not perform all necessary checks can allow an attacker to authenticate herself in the system and access its resources.

Evaluation

The goal of this study was to evaluate how faithfully the categories reflect real vulnerabilities and to assess the taxonomy's potential for classifying vulnerabilities unambiguously. This analysis leveraged three well-known public vulnerability databases: JBoss is prone to multiple vulnerabilities, including an information-disclosure issue and multiple authentication-bypass issues. An attacker can exploit these issues to bypass certain security restrictions to obtain sensitive information..."

Table 2: Examples of manually classified vulnerabilities.

SecurityFocus (SF) [4]

, National Vulnerability Database (NVD) [START_REF]National Vulnerability Database[END_REF], and Open Source Vulnerability Database (OSVDB) [START_REF]Open Source Vulnerability Database[END_REF].

The study employed machine learning to classify a large number of vulnerabilities according to the proposed taxonomy. In this analysis we used the Weka data mining software [START_REF] Witten | Data Mining: Practical Machine Learning Tools and Techniques[END_REF]. The study started with the manual classification, according to the proposed taxonomy, of 728 vulnerabilities from SecurityFocus (202 vulnerabilities), NVD (280 vulnerabilities), and OSVDB (246 vulnerabilities) databases. This manual classification was done independently by four of the authors, with an inter-rater agreement of approximately 0.70 (see Table 2). A vulnerability report contains the following attributes (names vary per database): ID, title, description, class, affected software and version, reporter, exploit and solution. For purposes of classification, the most important attributes in a vulnerability report are the title and the description. The class attribute was observed to be highly ambiguous; SecurityFocus, for instance, classifies highly distinct vulnerabilities as Design error. The manual classification selected vulnerabilities in descending chronological order, starting with the most recent vulnerabilities in the respective databases. As some categories were under-represented in the most recent set of reported vulnerabilities and the goal was to build a large and well-represented training set, the authors manually searched for reports fitting under-represented categories in the past. This process showed that the taxonomy was easily applied, even though some questions were raised about vulnerabilities with poor or ambiguous descriptions. Table 3 shows a summary of the manual classification. Approximately 12% of the most recent vulnerability reports contain insufficient or ambiguous information to reason about the corresponding security flaw. For example, the SecurityFocus vulnerability report with BID 55977 only reveals that a certain software is vulnerable. To avoid polluting the training set and confusing the machine learning classifier, all vulnerabilities with insufficient or ambiguous descriptions (87 total) were filtered out of the manually labeled set.

The study proceeded with the automated extraction of all vulnerability reports from NVD, OSVDB and SecurityFocus for the periods of 2013-2012, 2009-2008, and 2004-2003. The goal was to classify vulnerabilities from three distinct periods over the last decade and identify trends and patterns. A total of 70919 vulnerabilities were extracted (37030 from OSVDB, 23155 from NVD and 10506 from Security Focus) forming the testing set to be categorized by the machine learning classifier. We used the Naïve Bayes algorithm as it is popular for text classification.

All the reports collected for the training and testing set were pre-processed by a parser that converted them into the Weka's ARFF format [START_REF] Witten | Data Mining: Practical Machine Learning Tools and Techniques[END_REF]. The parser used the Weka's String to Word vector filter [START_REF] Witten | Data Mining: Practical Machine Learning Tools and Techniques[END_REF], which turned each word in the title or description into an attribute, and checked whether or not it was present. The filter removed stopwords and established a threshold on the number of words kept per machine learning sample.

Table 4 summarizes the results obtained for the automated classification of vulnerabilities for the three databases studied. Control-flow hijacking vulnerabilities make more than 50% of all reported vulnerabilities in all databases, followed by Adversarial accessibility (19%), Exhaustion (16%), Side-channels (3%) and Process confusion (2%). This trend was consistent in all databases and did not change much over the last decade.

The standard method of stratified tenfold cross validation [START_REF] Witten | Data Mining: Practical Machine Learning Tools and Techniques[END_REF] was used to predict the success rate of the classifier, which obtained, respectively, success rates of 84.6%, 73.1%, and 82% for the OSVDB, NVD, and SecurityFocus databases. The authors believe that two reasons prevented the classifiers from obtaining higher success rates: (i) the non-negligible number of reports with insufficient information about the vulnerability; approximately 12% for the most recent vulnerabilities appearing in the training set for all three databases, and (ii) DoS vulnerabilities, which depending on how they are exploited can be classified as Exhaustion or Control-flow hijacking. For example, an attack that works by sending a very large number of requests to a server, so as it does not have sufficient resources to serve legitimate requests exploits an Exhaustion vulnerability. On the other hand, a buffer overflow that crashes the application (still changing the control-flow according to the attacker's choice) is usually named a DoS attack in vulnerability reports, even though the root cause of the vulnerability does not involve exhaustion of resources. Table 5 shows examples of vulnerabilities automatically categorized by the classifier.

Discussion

Approximately 12% of all examined reports do not provide sufficient information to understand the corresponding vulnerabilities. These descriptions specify the capabilities of attackers after the vulnerability is exploited, or just mention that an unspecified vulnerability exists. Also, important information on the process of finding vulnerabilities is usually not provided: reporter contact information, tools used to discover vulnerabilities, whether the vulnerability was discovered through normal software usage or careful inspection, exploit examples and steps to reproduce the vulnerability. Certain reports provide URLs for exploits or steps to reproduce the flaw, but many of these links are invalid as if this information were ephemeral. This information should be permanently recorded; it is The lack of this important information in vulnerability reports shows that the roles played by reporters and developers are undermined. Reports discussing strategies for finding vulnerabilities could help developers designing more secure software. Further, it would be invaluable to the security community and other developers information on how the vulnerability was addressed. For example, was the vulnerability caused by a weakness on a particular API ? Did the developer use a particular tool or strategy to address the vulnerability?

A lack of standardization among vulnerability reports across databases was also observed. This makes it very difficult to understand actual trends and statistics about vulnerabilities; they are viewed as one of a kind and not addressed together according to their similarities. Finally, there is no guarantee that a vulnerability is reported in a public database only after the vendor had been informed about the issue. A responsible reporter should always report the vulnerability first with the vendor or developer and allow them a reasonable amount of time (e.g., 30 days) to address the issue before making it public in a database.

Related Work

The first efforts towards understanding software vulnerabilities happened in the 70s through the RISOS Project [START_REF] Abbot | Security Analysis and Enhancements of Computer Operating Systems[END_REF] and the Protection Analysis study [START_REF] Ii | Protection Analysis Project Final Report[END_REF]. Landwehr et al. [START_REF] Landwehr | A Taxonomy of Computer Program Security Flaws[END_REF] proposed a taxonomy based on three dimensions: genesis, time, and location, and classified vulnerabilities as either intentional (malicious and non-malicious) or inadvertent. Aslam [START_REF] Aslam | A Taxonomy of Security Faults in the UNIX Operating System[END_REF] introduced a taxonomy targeting the organization of vulnerabilities into a database and also the development of static-analysis tools. Bishop and Bailey [START_REF] Bishop | A Critical Analysis of Vulnerability Taxonomies[END_REF] analyzed these vulnerability taxonomies and concluded that they were imperfect because, depending on the layer of abstraction that a vulnerability was being considered in, it could be classified in multiple ways.

Lindqvist and Jonsson [START_REF] Lindqvist | How to Systematically Classify Computer Security Intrusions[END_REF] presented a classification of vulnerabilities with respect to the intrusion techniques and results. The taxonomy on intrusion techniques has three global categories (Bypassing Intended Controls and Active and Passive Misuse of Resources), which are subdivided into nine subcategories. The taxonomy on intrusion results has three broader categories (Exposure, Denial of Service and Erroneous Output), which are subdivided into two levels of subcategories.

More recently the Common Weakness Enumeration (CWE) [START_REF]The Common Weakness Enumeration[END_REF] was introduced as a dictionary of weaknesses maintained by the MITRE Corporation to facilitate the use of tools that can address vulnerabilities in software. The Open Web Application Security Project (OWASP) was also created to raise awareness about application security by identifying some of the most critical risks facing organizations. Even though these projects do not define themselves as taxonomies, their classification is ambiguous. For example, CWE-119 and CWE-120 are two separate weaknesses that address buffer overflows. Also, OWASP classifies injection and XSS as different categories, even though XSS concerns malicious code being injected into a web server.

There are also discussions about the theoretical and computational science of exploit techniques and proposals to do explicit parsing and normalization of inputs [START_REF] Bratus | Exploit Programming: From Buffer Overflows to[END_REF][START_REF] Samuel | Let's Parse to Prevent pwnage (invited position paper)[END_REF][START_REF] Crandall | Holographic Vulnerability Studies: Vulnerabilities as Fractures in Terpretation as Information Flows Across Abstraction Boundaries[END_REF][START_REF] Pieters | Vulnerabilities and responsibilities: dealing with monsters in computer security[END_REF]. Bratus et al. [START_REF] Bratus | Exploit Programming: From Buffer Overflows to[END_REF] discuss "weird machines" and the view that the theoretical language aspects of computer science lie at the heart of practical computer security problems, especially exploitable vulnerabilities. Samuel and Erlingsson [START_REF] Samuel | Let's Parse to Prevent pwnage (invited position paper)[END_REF] propose input normalization via parsing as an effective way to prevent vulnerabilities that allow attackers to break out of data contexts. Crandall and Oliveira [START_REF] Crandall | Holographic Vulnerability Studies: Vulnerabilities as Fractures in Terpretation as Information Flows Across Abstraction Boundaries[END_REF] discussed in a position paper the information-flow nature of software vulnerabilities.

In this work vulnerabilities are viewed as fractures in the interpretation of information as it flows in the system. It is not attempted to pinpoint a location for a vulnerability because they can manifest in several locations or semantic boundaries. Further, the primary goal of our taxonomy is to address ambiguity, which makes it difficult to reason about vulnerabilities effectively.

Conclusions

This paper presented a new vulnerability taxonomy that views vulnerabilities as fractures in the interpretation of information as it flows in the system. Notorious vulnerabilities are discussed in terms of the taxonomy's categories. A machine learning study evaluating the taxonomy is presented. Almost 71000 vulnerabilities were automated classified with an average success rate of 80%. The results showed the taxonomy's potential for unambiguous understanding of vulnerabilities. Lessons learned were discussed: (i) control-flow hijacking vulnerabilities represent more than 50% of all vulnerabilities reported, a trend that was not changed over the last decade, (ii) approximately 12% of recent vulnerabilities reports have insufficient information about the security flaw, (iii) the lack of standards in reporting makes it difficult to address vulnerabilities scientifically. This work will hopefully shed light on how the security community should approach vulnerabilities and how to best develop automatic diagnostic tools that find vulnerabilities automatically across layers of abstraction.

Table 3 :

 3 Manual classification of vulnerabilities.

Table 4 :

 4 Automated classification of vulnerabilities.

	Period Total Control-flow	Process	Side-channels Exhaustion Adversarial
	hijacking	confusion	accessibility

  Cisco Video Surveillance Manager does not require authentication for access to VSMC monitoring pages, allows remote attackers to obtain sensitive configuration information.

	Category	Database/ID Description (Abridged)
	Control-flow	NVD	2003-	"XSS vulnerability in member.php of XMBforum XMB allows re-
	hijacking	0375		mote attackers to insert arbitrary HTML and web script via the
				"member" parameter."
	Process con-	OSVDB 94899 "DirectAdmin Backup System contains a flaw as an unspecified
	fusion			email account function creates temporary files insecurely. It is
				possible for attacker to use a symlink attack against an unspec-
				ified file to gain elevated privileges".
	Side-channels OSVDB 95626 "WhatsApp Messenger contains a flaw triggered when attacker in-
				tercepts a payment request via a MiM attack ... allow the attacker
				to redirect user to arbitrary web page".
	Exhaustion	SF 58500		"IBM Integrator is prone to a DoS vulnerability. Remote attackers
				can exploit this issue to cause an application to consume excessive
				amounts of memory and CPU time, resulting in a DoS condition".
	Adversarial	NVD	2013-
	accessibility	3431

Table 5 :

 5 Examples of vulnerabilities automatically categorized by the classifier. invaluable to educate developers during the software development cycle and help the security community build a body of knowledge about the nature of vulnerabilities.


Enrico Bazzoli

Claudio Criscione

Federico Maggi

Stefano Zanero

XSS PEEKER: Dissecting the XSS Exploitation Techniques and Fuzzing Mechanisms of Blackbox Web Application Scanners

Black-box vulnerability scanners can miss a non-negligible portion of vulnerabilities. This is true even for cross-site scripting (XSS) vulnerabilities, which are relatively simple to spot. In this paper, we focus on this vulnerability class, and systematically explore 6 black-box scanners to uncover how they detect XSS vulnerabilities, and obtain useful insights to understand their limitations and design better detection methods. A novelty of our workflow is the retrofitting of the testbed so as to accommodate payloads that triggered no vulnerabilities in the initial set. This has the benefit of creating a systematic process to increase the number of test cases, which was not considered by previous testbed-driven approaches.

Introduction

Web application vulnerabilities are one of the most important and popular security issues, constantly making it to the top list of disclosed vulnerabilities [START_REF] Secureworks | Dell SecureWorks Threat Report for[END_REF][START_REF] X-Force | [END_REF]. Cross-site scripting (also known as XSS) is a prevalent class of web application vulnerabilities [START_REF]Open Web Application Security Project[END_REF]. In June 2013, XSS was reported as the most prevalent class of input validation vulnerabilities by far [START_REF] Tudor | Web Application Vulnerability Statistics[END_REF].

Black-box vulnerability scanners are widely used in the industry but, unfortunately, they can miss a non-negligible portion of vulnerabilities [START_REF] Doupé | Why johnny can't pentest: An analysis of blackbox web vulnerability scanners[END_REF][START_REF] Suto | Analyzing the accuracy and time costs of web application security scanners[END_REF][START_REF] Vieira | Using web security scanners to detect vulnerabilities in web services[END_REF] or report non-existing, non-exploitable or uninteresting vulnerabilities [START_REF] Vieira | Using web security scanners to detect vulnerabilities in web services[END_REF]. Although XSS vulnerabilities are relatively easy to discover, previous work showed that black-box scanners exhibit shortcomings even in the case of XSS flaws.

To our knowledge there is no detailed study of black-box web vulnerability scanners that focused specifically on XSS vulnerabilities and their detection approaches. Previous work and commercial benchmarks included XSS bugs as part of the set of flaws in testbed web applications, but not as the main focus. Also, previous work measured the detection rate and precision of the scanners mainly with the goal of benchmarking their relative performance. Although these indicators are important, we believe that providing precise insights on the structure, generality, fuzzing mechanism and overall quality of the XSS detection approaches could help web application developers to design better escaping and validation routines, and scanner vendors to understand the reasons behind scanner weaknesses.

These goals require a new perspective. A naïve approach would be to launch the scanners against large set of web applications, with difficult-to-find entry points, complex session mechanisms, etc. Normally, this kind of testbeds are adopted by vendors to demonstrate their product's sophistication. However, since our goal is not to challenge a scanner, but to analyze its payloads, exploitation approaches and fuzzing algorithms, we need a comprehensive set of targeted test cases that the scanner can easily find and exercise, such that the number of extracted payloads is maximized. From here the name of our tool, "XSS PEEKER," that implements our workflow.

XSS PEEKER works at the network level and decodes the HTTP layer to find and extract the XSS payloads. In addition, XSS PEEKER streamlines tedious tasks such as finding groups of related payloads, which we call templates, making automatic analysis and result visualization feasible even in the case of large amounts of payloads. Our research is not limited to individual scanners: we observed how cross-scanner analysis yields very interesting results.

Part of our contributions is a testbed web application, Firing Range (http:// public-firing-range.appspot.com) It is designed such that it is easy for the scanner to find the vulnerable entry points. Moreover, it is very easy to add new vulnerable entry points. Indeed, as part of our workflow, whenever XSS PEEKER encounters a payload that does not trigger any vulnerability, it displays an informative alert to the developer who can quickly prepare a test case that would satisfy specifically that payload. We applied this process iteratively, running new scans and collecting new payloads. Last, a difference of our testbed application with respect to the state of the art is the inclusion of specific test cases for DOM XSS vulnerabilities, one of the most recently discovered and emerging sub-classes of XSS flaws [START_REF] Heiderich | mXSS attacks: Attacking well-secured web-applications by using innerHTML mutations[END_REF][START_REF] Lekies | 25 million flows later: Large-scale detection of dom-based xss[END_REF].

Overall, our methodology and results are useful to answer questions such as: which payloads are used by security scanners to trigger XSS vulnerabilities? Can we rank them and identify a set of "better" payloads? Is a scanner exercising all the entry points with the expected payloads? A direct application of our results toward improving the quality of a scanner is, for example, to minimize the size and number of characters used in the payloads, while keeping the detection rate constant. Payloads of short size and with a smaller character set have in general higher chances of bypassing filters, and thus higher possibilities of uncovering unknown vulnerabilities. Using better payloads also means generating fewer requests and thus completing a test in a shorter time (i.e., better scanning efficiency). Clearly, these are just a couple of examples, which we formalize through a set of criteria later in the paper. This type of analysis should be focused on a scanner testing phase in isolation, which is best achieved with a fully synthetic test bed: a realistic test application would not improve the quality of our analysis in any of our measures and would introduce yet another experimental parameter to factor in all our considerations.

In summary, we make the following contributions:

-A publicly available testbed web application that exposes a wide range of non-trivial XSS vulnerabilities, augmented with the new cases progressively discovered while running the scanners. -A methodology to analyze how black-box web application scanners work by [START_REF] Bau | State of the art: Automated blackbox web application vulnerability testing[END_REF] extracting the payloads from the HTTP requests, (2) clustering them to scale down the evaluation challenge and keep it feasible, and (3) evaluating each cluster in terms of use of evasion, compactness, and other quality indicators. -A publicly available prototype of XSS PEEKER (https://goo.gl/zJrJ1i) -A detailed evaluation of 6 scanners: Acunetix 8.0, NetSparker 3.0.15.0, N-Stalker 10.13.11.28, NTOSpider 6.0.729, Skipfish 2.10b and w3af 1.2.

Background

Before describing our approach in detail, we introduce the essential background concepts and terminology on web application XSS vulnerabilities and black-box scanners. XSS attacks consist in the execution of attacker-controlled code (e.g., JavaScript) in the context of a vulnerable web application. In this paper, we refer to the portion of malicious code as payload. Without aiming for a complete taxonomy, XSS vulnerabilities and attacks can be divided in stored and reflected. In reflected attacks the victim is tricked (e.g., through links or short URLs [START_REF] Maggi | Two years of short urls internet measurement: Security threats and countermeasures[END_REF] embedded in e-mail or instant messages) into sending a specially crafted request-which embeds the actual payload, which is bounced back to the client immediately. In stored XSS attacks the moment the payload is injected is decoupled from the moment that it is effectively displayed and executed by the victim, as the attacker's code achieves some form of persistence.

DOM-based attacks [START_REF] Klein | DOM based cross site scripting or XSS of the third kind[END_REF] can be seen as an orthogonal category. They rely on the insecure handling of untrusted data through JavaScript (e.g., document.write('<script. ..>')) rather than static inclusion of payload (e.g., <script...>) in the rendered page.

Black-box web vulnerability scanners leverage a database of known exploits, including XSS payloads, used to trigger and detect potential vulnerabilities. They start by crawling the target web application to enumerate all reachable entry points (e.g., links, input fields, cookies), then they generate (mutations of) input strings based on their database, inject the resulting payload in the entry points and finally analyze the HTTP responses using an oracle to validate the presence of vulnerabilities (e.g., by looking for the injected payload in the output).

Firing Range: Test Case Generation

The implementation of the testbed web application is a key point. Given our goals and needs, the requirements of such a testbed are: (i) to have clearly defined vulnerabilities and entry points, (ii) to be easily customizable, (iii) to contain the main types of XSS vulnerabilities. Large, full-fledged testbed web applications have been implemented in previous works [START_REF] Bau | State of the art: Automated blackbox web application vulnerability testing[END_REF][START_REF] Doupé | Why johnny can't pentest: An analysis of blackbox web vulnerability scanners[END_REF][START_REF]Hacme Bank v2.0[END_REF] and are constantly made available to the public, but they do not entirely meet our requirements and approach. Although requirement (iii) is easy to ensure by modifying existing testbed applications, ensuring (i) and (ii) implies a complete redesign and re-engineering. In fact, existing testbed applications are not focused on extracting as many payloads as possible from the scanner. Contrarily, they are focused on challenging the scanner's capabilities of discovering hard-to-find vulnerable entry points. Given these premises, we decided that it was easier to implement our own testbed and release it to the community.

Design Challenges

Meeting the above requirements is tricky. On one hand, as pointed out in [START_REF] Doupé | Why johnny can't pentest: An analysis of blackbox web vulnerability scanners[END_REF], the complexity of some applications can hinder the coverage of the scanner. On the other hand, there is no guarantee of comprehensiveness of the testbed.

We decided to address these shortcomings explicitly while designing Firing Range. Our testbed exposes all vulnerabilities through HTML anchors, and vulnerable parame-ters are provided with sample input to improve discoverability. This approach allows to run testing sessions by providing the full list of vulnerable URLs, thus removing any crawling-related failure entirely. Furthermore, each vulnerable page is served as a standalone component with no external resources such as images or scripts, as to create a minimal, clean and efficient test case. The result is that the scanner focuses on the juicy part: the exploit generation and fuzzing.

Implementation Challenges

The main implementation challenge when creating a testbed is of course deciding which tests to include. We wanted our initial set to cover as many cases as possible, but there was no such list readily available in previous literature.

Since we basically wanted to test the detection of XSS on an HTML page, we observed that the HTML parsers in modern browsers have a finite set of states, which make a good starting point to create test cases. Thus, we created one test per HTML parser state. To this end we analyzed the different contexts identified by the contextual auto-escaper described in [START_REF] Samuel | Context-sensitive auto-sanitization in web templating languages using type qualifiers[END_REF]: simply reversing the perspective of a parser, tasked with applying proper escaping to malicious payloads, provided us with clear samples of the different contexts. We generated test cases covering each of them with the goal of (1) producing a "vulnerable baseline" of the different states of an HTML parser and (2) inducing the scanners to inject as many payloads as possible.

HTML contexts are however not enough to generate test cases for DOM XSSs, which exploits interactions between the DOM generated by parsing the original HTML and JavaScript code. For DOM XSS, we started from the XSS Wiki 1 , and other openly available collections of sample vulnerabilities, and generated a list of valid DOM sinks and sources-which, notably, include sources other than URLs such as cookies and browser storage. Each one of our DOM tests couples one of these sinks and sources. In the following example, the source is location.hash and the sink is innerHTML of a <div> node: We manually verified all the initial tests as exploitable with an appropriate payload or attack technique.

Iteratively Discovered Test Cases

During our experimental evaluation XSS PEEKER discovered payloads that were not exercising any test case (i.e., vulnerability). Instead of limiting our analysis to report this gap, our approach is to iteratively construct new test cases and progressively re-run all the scanners. In other words, our testbed application is dynamic by design. The details of this iterative procedure are described in Section 4.4, whereas the numbers of test cases that we had to add in order to accommodate an existing exploitation payload are reported in Section 5.4.

This iterative process produced 42 new test cases that were not identified by our initial seeding. In other word, we use the testbed web application as an oracle with respect to the scanner, and we use the scanner as an oracle with respect to the web application. As a result, this dual approach greatly improved the testbed and provided new insights on the internals of each scanner.

When considering XSS PEEKER's analysis perspective, there is no functional difference between stored and reflected XSS. The difference is whether the echoed payload is stored or not. However, from the point of view of the payload analysis, which is our core focus, reflected or stored XSSs are equivalent. Therefore, for ease of development and of experimental repeatability, our testbed web application only contains reflected vulnerabilities.

Our publicly available testbed includes full details of the vulnerabilities. For the sake of brevity, we refer the reader directly to http://public-firing-range.appspot. com for a complete list of live test cases and commented source code.

XSS PEEKER: Analysis Workflow

XSS PEEKER automates the extraction and analysis of XSS payloads by following an iterative approach, divided in four phases (the first three completely automated, the fourth partially relying on manual inputs).

Phase 1 (Payload Extraction)

The high-level goal of this phase is to obtain, for each scanner, the entire set of XSS payloads used by each scanner for each entry point in the testbed application. To this end, this phase first captures (using libpcap) the traffic generated by the scanners during the test and performs TCP stream reassembly to the full HTTP requests.

The first challenge is to automatically separate the HTTP requests used for the actual injection (i.e., containing one or more payload) from the requests used for crawling or other ancillary functionalities, which are not interesting. The ample amount of payloads generated makes manual approaches unfeasible. Therefore, we rely on two heuristics: Signature Payloads: Most scanners use signature payloads (i.e., payloads that contain strings that are uniquely used by that scanner). Therefore, we derived the signature payloads for each scanner and compiled a whitelist that allows this heuristic to discard the uninteresting requests. Attack Characters: Since we know the testbed application, we guarantee that there is no legitimate request that can possibly contain certain characters in the header or body parameters. These characters include, for example, <, >, ', ", and their corresponding URL-percent encodings. Such characters should not be present in a crawling request by construction, and since they are often required to exploit XSS vulnerabilities, we have empirically observed them as linked to vulnerability triggering.

To complement the previous heuristics and maximize the number of identified payloads, we perform pairwise comparisons between requests issued by each couple of scanners.

For each couple, we extract the body and URL of the two requests and check if they have the same path and the same query parameters. If so, we compare the values of each query parameter. By construction, Firing Range provides only a single value for each parameter, thus any mismatch has to be originated by the scanner fuzzing routine. Once a pair of requests is flagged as a mismatch, we performed manual inspection to isolate the payload. The number of such cases is rare enough to make this a manageable process. We iteratively applied and improved these heuristics until this cross-scanner analysis generated empty output, and we could confirm through manual inspection that no more test requests were missed (i.e., all payloads considered). We can focus just on query parameters because of the design of our testbed, which provides injection points exclusively on query parameters. Even if almost all scanners also test path injection, we chose not to analyze them. Indeed, manual analysis confirmed that scanners used the very same set of payloads observed during parameter injection.

Phase 2 (Payload Templating)

Given the large number of payloads generated by each scanner, manually analyzing and evaluating each of them separately is practically unfeasible. A closer inspection of the payloads, however, revealed self-evident clusters of similar payloads. For example, the following payloads: <ScRiPt>prompt(905188)</ScRiPt> and <ScRiPt> prompt(900741)</ScRiPt> differ only for the parameter value. To cluster similar payloads, inspired by the approach presented in [START_REF] Pitsillidis | Botnet judo: Fighting spam with itself[END_REF], we developed a recursive algorithm for string templating. Without going into the details, the approach presented in [START_REF] Pitsillidis | Botnet judo: Fighting spam with itself[END_REF] is to start from a set of template elements that produce fully random or dictionary based sequences of symbols. Using a large corpus of spam emails, the approach is to derive the full spam email template by generalizing the template elements. Emails, however, are much larger, structured, and richer of contextual symbols than our exploitation payloads. Therefore, the approach described in [START_REF] Pitsillidis | Botnet judo: Fighting spam with itself[END_REF] cannot be applied as is. Essentially, we cannot easily define the concept of "template elements" in such a small space. Therefore, as part of our contributions, we create a new approach that is well suited for short strings. In a nutshell, rather than following a top-down approach that starts from template elements, our idea is to elicit the template following a bottom-up approach, starting from the strings that it supposedly generated.

More formally, a template, in our definition, is a string composed by lexical tokens (e.g., a parenthesis, a function name, an angular bracket), that are common to all the payloads that generated it, and variable parts, which we represent with placeholders. The NUM placeholders replace strings that contains only digits, whereas the STR placeholders replace strings that contains alphanumeric characters. For instance, the template for the above example is <ScRiPt>prompt(90_NUM_)</ScRiPt>

To generate the templates we leveraged the Levenshtein (or edit) distance (i.e., the minimum number of single-character insertions, deletions, or substitutions required to transform string A to string B).

At each recursion, our algorithm receives as an input a list of strings and performs a pairwise comparison (without repetition) between elements of the input list. If the Levenshtein distance between each two compared strings is lower than a fixed threshold, we extract the matching blocks between the two strings (i.e., sequences of characters common to both strings). If the length of all matching blocks is higher than a given threshold, the matches are accepted. Non-matching blocks are then substituted with the corresponding placeholders. The output of each recursion is a list of generated templates. All payloads discarded by the Levenshtein or matching-block thresholding are appended to the list of output templates, to avoid discarding "rarer" payloads (i.e., outliers) and losing useful samples. The thresholds (maximum Levenshtein distance and minimum matching block length) are decremented at each cycle by an oblivion factor, making the algorithm increasingly restrictive. We selected the parameters, including the oblivion factor, through empirical experimentation, by minimizing the number of templates missed. This automatic selection yielded the following values: 20, 0.9 (default case); 20, 0.9 (Acunetix), 15, 0.5 (NetSparker); 15, 0.8 (NTOSpider); 15, 0.9 (Skipfish); 15, 0.5 (W3af). The algorithm stops when a recursion does not yield any new templates.

Phase 3 (Template Evaluation)

We want to assess the quality of payloads in terms of filter-evasion capabilities and amount of mutations used by the scanner. Given our observations above, we apply such evaluation to templates, as opposed to each single payload.

More precisely, the quality of a template is expressed by the following template metrics, which we aggregate as defined in Section 5.3. Note that the rationale behind each metric is explained on payloads, whereas the metric itself is calculated on the templates. M1 (Length), int: The longer a payload is, the easier to spot and filter (even by accident). Thus, we calculate the length of each payload template to quantify the level of evasion capability. M2 (Number of distinct characters), int: The presence of particular characters in a payload could hit server-side filters, or trigger logging. The presence of a character instead of another could reveal an attempt to mutate the string (e.g., fuzzing). A symbol can have different meanings depending on the actual context. From this rationale we obtain that a payload with a small set of characters is "better" than one leveraging rare characters. We calculate this metric on the variable part of each template (i.e., excluding the STR ad NUM tokens). M3 (Custom callbacks) bool: Rather than using standard JavaScript functions like alert, a scanner can use custom JavaScript function callbacks to bypass simple filters. We interpret this as an evasion attempt. If a template contains a function outside the set of built-in JavaScript functions, we set this metric to true. M4 (Multiple encodings), bool: Encoding a payload may let it pass unnoticed by some web applications' filters. However, some applications do not accept certain encodings, resulting in the application not executing the payload. A payload that uses multiple encodings is also more general because, in principle, it triggers more state changes in the web application. We set this metric to true if the template contains symbols encoded with a charset other than UTF-8 and URL-percent, thus quantifying the level of evasion. M5 (Number of known filter-evasion techniques), int: With this metric we quantify the amount of known techniques to avoid filters in web applications. For each template we calculate how many known techniques are used by matching against the OWASP list2 .

Although other metrics could be designed, we believe that these metrics are the bare minimum to characterize a scanner's capabilities and understand more deeply the quality of the payloads that it produces and process.

Phase 4 (Retrofitting Negative Payloads)

At the end of a scan, each scanner produces a report of the detected vulnerabilities. We use a report-parsing module that we developed (and released) for each scanner, and correlate the results with the payloads extracted. In this way we identify payloads that triggered vulnerabilities, which we call positive payloads and those that did not, called negative payloads.

We manually verified each negative payload to ensure that it was not our reportparsing module failing to correlate. We found that there are at least four reasons for which a negative payload occur: -The payload was malformed (e.g., wrong or missing characters, incorrect structure) and it was not executed. This is a functional bug in the scanner. -The payload was designed for a different context than the one it was mistakenly injected in. -The scanner used what appears to be the "right" payload for the test case, but the detection engine somehow failed to detect the exploit. -The payload was redundant (i.e., the scanner already discovered a vulnerability) in the same location thanks to another payload, and thus will not report it again. Since one of our goals was to create a testbed application as complete as possible, we wanted to ensure that all negative payloads had a matching test case in our application. With manual analysis, we proceeded to discard malformed and redundant payloads from the list of negative payloads. For each remaining negative payloads we produced a specific vulnerable test case.

To avoid introducing a bias in the results, we crafted each new test case to be triggered exclusively by the payload type for which it has been created, whereas the other payloads of the same scanner are rejected, filtered, or escaped. Of course, nothing would prevent other scanners from detecting the case with a different payload and that was indeed the intended and expected behavior.

Experimental Results

We tested 4 commercial scanners (in random order, Acunetix 8.0, NetSparker 3.0.15.0, N-Stalker 10.13.11.28, NTOSpider 6.0.729), for which we obtained dedicated licenses with the support of the vendors, and 2 open-source scanners (in random order, Skipfish 2.10b, and w3af 1.2). Because performing a comparative analysis is not the point of this paper, the scanners are weakly anonymized and they appear as Scanner1, Scanner2, etc., in our results.

We installed each scanner on a dedicated virtual machine (VM) to guarantee reproducibility and isolation (i.e., Debian 6.0.7 for Skipfish and W3af, and Windows 7 Professional for Acunetix, NetSparker, N-Stalker and NTOSpider). We used Wireshark to capture the traffic. When possible, we configured each scanner to only look for XSS vulnerabilities, and to minimize the impact of other variables, we left the configuration to its default values and kept it unchanged throughout all the tests. We tested Firing Range several times with each scanner. No scanner reported false positives, which is an expected result since we did not design any test cases to trick them like Bau et al. [START_REF] Bau | State of the art: Automated blackbox web application vulnerability testing[END_REF] did in their testbed application.

Of course, simply running scanners against a test application and analyzing their reports is not enough to evaluate their payloads. As Doupé et al. [START_REF] Doupé | Why johnny can't pentest: An analysis of blackbox web vulnerability scanners[END_REF] did in their study, we wanted to understand the behavior of a scanner in action to be able to explain their results. Our approach, however, differs noticeably since Doupé et al.'s main concern is about the crawling phase of the scanner, whereas we focus on the attack phase, and specifically on the payloads.

Payload Extraction Results

The number of extracted payloads for all scanners is shown in Fig. 1(a).

Since the detection rate is approximately the same for all scanners (on the first version of Firing Range, before Phase 4 (Retrofitting Negative Payloads)), the number of distinct payloads, shown in Fig. 1(a), is interesting: the detection technique of Scanner 3 uses far fewer payloads, while achieving the same detection of others. The comparatively larger number of payloads observed in the first 2 scanners is due to the use of unique identifiers tied to each of requests. Arguably, these identifiers are used to link a result back to the request that originated it even if server side processing had moved it around-this property is important when detecting stored XSS.

Recommendation (Baseline Payloads) Having a good, strong source of baseline payloads (e.g., from exploit databases 3 ) makes a significant difference in terms of payloadto-detection rate. Having a diverse set of distinct payloads is better (although harder) than having a small set of initial payloads used to generate many payloads by automatic mutation or fuzzing. This does not mean that scanner designers should not mutate the baseline payloads to generate new ones. Mutation and fuzzing are fundamental, but should not substitute the research effort by the vendor, who should ensure a rich supply of exploitation payloads whenever new vulnerabilities are discovered.

Payload Templating Results

The number of payloads alone, however, does not tell much about the actual quality and type of the payloads. More interesting conclusions about the fuzzying algorithm adopted by each scanner can be drawn by comparing . Indeed, after applying the clustering process of Phase 2 (Payload Templating), we notice immediately the limited number of templates (i.e., reflecting the attack patterns), as shown in Fig. 1(b).

The larger number of templates generated for Scanner 2 is an index of lower efficiency of the scanner, in terms of amount of requests and time spent for detecting a given set of vulnerabilities. In fact, while detecting the very same set of vulnerabilities as the other scanners, Scanner 2 employs 3-4 times the number of payload templates (i.e., distinct exploit patterns).

At this point in the analysis we could already see some scanners emerging as clearly more efficient due to the smaller number of templates they use. For example, Scanner 2 uses more complex payload templates such as:

-STR-'"--></style></script><script>alert(0x0000-STR-_NUM_) </script>

The templates for Scanner 4 are numerous and very different from each other, due to the wide variety of generated payloads. The (low number of) templates from Scanner 5 show that its payloads are significantly different from the rest. The templates that cover most of the payloads are:

-->">'>'"<obf000084v209637> .htaccess.aspx-->">'>'"<obf000085v209637> .htaccess.aspx-->">'>'"<obf000_NUM_v209637> -STR--->">'>'"<obf_NUM_v209637> which capture the scanner developer's particular interest in generic payloads that can highlight the incorrect escaping of a number of special characters at once. This approach is not found in any other scanner. Scanner 6 created a large number of templates, sign of strong use of non-trivial mutations and fuzzying.

Recommendations (Mutation and Context) From our results we learn that designing good mutation mechanism is not easy to implement. Naïve approaches such as those adopted by Scanner 1, which only appends a progressive number to "fuzz" the payload, do not pay back in more vulnerabilities detected. Instead, it is inefficient as it exercises the application with the very same (small) number of payload templates, which are actually all duplicate (except for the progressive number). This can be partially mitigated if the scanner implements an intelligent algorithm that figures out that N "very similar payloads" (e.g., same strings, except for 1-2 characters) are continuously generating the same (negative) result. Setting a limit on this is a simple yet effective technique to keep efficiency under control. Moreover, a contextual parsing approach is recommended to select a candidate subset of potentially successful payloads, before attempting the injection. Although having an identifier (e.g., incremental number) for each injected payload is useful (e.g., for matching stored XSS or multiple injection points in a single page), it should not be used alone as a fuzzying mechanism.

Template Evaluation Results

During this phase we evaluated each of the templates on the metrics defined in Section 4.3. Fig. 2 reports the mean of M1 (Length) calculated over the number of templates produced by each scanner. This is an interesting finding, which can be interpreted in two ways. On the one side, the length of the templates is in line with the minimum length of real-world payloads required to exploit XSS vulnerabilities, which is around 30 characters [START_REF] Gnarlysec | XSS and ultra short URLs[END_REF][START_REF] Toews | XSS shortening cheatsheet[END_REF], which somehow justifies the choice of the payloads. On the other hand, such a long string may fail to exploit a vulnerable entry point that simply cuts the payload off, even without a proper filter. Although this can be a good way for the scanner to avoid flagging unexploitable vulnerabilities (false positives), it has been shown that multiple small payloads can be combined to generate a full attack [START_REF] Mutton | XSS in confined spaces[END_REF]. However, the scanners that we examined miss these occurrences.

We notice that Scanner 1 employs significantly longer payloads than Scanner 5. This can be explained, considering that Scanner 5's M5 is zero, meaning that it uses no known filter-evasion techniques: thus, Scanner 5 is less sophisticated than Scanner 1.

Using M2-M5, we derived Table 1, which gives a bird's eye view on the use of mutations, filter evasion and use of callbacks from each scanner. Regarding mutations and callbacks, we use M3 (Custom callbacks) and M4 (Multiple encodings), respectively, whereas for filter evasion, if at least one template has a non empty character set (from M2), uses multiple encodings (from M4), and adopt at least one evasion technique (from M5) we conclude that the scanner performs filter evasion.

As it can be seen, both random mutations and filter-evasion techniques are widely employed in the scanners that we tested. Nevertheless, these techniques are ineffective at triggering all the vulnerable entry points. In fact, most of them yield poor-quality payloads, as detailed in Section 5.4. Instead, the use of custom callbacks over parsing or standard JavaScript functions is not common among the scanners that we tested. Recommendations (Payload Quality Ranking) This section answers one of the question that we stated in the introduction of this paper, which is actually one of the questions that motivated us to pursue this work: Can we rank [them] and identify a set of "better" payloads? Although the results of our experiments can be used to rank the scanners based, for example, on the quality metrics, unfortunately none of the scanners that we reviewed offer feedback to the analyst regarding the "quality" of the payloads that successfully triggered vulnerabilities. Although we can assume that it always possible for an attacker to send an arbitrarily crafted URL to the victim, certain payloads are more likely to pass undetected by network-or application-level protections (e.g., ModSecurity), which in case of legacy web applications are the only viable alternative to patching. Therefore, a successful payload with, say, a small character set, which adopts filter-evasion techniques while keeping a short length overall, should be ranked as "high risk". This additional feedback could be part of the typical vulnerability-ranking part that some vendors include in their reports.

Retrofitting Negative Payloads Results

XSS PEEKER's workflow iteratively adds new test cases in our testbed to account for payloads that triggered no vulnerabilities (i.e., negative payloads). This section answers one of the questions that we stated in the introduction, that is: "Is a scanner exercising all the entry points with the expected payloads?" The expected result was the reduction of the number of negative payloads to zero. So, we ran all the scanners against the new testbed and analyzed the results (note that the payloads of Scanner 1 were all already covered by our initial test case, thus we created no additional cases).

Unfortunately, as Fig. 3(a) shows, scanners failed to detect most of the new vulnerabilities. The most surprising finding is that the very same scanners that generated a negative payload would still fail to detect and report the new test case introduced for it, even if we manually confirmed that all of the negative payloads do in fact trigger an XSS on their respective new cases.

This may indicate a bugged response-parsing routine (i.e. a functional bug in the scanner). In other cases, by manually analyzing the requests toward the new cases, we discovered that some scanners did not exercise the test case with the "right" payload: a faulty (or random) payload-selection procedure somehow failed to choose it, using it instead in test cases where it would turn out to be ineffective; a different bug class in scanners. Another interesting result is that, after the introduction of the new cases, some scanners started using payloads we had not observed before. This behavior suggests some degree of context awareness, as scanners would only generate this new set of templates after having observed these new contexts. However, even in this case we observed a staggering high rate of failures for the new corner cases we added.

These findings would have been very difficult to reveal with a static set of vulnerabilities, as opposed to the incremental approach that we adopted. Fig. 3(b) shows the overall results produced by each scanner after including the new test cases. Although scanners did not achieve the expected results, this process allowed us to greatly increase our coverage of test cases for attacks supported by the analyzed scanners, and to produce a state of the art testbed for future work.

Recommendations (Continuous Testing) Although scanner vendors take testing very seriously, it is not trivial to account for the side-effect caused by adding new vulnerability cases to the testbed web applications. Adopting a retrofitting approach similar to the one that we implemented could be a first step toward finding corner cases (e.g., exercising a vulnerability with an incorrect payload) or bugs similar to the ones that we discovered.

Conclusions

This vertical study on XSS vulnerability scanners proposes quality metrics of 6 commercial and open-source products through passive reverse engineering of their testing phases, and manual and automated analysis of their payloads. Furthermore, we created a reusable and publicly available testbed.

By iterating on payloads that triggered no test cases, we were able to noticeably improve our test application and draw important conclusions about each scanner's inside workings. One of the key results is that, despite having some kind of awareness about context, all of the tested scanners were found wanting in terms of selecting the attack payloads and optimizing the number of requests produced. A significant number of detection failures suggest bugs and instability in the detection engines, while the high variance in types and features of the payloads we inspected makes the case for cooperation in defining common, efficient and reliable payloads and detection techniques.

  (a) Extracted payloads. (b) Generated templates.

Fig. 1 :

 1 Fig. 1: Output summary of Phase 1 and Phase 2, respectively.

  Fig 1(a) vs. Fig 1(b)

Fig. 2 :

 2 Fig. 2: Mean M1 (Length) over the templates of each scanner.

  (a) "New" test cases alone. (b) Overall comparison.

Fig. 3 :

 3 Fig.3: Summary of final results after Phase 4. "New", "Base" (initial test cases), "Detected" (true positive), and "Not Detected" (false negative).

  We then varied the sources and sinks, obtaining test cases like the following ones:

	var payload = window.name;
	eval(payload);
	</script>
	</body>
	<body>
	<script>
	var payload = document.documentURI;
	document.write(payload);
	</script>
	</body>
	DOM XSS from window.name to eval().
	<body>
	<script>
	1 https://code.google.com/p/domxsswiki/wiki/Introduction

DOM XSS from location.hash to innerHTML. <body> <script> var payload = window.location.hash.substr(1); var div = document.createElement('div'); div.id = 'divEl'; document.documentElement.appendChild(div); var divEl = document.getElementById('divEl'); divEl.innerHTML = payload; </script> </body> DOM XSS from documentURI to document.write().

Table 1 :

 1 Summary of template evaluation.

	3 http://exploit-db.com/search/?action=search&filter_description=cross&
	filter_platform=0&filter_type=6

https://www.owasp.org/index.php/XSS_Filter_Evasion_Cheat_Sheet