Mehrnoosh Shakarami
email: mehrnoosh.shakarami@my.utsa.edu

Ravi Sandhu
email: ravi.sandhu@utsa.edu

Refresh Instead of Revoke Enhances Safety and Availability: A Formal Analysis

Keywords: ABAC, Refresh, Consistency, Safety, Availability

Due to inherent delays and performance costs, the decision point in a distributed multi-authority Attribute-Based Access Control (ABAC) system is exposed to the risk of relying on outdated attribute values and policy; which is the safety and consistency problem. This paper formally characterizes three increasingly strong levels of consistency to restrict this exposure. Notably, we recognize the concept of refreshing attribute values rather than simply checking the revocation status, as in traditional approaches. Refresh replaces an older value with a newer one, while revoke simply invalidates the old value. Our lowest consistency level starts from the highest level in prior revocation-based work by Lee and Winslett (LW). Our two higher levels utilize the concept of request time which is absent in LW. For each of our levels we formally show that using refresh instead of revocation provides added safety and availability.

Introduction

In Attribute-Based Access Control (ABAC), access decisions are made based on attribute values of subjects, objects and environment with respect to a given policy. Attribute values for subjects and objects are typically provisioned by an Attribute Authority (AA) and presented in credentials as name, value pairs. A credential must be trustworthy, perhaps by a cryptographic signature or trusted delivery. Attribute values are susceptible to change. Ideally the decision point should know real-time values, which is practically impossible due to inherent delays of distributed systems and performance costs. This can lead to granting access when it should be denied (safety violation) or denying access when it should be granted (availability violation). The longer the gap between updates of credentials, the higher the risk of relying on stale attribute values.

In this paper we formally characterize three increasingly strong levels of consistency to restrict the exposure of the decision point to stale attribute values. For simplicity, we develop our formalism based on changing subject attribute values. Extension to changing object and environment attribute values is straightforward. Extension to policy changes is more subtle. Policy changes may require additional credentials come into play. While acquiring these additional credentials the policy may change again. In principle, this could lead to an infinite regress. In practice such an infinite regress is unlikely. Policies composed of multiple sub-policies specified by different authorities also raise issues of policy conflicts [START_REF] Cheminod | Toward attribute-based access control policy in industrial networked systems[END_REF][START_REF] Lupu | Conflicts in policy-based distributed systems management[END_REF]. A formal treatment of policy changes is beyond our scope.

The closest prior work is by Lee and Winslett (LW) [START_REF] Lee | Safety and consistency in policy-based authorization systems[END_REF][START_REF] Lee | Enforcing safety and consistency constraints in policybased authorization systems[END_REF]. Our paper is inspired by LW but presents a completely new perspective by considering refresh instead of revocation. We build our levels on top of the highest consistency level of LW, recasting it in the refresh framework. Taking request time into account, we propose two higher consistency levels not available in LW.

Our main contribution is to develop a formal framework for safety, availability and consistency problems of ABAC systems, via introducing the refresh scenario instead of the traditional revocation check. As we will show, this enhanced possibility of getting a new value rather than an invalid response enhances safety and availability. We also define the concept of being satisfactory for an attribute value with respect to a policy, which is first introduced in our work to the best of our knowledge. Relying on the history of satisfactory attribute values, we introduce additional flexibility to grant access to authorized users.

The paper is organized as follows. A review of related work and a comparison to LW is given in Section 2. Section 3 documents our system model and assumptions. The formalism of our consistency levels along with guaranteed properties by each specification is given in Section 4. Limitations and practical implications are discussed in Section 5. Section 6 concludes the paper.

Related Work

There is a rich body of research work on consistency in distributed systems [START_REF] Adya | Weak consistency: a generalized theory and optimistic implementations for distributed transactions[END_REF][START_REF] Bernstein | Concurrency control in distributed database systems[END_REF][START_REF] Harding | An evaluation of distributed concurrency control[END_REF][START_REF] Perrin | Distributed systems: concurrency and consistency[END_REF][START_REF] Steen | Distributed Systems[END_REF]. Many access control models are not completely compatible with distributed systems in that they are not deployed for such systems in the first place [START_REF] Hu | Access control for emerging distributed systems[END_REF]. ABAC is well adjusted to distributed environments due to its flexibility and granularity. In this paper, we consider an ABAC model to be in place and define consistency as communicating credentials' updates as quickly as possible to the decision point. To the best of our knowledge there is very limited directly related research in this arena. Especially there is no work done toward utilizing the refresh operation to obtain recent information.

The closest to our research is LW in trust negotiation environments [START_REF] Lee | Lightweight consistency enforcement schemes for distributed proofs with hidden subtrees[END_REF][START_REF] Lee | Safety and consistency in policy-based authorization systems[END_REF][START_REF] Lee | Enforcing safety and consistency constraints in policybased authorization systems[END_REF]. Another closely related research is on stale-safety which tries to safely useis stale attributes [START_REF] Krishnan | Stale-safe security properties for group-based secure information sharing[END_REF][START_REF] Krishnan | Authorization policy specification and enforcement for group-centric secure information sharing[END_REF]. Although the problem is similar, it mainly differs from our work since it has been applied in a non-ABAC, single authority environment. Policy staleness of cloud transactions proposed in [START_REF] Iskander | Enforcing policy and data consistency of cloud transactions[END_REF].

Ciphertext-Policy Attribute-Based Encryption [START_REF] Chase | Multi-authority attribute based encryption[END_REF][START_REF] Chase | Improving privacy and security in multi-authority attributebased encryption[END_REF] is broadly applicable in decentralized multi-authority environments, but presents challenge to handle attribute revocation [START_REF] Sciancalepore | On the design of a decentralized and multiauthority access control scheme in federated and cloud-assisted cyber-physical systems[END_REF][START_REF] Yang | Attributed-based access control for multi-authority systems in cloud storage[END_REF][START_REF] Yang | Expressive, efficient, and revocable data access control for multiauthority cloud storage[END_REF][START_REF] Yang | DAC-MACS: Effective data access control for multiauthority cloud storage systems[END_REF]. Moreover, it imposes a heavy performance burden which makes it impractical [START_REF] Garrison | On the practicality of cryptographically enforcing dynamic access control policies in the cloud[END_REF]. There are other researches concerning the policy consistency in distributed environments [START_REF] Garrison | On the practicality of cryptographically enforcing dynamic access control policies in the cloud[END_REF][START_REF] Iskander | Enforcing policy and data consistency of cloud transactions[END_REF][START_REF] Myers | X.509 internet public key infrastructure online certificate status protocol-OCSP[END_REF][START_REF] Zahoor | Authorization policies specification and consistency management within multi-cloud environments[END_REF] focusing on cloud environments. In this paper, policy assumed to be known with high assurance. There are research works utilizing revocation in authenticated dynamic dictionaries [START_REF] Goodrich | Authenticated dictionaries for fresh attribute credentials[END_REF][START_REF] Reyzin | Improving authenticated dynamic dictionaries, with applications to cryptocurrencies[END_REF][START_REF] Tamassia | Independently verifiable decentralized role-based delegation[END_REF], which enable dissemination of information from a secure central repository to multiple recepients.

Comparison to LW Model LW presented the first organized work on consistency in trust negotiation systems. They proposed four consistency levels based on timeliness of credentials revocation checks. In common with our model they considered every credential to have its lifetime specified by start time and end time. While all levels in LW model utilize the notion of receive time of credentials, we are agnostic to it. We consider decision time as central and utilize it explicitly in all levels, whereas in LW it is explicit only in top two levels. Revocation check in LW is replaced with refresh in our model, as will be discussed in next section. An alternate formulation of LW without use of receive time is given in [START_REF] Shakarami | Safety and consistency of subject attributes for attribute-based pre-authorization systems[END_REF], which includes an additional level based on request time.

Figure 1-(a) shows the levels in LW which are partially ordered. We do not recommend using incremental and internal levels since in both cases decision point may use a credential which is known to be expired or revoked. Our proposed levels are shown in Figure 1-(b) with a total order among levels. We set LW's highest level as the base level in our definitions. By taking request time into account, we propose two additional stronger levels of consistency. We provide further availability in our model by letting the decision point consider valid authorization should the current and cached values of relevant credentials be satisfactory, as defined in following sections.

System Model and Assumptions

We assume an ABAC authorization system in a distributed multi-authority environment. For a particular access request, there is a single decision point which determines whether or not the access is allowed by the access control policy based on attribute values. For convenience we use the terms attribute and attribute value interchangeably. The main focus of this paper is to limit the exposure of the decision point to outdated attributes by enforcing timeliness of checking subjects' attributes freshness. time of kmax(t)-th refresh of ci t i start,kmax(t) attribute start time of ci after kmax(t)-th refresh t i end,kmax(t) attribute expiration time of ci after kmax(t)-th refresh

Refresh vs. Revocation

Subject attributes might change during credential lifetime. A change could be a new value, a new lifetime or a premature revocation. In all cases the decision point needs to be updated about the latest changes of the attribute through either revocation or refresh. In revocation, AA would represent the current status of the credential as either Valid (no change) or Invalid (otherwise). However, with refresh AA can indicate the credential's status as Still-Good, New-Value or Invalid. Still-Good and Invalid correspond to Valid or Invalid in revocation scenario. New-Value reflects any change in credential's start time, end time or new value. So, Invalid status in revocation splits in two possibilities of Invalid and New-Value in refresh (see Figure 2-a). Thereby, refresh can allow more accesses than revoke and deny fewer accesses (see Figure 2-b).

Refresh function is defined as follows. T is the set of possible time stamps and C represents the set of all credentials in the system. Table 1 defines the symbols used in this definition and throughout the paper.

Refresh : C × T → {Invalid , Still -Good , New -Value} (1) Refresh(ci, t) =                Invalid ⇐⇒ (t ≥ t i end,kmax(t) ) ∨ (t ≥ t i revoc ) N ew-V alue ⇐⇒ (t i start,kmax(t) = t i start,kmax(t)-1 ) ∨(t i end,kmax(t) = t i end,kmax(t)-1 ) ∨ (val i kmax(t) = val i kmax(t)-1 ) Still-Good ⇐⇒ (t i start,kmax(t) = t i start,kmax(t)-1 ) ∧(t i end,kmax(t) = t i end,kmax(t)-1 ) ∧ (val i kmax(t) = val i kmax(t)-1 )
Following example highlights the benefits provided by considering refresh rather than revocation. Although granting illegitimate access is considered as a greater risk in many systems, availability is also important in which a legitimate user should not be denied access.

Example 1. Authorization policy in a coding company grants read access to a project's code to managers and test engineers and read/write access to developers. Alice was a test engineer. But her role has changed to a developer in the same project. Subsequently she submits a write request to the decision point. In revocation, checking her cached role credential results in Invalid response since she is no longer a test engineer. So her request would be denied. In refresh, however, New-Value response along with a new credential asserting her new role would be returned and access would be granted, as it should be based on policy.

Claim. If a subject can proceed to utilize a requested access in a revocation scenario, it can proceed in a refresh scenario as well. But there are scenarios in refresh-based systems which let the subject proceed, whereas it would be denied in revocation-based systems.

Proof. If nothing changed about a required credential, revocation and refresh would return Valid/Still-Good respectively. So, the first part of the claim follows. For the second part, it is possible that a required credential has changed with respect to start/end time or the value. So AA response in revocation scenario will be Invalid which prohibits subject's access. However with refresh the response would be New-Value, so access would be granted (see Figure 2).

System Assumptions

Without loss of generality, we suppose that the policy is stated in Disjunctive Normal Form (DNF), which is the disjunction of different conjuncts. The decision point tries to find the first conjunct which satisfies the desired level of consistency. This conjunct is called the View of the decision point at any specific time t with respect to the policy P which we denote as V P,t DP . We assume the decision point can instantaneously check the policy and identify the view. Definition 1. At any time t, we call the set of subject's attributes included in V P,t DP as the relevant credentials.

We make following assumptions in this paper.

1. Attributes do not change as the result of attribute credentials usage, that is we assume attributes to be immutable in sense of [START_REF] Park | The UCONABC usage control model[END_REF]. 2. We will not utilize any expired credential. If any required credential is beyond its end time, decision point polls AA to get a new credential for the attribute. 3. We do not refresh any credential after it has been found to be Invalid. 4. There is one instantaneous decision time (t d ) and one instantaneous request time (treq). 5. V P,t d DP is the only view of our interest as described above. 6. If refresh returns a New-Value result, its start time cannot be prior to its previous start time, i.e., t i start,k ≥ t i start,k-1 . 7. AA will not return a credential along with New-Value which has not been started yet, so,

t i ref,k ≥ t i start,k .
4 Consistency Levels Formal Characterization

Preliminaries

Satisfactory Values We define an attribute to be satisfactory if and only if its value fulfills the policy conditions. For instance if the policy requires the security level to be at least 3, any security level credential with the value greater than or equal to 3 is considered as satisfactory. Obviously the same credential may not be satisfactory with respect to another policy. We formally define satisfactory with respect to a policy P at the specific time t as follows.

Definition 2. The view at time t has the structure V P,t DP = 1≤i≤n F (i) in which F (i) is an atomic expression specifying required conditions for ci's value. We define Sat as follows to determine satisfactory requirements for ci's value.

Sat P,t c i = True ⇐⇒ F (val i kmax(t) ) = True (2)
Freshness We rely on the freshness concept in refresh scenario, compared to validity in revocation scenario. We formally define freshness via Fresh function as follows. When Fresh is used in a boolean expression, we understand Fresh(ci, t) to be False when its value is Unknown.

Fresh : C × T → {True, False, Unknown} Fresh(ci, t) =                True ⇐⇒ (t i start,k ≤ t ≤ t i ref,k ) ∧ (Refresh(ci, t i ref,k ) = Invalid ) Unknown ⇐⇒ (t i ref,k < t < t i end,k ) ∨ (t ≥ t i ref,kmax(t) ) False ⇐⇒ [(t ≥ t i ref,k ) ∧ (Refresh(ci, t i ref,k = Invalid ))] ∨[t ≥ t i end,kmax(t) ]) (3)
Following example is used throughout the paper. We now introduce three levels of consistency taking both old and new values of relevant credentials into account. We provide specifications and consequent properties guaranteed by each level in the rest of this section.

Interval Consistency

At this level, it is required to find overlap of freshness intervals (simultaneous freshness) of relevant credentials before the decision time. In Example 2, suppose Bob requests access to project documents on Jan 18th. Based on refresh results at Jan 15th, decision point finds simultaneous freshness of relevant credentials during Jan 10th-Jan 15th with satisfactory values. So the access will be granted. The stipulated overlap could be found for most recent refresh results of relevant credentials (Figure 3-(a)) or by considering both old and new refresh results (Figure 3-(b)). In these and subsequent figures if any refresh shown on the first line, it returns Still-Good while any other refresh returns New-Value. Moreover, in all cases the values of the three credentials are satisfactory. In Figure 3-(a) the overlap is for the most recent refreshed values, whereas in Figure 3-(b) the overlap is for a mix of the refreshed values, one new and two older. Specification. Every credential has been refreshed at least once before the decision time and found to be fresh. Most recent values of all relevant credentials are satisfactory with respect to the policy. Any overlap of freshness intervals for the freshest/cached credentials is acceptable so long as the values are satisfactory.

Interval (V P,t d DP ) ⇐⇒ (∃t ≤ t d )(∀ci ∈ V P,t d DP ) [ max ∀c j ∈V P,t d DP t j start,kmax(t) ≤ t i ref,kmax(t) < min ∀c i ∈V P,t d DP t i end,kmax(t) ∧ Fresh(ci, t i ref,kmax(t) ) ∧ Fresh(ci, t i ref,kmax(t d ) ) ∧ Sat P,t c i ∧ Sat P,t d c i ∧ max ∀c i ∈V P,t d DP t i start,kmax(t d ) < t d < min ∀c i ∈V P,t d DP t i end,kmax(t d ) ] (4)
Property 1. There is a time interval during which all relevant credentials were simultaneously fresh with satisfactory values with respect to the policy.

Proof. Based on Equation (4), there exists a time (t) prior to the decision time at which the latest refresh of every relevant credential happens after all have been started and before any of them ends. This implies all credentials are simultaneously fresh during [max Comparing with Revocation-Based Scenario Based on the Claim in Section 3.1, revocation and refresh are the same in case of Valid and Still-Good responses from AA. But if the result is New-Value, the corresponding revocation result would be Invalid which denies the access. In Example 2, if Bob requests access to the project's documents on Jan 25th and decision point rechecks the credentials, although Bob's role has changed, he would get the access in the refresh scenario whereas he would be denied in revocation scenario.

∀c i ∈V P,

Interval Consistency with Request Time

In first level, the decision point relies on what avails of previous refresh results for relevant credentials and access would be denied in case of any unrefreshed credential. By considering the request time we could compensate for missing refreshes. In Example 2, if Bob requests for accessing project's documents on January 14th, the access would be denied at first level since there is no refresh result available for required credentials. At second level, the decision point refreshes the credentials after the request time and then checks the consistency requirements. Figure 4 shows similar example where the top credential is refreshed after request time.

Specification. Decision point refreshes any credential with missing refresh results after the request time. Afterwards, relevant credentials should satisfy the interval consistency (previous level) requirements. In other words the policy will not frequently change in the system. Proof. Use of the same requirement of Interval (V P,t d DP ) guarantees the same property of freshness overlap of relevant credentials. Any missing refresh results would be compensated after request time. It is possible that the gap between the request time and decision time does not last enough to compensate all the lacking information, but we consider it as an administrative setting which is out of scope for this paper to quantify. Property 2. Every interval consistent view with request time satisfies the interval consistency requirements as well.

Proof. The proof is trivial since this level is defined based on interval level. Property 3. An interval consistent view may deny access allowed by interval consistent with request time.

Proof. Since we do not consider request time in first level, there is no opportunity to compensate possible missing refreshes which could enable access.

Comparing with Revocation-Based Scenario Considering the formal specification in Equation ( 5), which is based on first level, the comparison is trivial. If refresh is substituted with revocation, system's availability would decrease as discussed in Section 3.1. The same situation may happen with regard to Example 2 as discussed in Section 4.2.

Forward-looking Consistency

This level provides simultaneous freshness of all relevant credentials after the request time, considering both new and old credentials. Overlapping interval could Fig. 5. Forward Looking consistency either include the request time (Figure 5-(a)) or not (Figure 5-(b)). In Example 2, if Bob requests access to project's documents on Feb 1st his credentials would be refreshed afterwards revealing changes in role and security level leading to deny. However in previous levels an unauthorized access may be granted.

Specification. Any relevant credential has to be refreshed at least once after the request time. All relevant credentials have to to be found simultaneously fresh at or after the request time. Proof. Comparing Equation ( 5) and ( 6) shows forward-looking consistency is a restricted version of its preceding level, so the proof is trivial. Property 3. Not every interval consistent with request time view is necessarily forward-looking as well.

Proof. At second level of consistency, only some credentials need to be refreshed after request time to compensate for lacking information. Whereas in forwardlooking consistency, all have to be refreshed after the request time.

Comparing with Revocation-Based Scenario Changing credentials in revocation scenario leads to hinder the access, whereas in refresh scenario, the New-Value in case of any changes would let the subject proceed. In Example 2, Bob's request to access project's documents at Jan 20th would be denied in a revocation-based scenario, however in refresh scenario access would be granted.

Limitations and Practical Issues

We presented three levels of consistency, where each higher level provides enhanced availability and safety at the cost of refreshing more frequently. We compared qualitative benefits of each level. Quantifying cost-benefit is highly implementation and application specific, and is beyond the scope of this paper. Furthermore, there are issues related to manage the risks inherent to applying ABAC in a distributed environment, since ABAC introduces new challenges in selecting appropriate trust models [START_REF] Hu | Guide to attribute based access control (ABAC) definition and considerations[END_REF]. Finally, the formal correctness and appropriateness of the proposed criteria notwithstanding, the underlying information could be vulnerable to attack. The attack models would depend on the particular protocols and data structures used to implement credential transfer and refresh. As such they are out of scope for an abstract framework.

Conclusion

We formally characterize the safety and availability problem in multi-authority distributed ABAC systems. Our major contribution is to utilize the concept of refresh, which provides new attribute values rather than simply invalidating old ones. We propose three consistency levels which are totally ordered in strictness.

Fig. 1 .

 1 Fig. 1. (a) LW Revocation-Based Levels [15, 16] (b) Our Refresh-Based Levels

Fig. 2 .

 2 Fig. 2. (a) Revocation vs. Refresh (b) Comparing Grant vs. Deny

Fig. 3 .

 3 Fig. 3. Interval Consistency

Proposition 1 .

 1 d DP ) ⇐⇒ (∀ci ∈ V P,t d DP ) [t i ref,kmax(treq ) = ⊥ ∨ (∃tr treq < tr < t d ) Refresh(ci, tr)] ∧ Interval (VWe assume the set of relevant credentials would not change during the short gap between request time and decision time, so, V

Fig. 4 .

 4 Fig. 4. Interval Consistency with Request Time

ForwardLooking

Table 1 .

 1 Summary Table of Symbols

	Symbol	Meaning
	treq	request time
	t d	decision time
	ci	i th credential
	t i

revoc actual revocation time of ci (the AA always knows this time) t i ref,k time of k-th refresh of ci t i start,k attribute start time of ci after k-th refresh t i end,k attribute expiration time of ci after k-th refresh kmax(t) latest refresh of ci before time t (ci is determined by context) val i kmax(t) the value of ci after kmax(t)-th refresh t i ref,kmax(t)

  Part of this interval is located after the request time since refresh has been done after it.

			(V	P,t d	P,t d DP )[(treq < t i ref,kmax(t) )
	∧ ( max ∀c i ∈V P,t d DP	t i start,kmax(t) ≤ t i ref,kmax(t) < min DP ∀c i ∈V P,t d	t i end,kmax(t) )
	∧ Fresh(ci, t i ref,kmax(t) ) ∧ Fresh(ci, t i ref,kmax(t d ) ) ∧ Sat P,t c i ∧ Sat P,t d c i
	∧ max ∀c i ∈V P,t d DP	t i start,kmax(t d ) < t d < min DP ∀c i ∈V P,t d	t i end,kmax(t d ) ]
				(6)
	Property 1. There is a time interval during which all relevant credentials are
	simultaneously fresh after the request time.
	Proof. Based on Equation (6), all relevant credentials are simultaneously fresh
	during [max ∀c i ∈V ref,kmax(t) )]. Property 2. Every forward-looking consistent view is interval consistent with P,t d DP t i start,kmax(t) , min ∀c i ∈V P,t d DP t i
	request time as well.

DP ) ⇐⇒ (∃t treq < t ≤ t d )(∀ci ∈ V

Acknowledgements This work is partially supported by NSF CREST Grant HRD-1736209 and DoD ARL Grant W911NF-15-1-0518.


Hossein Shirazi

Bruhadeshwar Bezawada

Indrakshi Ray

Chuck Anderson

Adversarial Sampling Attacks Against Phishing Detection

Keywords: Phishing, Machine Learning, Adversarial Sampling, Classifiers

Phishing websites trick users into believing that they are interacting with a legitimate website, and thereby, capture sensitive information, such as user names, passwords, credit card numbers and other personal information. Machine learning appears to be a promising technique for distinguishing between phishing websites and legitimate ones. However, machine learning approaches are susceptible to adversarial learning techniques, which attempt to degrade the accuracy of a trained classifier model. In this work, we investigate the robustness of machine learning based phishing detection in the face of adversarial learning techniques. We propose a simple but effective approach to simulate attacks by generating adversarial samples through direct feature manipulation. We assume that the attacker has limited knowledge of the features, the learning models, and the datasets used for training. We conducted experiments on four publicly available datasets on the Internet. Our experiments reveal that the phishing detection mechanisms are vulnerable to adversarial learning techniques. Specifically, the identification rate for phishing websites dropped to 70% by manipulating a single feature. When four features were manipulated, the identification rate dropped to zero percent. This result means that, any phishing sample, which would have been detected correctly by a classifier model, can bypass the classifier by changing at most four feature values; a simple effort for an attacker for such a big reward. We define the concept of vulnerability level for each dataset that measures the number of features that can be manipulated and the cost for each manipulation. Such a metric will allow us to compare between multiple defense models.

Introduction

Motivation

Phishing, as defined in [START_REF] Zhang | A survey of cyber crimes[END_REF], is an attempt to obtain sensitive information such as user-names, passwords, and credit card details by masquerading as a trustworthy entity in an electronic communication. The first recorded mention of the term is found in the hacking tool against American Online (AOL) users in the 1995 named AOHell while the technique was elaborated earlier in a presentation to the International HP Users group, Interix, by Felix and Hauck in 1987 [START_REF] Felix | System security: a hacker's perspective[END_REF]. Phishing attacks have shown remarkable resilience against a multitude of defensive efforts, and attackers continue to generate sophisticated phishing websites that closely mimic legitimate websites. While there were 328,000 unique attacks reported in 2007, this number almost quadrupled by 2017 [START_REF] Apwg | Phishing attack trends report -3q 2018[END_REF].

Phishing, when viewed as a social-engineering attack, cannot be solved solely by educating the end users, and hence, automatic detection techniques are essential. Several defenses were proposed against phishing attacks, such as URL blacklisting, keyword-based filtering, IP address filtering, and machine learning based techniques. Solutions like URL-blacklisting are no longer effective as attackers can bypass such techniques through simple URL manipulation or by hosting websites on popular free hosting services on the Internet. Machine learning based techniques appear to be a promising direction.

Problem Statement

The studies in the existing literature emphasize on feature definition or enhancing the statistical learning models to discriminate between phishing and legitimate websites. The state-of-the-art solutions for phishing detection [START_REF] Niakanlahiji | Phishmon: A machine learning framework for detecting phishing webpages[END_REF][START_REF] Jiang | A deep learning based online malicious url and dns detection scheme[END_REF][START_REF] Pereira | Dictionary extraction and detection of algorithmically generated domain names in passive dns traffic[END_REF][START_REF] Shirazi | kn0w thy doma1n name": Unbiased phishing detection using domain name based features[END_REF][START_REF] Li | A stacking model using url and html features for phishing webpage detection[END_REF] use engineered features based on observations made by the research experts in this domain on publicly available datasets. One crucial assumption, in existing machine learning based phishing detection approaches, is that the training data collection process is independent of the attackers' actions [START_REF] Dalvi | Adversarial classification[END_REF]. However, in adversarial contexts, e.g. phishing or spam filtering, this is far from the reality as attackers either generate noisy data samples or generate new attack samples by manipulating features of existing ones. The noisy data samples result in a classification model with low accuracy and requires a higher effort for an attacker. The manipulation of features results is a more dangerous scenario wherein an attacker can bypass an existing classifier without much effort. In this work, we explore and study the effect of adversarial sampling on phishing detection algorithms in depth, starting with some simple feature manipulation strategies, and show some surprising results that demonstrate impact on the classification accuracy with trivial feature manipulation.

Proposed Approach and Key Contributions

We gathered four separate publicly available datasets developed by other researchers and applied adversarial sampling techniques to evaluate the robustness of the trained model against artificially generated adversarial samples. Although we do not show any solution to address this current threat, we show the vulnerability of the current approaches, and explored the robustness of the datasets against the engineered features, and the learning models. Our key contributions are as follows:

-We modeled the threat against the current defense and detection mechanism and explained the attackers' access and knowledge, which the attackers utilize to attack any given trained classifier model.

-We define the vulnerability level of phishing instances, which quantifies the attackers' efforts, and describe an approach to manipulate phishing instances and create new samples. -We surveyed a full range of phishing detection techniques focusing on the machine learning based approaches. We showed the weakness of some wellknown machine learning approaches and emphasized on how a phisher can generate new phishing website instances to evade a trained classifier in each of these approaches. -We built an experimental setup and conducted a wide range of experiments and analyzed how vulnerable the datasets and learning model are by testing against the adversarial samples.

The rest of this paper is organized as follows. In Section 2, we describe a wide range of defense mechanisms against phishing attacks in the literature. Also, we describe the various adversarial attacks against the machine learning classifiers in non-phishing domains. In Section 3, we model the threat from three points of view: attackers' goal, knowledge, and influence. In Section 4, we simulated adversarial sampling attack followed by the assessing vulnerability level and quantifying the cost of the attack. In Section 5, we explain the results of our experiments to prove the robustness of the classifiers and datasets against these attacks. In Section 6, we conclude the paper and discuss some future work.

2 Related Work

Machine Learning for Phishing Detection

For phishing website detection, machine learning algorithms are well suited as they can assimilate common attack patterns such as hidden fields, keywords, and page layouts, across multiple phishing data instances and create learning models that are resilient to small variations in future unknown phishing data instances. In the prior machine learning approaches, researchers engineered novel sets of features from diverse perspectives based on public datasets of phishing and legitimate websites. While these approaches have demonstrated excellent results for detecting phishing websites, they also suffer from serious disadvantages due to adversarial sampling as we show in the following discussion. Niakanlahiji et al. [START_REF] Niakanlahiji | Phishmon: A machine learning framework for detecting phishing webpages[END_REF] introduced PhishMon, a scalable feature-rich framework with a series of new and existing features derived from HTTP responses, SSL certificates, HTML documents, and JavaScript files. The authors reported an accuracy of 95% on their datasets.

According to a Symantec report [START_REF][END_REF], the number of URL obfuscation based phishing attacks was up by 182.6% in 2017. Some URL obfuscation techniques used by attackers are: misspelling of the targeted domain name, using the targeted domain name in other parts of the URL like the sub-domain, adding sensitive keywords like 'login', 'secure' or 'https' etc. Sahinguz et al. [START_REF] Ozgur | Machine learning based phishing detection from urls[END_REF] proposed a real-time detection mechanism based on Natural Language Processing (NLP) of URLs. The technique used a large dataset without requiring third-party services, and focused on features derived from URL obfuscation, and achieved an accuracy of more than 95%.

Verma et al. [START_REF] Verma | On the character of phishing urls: Accurate and robust statistical learning classifiers[END_REF] defined lexical, distance, and length related features for the detection pf phishing URLs. They employed the two-sample Kolmogorov-Smirnov statistical test along with other features to detect phishing websites. They conducted a series of experiments on four large proprietary datasets and reported an accuracy of 99.3% with a false positive rate of less than 0.4%.

Jiang et al. [START_REF] Jiang | A deep learning based online malicious url and dns detection scheme[END_REF] merged information from DNS and the URL to develop a Deep Neural Network (DNN) with the help of NLP to detect phishing attacks. While other approaches need to specify features explicitly, this method extracts hidden features automatically. The approach relies on the information from DNS and thus, requires third-party services.

Attackers use Domain Generation Algorithms (DGA) to dynamically generate a large number of random domain names for adversarial purposes including phishing attacks. Pereira et al. [START_REF] Pereira | Dictionary extraction and detection of algorithmically generated domain names in passive dns traffic[END_REF] introduced an approach for detecting such domains. These domains are considered as legitimate for detection mechanisms and human analysis. The authors used a graph-based algorithm to extract the dictionaries that have been used by attackers to detect malicious domains.

While these proposed approaches are promising, they often do not considering the page content. Attackers have full control over the URL except for the Second Domain Level (SLD), and thus, they can create any URL to bypass the classifier. Also, the content of the website is the most critical factor to lure the end-users rather than the URL or domain name themselves. Therefore, any solution not considering the website content would not be useful in the real world.

Tian et al., [START_REF] Tian | Needle in a haystack: Tracking down elite phishing domains in the wild[END_REF] studied five types of domain squatting over a large DNS dataset of over 224 million registered domains. They identified 657 thousand domains that potentially targeted 702 popular websites. Using visual and Optical Character Recognition (OCR) analysis, they created a highly accurate classifier and found more than one thousand new phishing instances of which 90% of them successfully evaded well-known blacklists even after one month. The authors combined two powerful techniques: domain squatting and OCR analyses on a large dataset. The advantage of this approach is in finding new instances that evaded the current classifiers. However, there is significant cost in keeping this information current.

Shirazi et al. [START_REF] Shirazi | kn0w thy doma1n name": Unbiased phishing detection using domain name based features[END_REF] observed two concerns with existing machine learning approaches: a large number of training features and bias in the type of datasets used. The study focused on the features derived from the domain name usage in phishing and legitimate websites and reported an accuracy of 97 -98% on the chosen datasets. To prove the performance of the whole model, they evaluated it with unseen phishing samples from a completely different source and achieved a detection rate of 99%.

Recently, Li et al. [START_REF] Li | A stacking model using url and html features for phishing webpage detection[END_REF] proposed an approach to extract the features from both URL and web page content and ran multiple machine learning techniques including GBDT, XGBoost, and LightGBM, in multiple layers, referred to as stacking approaches. The URL-based feature set includes eight features in total e.g. using IP address, suspicious symbols, sensitive vocabulary. The HTML based category includes features like Alarm Window, Login Form, Length of HTML Content. With The dataset has 20 features in total. The experiment has been conducted on three datasets, of which two are large ones with 50K instances and the accuracy is more than 97% in all cases. Although this approach is similar to recent machine learning approaches and does not use third-party services, it is similar to other previous work like [START_REF] Shirazi | kn0w thy doma1n name": Unbiased phishing detection using domain name based features[END_REF].

Learning in Adversarial Context

The proposed defense mechanisms in the literature widely employed machine learning techniques to counter phishing attacks. However, adversarial sampling attacks can threaten the current defense mechanisms. While there are some general analysis of the vulnerabilities of classification algorithms and the corresponding attacks [START_REF] Huang | Adversarial machine learning[END_REF], to the best of our knowledge, there is no other study on adversarial sampling in the context of the phishing attacks. Thus far, researchers studied and formulated these threats in a general manner or in other application contexts like image recognition. In the following, we briefly explore these efforts. Dalvi et al. [START_REF] Dalvi | Adversarial classification[END_REF] studied the problem of adversary learning as a game between two active agents: data miner and adversary. The goal of each agent is to minimize its cost and maximize the cost to the other agent. The classifier adapts to the environment and its settings either manually or automatically in this approach. The authors assumed that both sides, including data miner and adversary, have perfect knowledge about a problem. This assumption, however, does not hold in many situations. For example, in the phishing detection system, the adversary does not know the training set or the actual classification algorithm used. In Section 3, we modeled the adversary and elaborated why the adversary cannot have perfect knowledge. The attackers may directly or indirectly target the vulnerabilities in the feature selection procedure. Although the attackers might target the trained classification system, it still is an indirect attack on the chosen features.

Xiao et al. [START_REF] Huang Xiao | Is feature selection secure against training data poisoning?[END_REF] explored the vulnerabilities of feature selection algorithms under adversarial sampling attacks. They extended a previous framework [START_REF] Biggio | Security evaluation of pattern classifiers under attack[END_REF] to investigate the robustness of three well-known feature selection algorithms.

There are few approaches that create more secure machine learning models. Designing a secure learning algorithm is one way to build a more robust classifier against these attacks. Demontis et al. [START_REF] Demontis | Yes, machine learning can be more secure! a case study on android malware detection[END_REF] investigated a defense method that can improve the security of linear classifier by learning more evenly-distributed feature weights. They presented a secure SVM called Sec-SVM to defend against evasion attacks with feature manipulation. Wang [START_REF] Wang | Analyzing the robustness of nearest neighbors to adversarial examples[END_REF] theoretically guaranteed robustness of k-nearest neighbors algorithm in the context of adversarial examples. They introduced a modified version of k-nearest neighbor classifier while k is equal to 1 and theoretically guaranteed its robustness in a large dataset.

Finally, there are some tools for bench-marking and standardizing performance of machine learning classifier against adversarial attacks in the literature. Cleverhans [START_REF] Papernot | cleverhans v1. 0.0: an adversarial machine learning library[END_REF] is an open-source library that provides an implementation of adversarial sample construction techniques and adversarial training for image datasets. Given the lack of such bench-marking tools for the phishing problem, we tested our approach with our own attack strategies and implementation.

Threat Model

In this section, we model the adversarial sampling attack against machine learning based phishing detection approaches. We start with the attacker's goal, knowledge, and influence in general machine learning solutions, and then we explain them in the context of phishing problem. We model the adversarial sample generation for existing phishing instances based on the attacker's abilities and then evaluate the cost that the adversary has to pay for the successful execution of this attack. Finally, we define the vulnerability level for the dataset.

Attacker's Goal

Biggioa et al. explored three different goals for the attackers namely security violation, attack specificity, and error specificity [START_REF] Biggio | Wild patterns: Ten years after the rise of adversarial machine learning[END_REF]. The goal of an attacker in the security violation is to evade well-known security metrics including availability, privacy, and integrity. The attacker may violate the availability of the system by denial-of-service attack. In this case, if the system cannot accomplish the desired task due to the attacker's behavior, the availability of the service would be affected. The attacker needs to obtain sensitive and private information of users with approaches like reverse-engineering to violate the user's privacy.

In the phishing context, the adversary will attack the integrity of the system. The integrity is violated if the attack does not violate the regular system behavior; however, the attacker violates the accuracy of the classifier e.g. by luring classifier to label maliciously crafted phishing instances as legitimate to evade the classifier. The attack specificity depends on whether an attacker wants to mis-classify a specific set of samples (like phishing) or any given sample. The error specificity relates to the attacker's effort to increase a specific type of error in the system and degrade other classifier scores.

In this study, we consider that the adversary desires to attack the specificity of the learning model. This leads to the incorrect classification of the adversarial phishing samples as legitimate and thereby, these samples will deceive the end users. Also, with respect to error specificity, the adversary wants to decrease the True Positive Rate (TPR).

Attacker's Knowledge

An attacker may have different levels of knowledge about the machine learning model. An attacker might have detailed knowledge, i.e., white-box or perfect knowledge, minimal knowledge about the model called zero knowledge [START_REF] Huang Xiao | Is feature selection secure against training data poisoning?[END_REF][START_REF] Biggio | Wild patterns: Ten years after the rise of adversarial machine learning[END_REF] and limited knowledge about the model known as the gray-box. If the adversary knows everything about the learning model, parameters, and the training dataset including the classifier parameters, then the attacker has perfect knowledge. In the zero knowledge, the adversary can probe the model by sending instances and observing the results. The adversary infers information about the model by choosing appropriate data samples. In the limited knowledge, it is assumed that adversary knows about features and their representation, and the learning algorithm. However, the adversary does not know about the training set or the algorithm's parameters.

From the dataset point of view, the attacker may have partial or full access to the training dataset. Attacker may also have partial or full knowledge about the feature representation or feature selection algorithm and its criteria. In the worst case scenario, an attacker may know about the subset of selected features. In our study, we assumed that the adversary has limited knowledge. The adversary knows about the classifier model and the feature set but does not know about training set, the classifier, or classifier's training parameters.

Attacker Influence

Two major types of attacker influence have been defined in the literature namely poisoning and evasion attacks. The poisoning attack refers to the case where the adversary injects adversarial instances into the training phase. This injection leads to bypass data later on in the testing phase of the experiment or even at the practical usage of the model. For example, email providers use spam detection services to block emails that include a link to phishing websites. The email system gives the users ability to override the email's label e.g. re-labeling a spam email as non-spam to deal in cases of False Positive detection, The system benefits from user's labeling to improve the accuracy by updating the training set. However, in a poisoning attack, an attacker with an authorized email account in the system can re-label the correctly detected spam emails as non-spam to poison the training set of the classifier.

While the attacker does not have access to the training set in the evasion attack, it tries to intentionally and smartly manipulate features to avoid samples being labeled correctly by the classifier at the testing phase. Similar to the previous example on the spam detection system, a phisher may send an email with intentionally misspelled words to evade the classifier.

In this work, we assume that the attacker has a limited knowledge about the learning model, but has unlimited access to the predict function of the learning model. The attacker can test as many instances as needed and get the results. With this assumption, an attacker can create a large number of new samples and test them against the classifier to see if they can bypass the model. In the next section, we describe our adversarial sampling approach and outline our method for measuring the effectiveness of the samples in lowering the classifier's accuracy. Section 5 studies how this attack can be effective by showing the degradation of the classifier's score under this attack.

Adversarial Sampling for Phishing

We simulate the attacker's approach to generate new adversarial samples based on the existing phishing instances that are detected by the classifier. The adversary generates new instances based on these current phishing instances to check whether the generated instances are able to evade the classifier. We assume that the attacker has full control on the URL and web page content except for the domain name, which is unique. The attacker has limited knowledge about the classifier and features, as we discussed earlier.

Defining the Dataset

We use similar notation to that used in [START_REF] Huang Xiao | Is feature selection secure against training data poisoning?[END_REF]. The whole dataset has been generated by a procedure 𝒫 : 𝒳 ↦ → 𝒴. In the experiment, we defined two types of instances: Legitimate (𝐿) and Phishing (𝑃 ). A learning algorithm trains from this dataset and will label the new instances. Each instance in the dataset has 𝑑 features that are represented as a 𝑑-dimensional vector and is labeled as legitimate (𝐿) or phishing (𝑃 ).

𝑥 𝑖 = [𝑥 1 𝑖 , • • • 𝑥 𝑑 𝑖 ] 𝑇 ∈ 𝒳 (1)
Each instance relates to the target label of 𝑦 𝑖 ∈ 𝒴 , 𝒴 ∈ {0, 1}. We denote this set 𝐷 with 𝑛 samples as follows: 𝒟 = {𝑥 𝑖 , 𝑦 𝑖 } 𝑛 𝑖=1 . The set 𝒯 is the subset with 𝑡 instances that the adversary can access, 𝒯 ⊆ 𝒟 , 𝑡 ≤ 𝑛

Selecting Features for Manipulation

To specify a subset of features, we introduce the notation 𝛷 = {0, 1} 𝑑 , where each element denotes whether the corresponding feature has been selected [START_REF] Zhang | A survey of cyber crimes[END_REF] or not (0). The first step for creating adversarial samples is to select one or more features for manipulation. 𝛷 𝑠 denote the set of all possible combinations of 𝑠 features, ( 𝑛 𝑠 ) , that have been selected and 𝜋 𝑠 𝑖 denotes 𝑖 𝑡ℎ such choice of features. For example, 𝜋 3 1 = (0, 1, 1, 1, 0) means that, the first combination from 𝛷 𝑠 , chooses features 2, 3, and 4, for manipulation. We formalize this in Equation 2:

𝜋 𝑠 𝑖 ∈ 𝛷 𝑠 𝑤ℎ𝑒𝑟𝑒 𝑖 ∈ ( 𝑛 𝑠 ) 𝑎𝑛𝑑 𝑑 ∑ 𝑖=1 𝜋 𝑠 𝑖 = 𝑠 (2)
Assigning new feature values is the next step after defining the subset of features for manipulation. We assumed that each feature value may be replaced by values that appeared in existing phishing instances. The intuition is that, if the value has been found to be assigned to that feature previously for a phishing instance, then the feature is more likely to get that value again in another phishing instance. Let 𝑇 𝑖 denote the set of all values that have appeared for the feature 𝑖 among the phishing instances. For example, 𝑇 2 = {-1, 0, 1} denotes existing phishing instances have values -1, 0, and 1 in the second feature.

For generating new instances, first, we need to generate all possible feature combinations with different lengths and then, for each combination, we need to permute all possible feature values from 𝑇 𝑖 . This process is done only for phishing instances that have been predicted correctly by the classifier. Algorithm 1 explains this process for a given phishing instance. It shows how the adversarial instances will be generated based on an original input and desired features for manipulation. There are two inputs for Algorithm 1: an original phishing instance and the selected features to manipulate, and returns as output, the new adversarial instances that have been generated. In lines 2 and 3, the algorithm loops over all of the selected features for manipulation. The algorithm gets all available values for them from the array 𝑇 that is previously defined. The algorithm adds a series of available feature values to the list 𝐿. Line 5 calculates a Now, each row in 𝐿_𝑝𝑟 has the values for all of the selected features for manipulation. We make another loop over 𝐿_𝑃 𝑟 to assign new values to the original phishing instance 𝑥. The algorithm saves them in the result array of 𝑔𝑒𝑛𝑆𝑎𝑚𝑝𝑙𝑒𝑠.

Adversary Cost

Attackers have to handle two challenges for generating adversarial instances. From a machine learning point of view, the dataset includes vectors, but the attacker has to change the website in a way that it generates the desired vector similar to adversarial samples. This is not a trivial process, and it has considerable cost for the attacker. Whereas adversarial samples may have a higher chance of evading the classifier, but they may not be visually or functionally similar to the targeted websites. This increases the chance of being detected by the end-user. Thus, the adversary wants to minimize two parameters: the number of manipulated features and the assigned feature values. We consider this as a cost function for the adversary.

In the previous section, we discussed how the attacker controls the number of manipulated features, but it is not the only parameter. If the manipulated feature values are far from the original values, it will increase the chance of evading the classifier. We study this hypothesis in Section 5. But, this will also change the visual appearance or behavioral functionality of the website from the targeted website, thereby, increasing the chance of phishing website being detected by the end-user.

In this work, we used the Euclidean distance between the original phishing sample and newly generated sample to estimate the cost, a higher distance indicates a larger cost. Consider 𝑥 𝑖 to be a phishing instance and 𝑥 ′ 𝑖 a manipulated one based on the original 𝑥 𝑖 instance. Both are vectors of size 𝑛. The Euclidean distance between 𝑥 𝑖 and 𝑥 ′ 𝑖 will be calculated by Equation 3:

𝑑(𝑥 𝑖 , 𝑥 ′ 𝑖 ) = ⎯ ⎸ ⎸ ⎷ 𝑛 ∑ 𝑘=1 (𝑥 𝑘 𝑖 -𝑥 ′ 𝑘 𝑖 ) 2 (3)
If 𝑙 is the number of manipulated features to generate 𝑥 ′ 𝑖 from 𝑥 𝑖 , and 𝑑 is Euclidean distance between them, the total cost 𝑐 will be derived from this equation: 𝒞(𝑥 𝑖 , 𝑥 ′ 𝑖 ) = (𝑙, 𝑑). This tuple will be used to evaluate the total cost for generating the adversarial instances.

Vulnerability Level

A phishing instance that has been predicted correctly is vulnerable at the level of 𝑙 with the cost of 𝑑 if there is at least one adversarial instance that can bypass with 𝑙 manipulated features and distance 𝑑. We call this instance vulnerable if, by manipulating 𝑙 features and with the distance of 𝑑, it can bypass the classifier. The goal of the attacker here is optimizing the 𝑙 and 𝑑; a multi-objective optimization problem for the attacker. For example, if we have a phishing instance, which has been detected by the classifier, but there is the new instance generated by manipulating 3 features and a Euclidean distance of 2.7 bypasses the classifier, the original sample is vulnerable at the level of 3 with a cost of 2.7.

Experiments and Results

In this section, we show the effectiveness of our threat model and proposed adversarial sampling attack that degrades the accuracy and efficacy of existing learning models. First, we discuss the datasets utilized and then, we elaborate on three different experiments we have conducted and their results.

Used Datasets

We obtained four publicly available phishing datasets on the Internet and the details of these datasets are given below.

Dataset 1: DS-1: This set includes 1000 legitimate websites from Alexa.com and 1200 phishing websites from PhishTank.com; 2200 in total. Each instance in this dataset has eight features and all are related to the domain name of the websites. The features used are domain length, presence of non-alphabetic character in the domain name, the ratio of hyperlinks referring to the domain name, the presence of HTTPS protocol, matching domain name with copyright logo, and matching domain name with the page title. With these features, Shirazi et al. [START_REF] Shirazi | kn0w thy doma1n name": Unbiased phishing detection using domain name based features[END_REF] reported an ACC of 97-98% in the experiments, which is significantly high.

Dataset 2: DS-2: Rami et al. [START_REF] Rami M Mohammad | An assessment of features related to phishing websites using an automated technique[END_REF] created this dataset in 2012 and shared it with UCI machine learning repository [START_REF] Dheeru | UCI machine learning repository[END_REF]. This set includes 30 features and are categorized into five categories: URL based, abnormal based, HTML-based, JavaScript based, and domain-based features. This dataset includes 4898 legitimate instances from Alexa.com merged with 6158 phishing instances from Phish-Tank.com; more than 11000 in total making it the most extensive dataset that we have used in this study.

Dataset 3: DS-3: In 2014, Abdelhamid et al. [START_REF] Abdelhamid | Phishing detection based associative classification data mining[END_REF] shared their dataset on UCI machine learning repository [START_REF] Dheeru | UCI machine learning repository[END_REF]. This dataset includes 651 legitimate websites and 701 phishing websites; 1352 instances in total and includes ten features combination of third-party services and HTML based features for each instance. Authors report an ACC between 90%-95% in their experiments.

Dataset 4: DS-4: This dataset is the most recent dataset publicly available in the literature that we could find and was published in 2018. It has been created by Tan et al. [START_REF] Lin | Phishing dataset for machine learning: Feature evaluation[END_REF] and was published on Mendeley3 dataset library. This set contains 5000 websites from Alexa.com and as well as those obtained by web crawling, labeled as legitimate, and 5000 phishing websites from PhishTank.com and OpenPhish.com. The authors collected this data from January to May 2015 and from May to June 2017. This dataset includes 48 features, a combination of URL-based, and HTML-based features. While this dataset includes the URL length and it may bias the dataset as Shirazi et al. [START_REF] Shirazi | kn0w thy doma1n name": Unbiased phishing detection using domain name based features[END_REF] explained it, but we kept all features.

Table 1 summarises the number of instances, features, and the portion of legitimate vs phishing instances in each dataset. We have a dataset with a large number of instances, DS-2, and DS-4 with 11000 and 10000 respectively. We also have small dataset DS-3 with 1250 instances. With respect to the number of features, DS-1, with just seven features is a dataset with a limited number of features and in comparison, DS-4 with 48 features is a large dataset. Also, we used an unbiased dataset like DS-1 and used DS-4 as well though it may be biased concerning some of the features like URL length. Besides, the features in each dataset are selected from different points of view such as URL-based features in DS-2, DS-3, and DS-4, or domain-related features in DS-1, and HTML-Based features in DS-2 and DS-4. These variations validate our hypothesis in a stronger and more general sense. Also, it shows that adversarial sampling is a serious problem that may be happening in different situations and needs to be addressed.

Exp-1: Evaluation of Datasets

In the first experiment, we tested the performance of each dataset against a wide range of classifiers. The experiment is as follows. We labeled phishing websites in all datasets as +1 and legitimate websites as -1. We used five-fold cross-validation to avoid issues of over-fitting and to test the performance of the learning model against unknown data instance classification. We used six different classifiers namely Decision Tree Decision Tree (DT), Gradient Boosting (GB), Random Forest (RF) , K-Nearest Neighbors (KNN), and Support Vector Machine (SVM) with two different kernels: Linear (lin) and Gaussian (rbf) to make the comparison between classifiers. We repeated each experiment 10 times and reported the average and standard deviation of the results. Table 2 explains the achieved results in this experiment.

For DS-1, RF and GB both generate the highest ACC and the TPR in each classifier is almost the same. Also, DS-1 has the best average of TPR among all classifiers. This means that, despite different classifiers, the features are welldefined. RF gives the best TPR (94.25%) and ACC (95.76%) on DS-2. Interestingly, the DT does not generate a good TPR (86.77%).

The experiments on DS-3 dataset did not yield a high TPR or the ACC. Both GB and SVM with Gaussian kernel has the TPR of 87%, which are not that much good. The best ACC, for this dataset, is from GB, with 83%. The experiment on DS-3 gave very good results. Both GB and RF gave a TPR over 97% and accuracy of 97%, which are very high. Also, this dataset has the best average of ACC among different classifiers meaning this dataset performs very well with different types of classifiers. With six different classifiers, the experiments on both DS-1 and DS-4 show an average ACC of more than 94%, which is significantly high. This confirms that these datasets are well-defined and have a good set of distinguishing features.

We used a single metric of f1 to compare all classifiers and datasets together. Table 3 shows the best f1 score for each dataset with the classifier that has produced that result. It is evident from this table that both GB and RF generate the best results among all of the experiments, so we selected these two classifiers for the next experiments.

Generating Adversarial Samples

In each dataset, we reserved 200 phishing instances and then trained the model without the 200 reserved phishing instances. The generated adversarial samples need to be similar and valid to the phishing examples; otherwise, those cannot be assumed to be phishing instances. To assign new values to the features and generate new instances, we just used previously seen values in the phishing instances. With this strategy, it is guaranteed that the newly assigned value is valid and has already been seen in other phishing instances in the dataset. We discussed this process earlier in Section 4. We randomly selected combination of features, up to four different features, and changed the values of each feature with all possible feature values.

After creating each new sample, we tested our new sample against the selected classifier and checked whether it could bypass the classifier or not. If it did, we consider the original phishing instance to be a vulnerable instance. We calculate the distance between the new instance and the original one to find the closest instance that can bypass the classifier.

Exp-2: Robustness of Learning Model

This experiment studies the robustness of datasets and learning models against generated adversarial samples. We selected one classifier that performs best for each dataset based on the f1 score from Table 3. For the datasets DS-1 and DS-3, we selected GB and RF for DS-2 and DS-4.

In this experiment, we counted the number of reserved phishing instances that are vulnerable. This means that, there should be at least one optimized manipulated instance based on the original sample that can bypass the classifier. With small perturbation on these instances, they can bypass the classifier and elude the users to release their critical information. Based on our hypothesis, these are vulnerable instances and can be assumed as a threat to the learning model. We repeated each experiment ten times and reported the average of the results.

Figure 1 shows the results of Exp-2. While the x-axis shows the number of manipulated features, zero manipulated feature means that the test happened with the original phishing instances without any perturbation. The trend of results reveals that by increasing the number of perturbation, the number of evaded samples increase proportionally. We continued increasing the perturbed features for up to four different features at a time. We observed that with four features, almost all phishing instances bypass the classifier model.

For example, Figure 1 shows that less than 4% of phishing instances in DS-1 can bypass the classifier without any perturbation. With only one manipulated feature, more than 20% of phishing instances can bypass the classifier. With two manipulated features, almost all of instances can bypass the GB. The results are almost the same for other datasets. In another case, while just 12% of original phishing instances (the instances without any changes) have been misclassified in DS-3, the results significantly go up to 65% with only one perturbed feature.

This experiment shows how vulnerable the machine learning models are for the phishing problem. Small perturbation on features can bypass the classifier and degrade the accuracy significantly.

Exp-3: Dataset Vulnerability Level

In this experiment, we studied the cost that an adversary has to pay to bypass a classifier. From an adversary point of view, it is not inexpensive to manipulate an instance with new feature values and bypass the classifier. In Section 4.3, we assessed the cost and in Section 4.4, we defined the term vulnerability level for one instance. Similar to previous experiment 5.4, we reserved 200 phishing instances from each dataset and chose the classifier for each dataset based on Table 3. For datasets DS-1 and DS-3, we chose GB while we chose RF for both DS-2 and DS-4 datasets. Averaging the vulnerability level for each of the 200 selected instances and repeating the experiment ten times, we assessed the vulnerability level for the whole dataset.

Figure 2 presents the results of this experiment for all datasets for two parameters: the number of manipulated features and the average cost of adversarial instances. It is evident that, by increasing the number of manipulated features, the cost also increases steadily. For example, for the dataset DS-1, the average cost, for adversarial samples, with one manipulated feature is 0.95 and with four manipulated features the cost is 3.93.

Furthermore, it is clear that the average cost for some datasets is more than that of other datasets. For example, in the DS-4, the adversary has to pay more cost especially when the number of features increases to three and four in comparison to the other datasets. This shows that this dataset is more robust against these attacks and has a lower vulnerability level. Also, it is clear that with one single feature manipulation with a small cost, it is possible to bypass a classifier. This needs to be considered when a dataset and features are designed.

Comparing the results with previous experiments

In this section, we compare our approach with some of the previous researches in this field. Table 4 compared nine different approaches in the literature. We summarized the pros and cons of each approach and show the dataset size and best accuracy results of each approach. We studied a wide range of previous efforts by focusing on machine learning techniques. Some of the techniques solely focused on the URL itself [START_REF] Ozgur | Machine learning based phishing detection from urls[END_REF][START_REF] Tian | Needle in a haystack: Tracking down elite phishing domains in the wild[END_REF] but others look at both URL and the content of the page [START_REF] Shirazi | kn0w thy doma1n name": Unbiased phishing detection using domain name based features[END_REF][START_REF] Bulakh | Countering phishing from brands' vantage point[END_REF]. The use of third-party services is another difference between approaches. While using third-party services like search or DNS inquires leverage the feature set and make the feature set more reliable it also endangers the privacy of the users. Third-party enquiries to fetch the feature value reveal the browsing history of the end-users The previous studies have been done on variable sizes of datasets. While some of the datasets have less than 5 thousand records [START_REF] Shirazi | kn0w thy doma1n name": Unbiased phishing detection using domain name based features[END_REF][START_REF] Bulakh | Countering phishing from brands' vantage point[END_REF], there are also datasets with millions of instances [START_REF] Jiang | A deep learning based online malicious url and dns detection scheme[END_REF][START_REF] Tian | Needle in a haystack: Tracking down elite phishing domains in the wild[END_REF]. Also, for approaches analyzing just the URL without the webpage content, creating massive datasets are easier. Most of the approaches achieved high accuracy of over the 95%. Both [START_REF] Pereira | Dictionary extraction and detection of algorithmically generated domain names in passive dns traffic[END_REF][START_REF] Verma | On the character of phishing urls: Accurate and robust statistical learning classifiers[END_REF] achieved accuracy of 99%, which is significantly high. Tian et al. [START_REF] Tian | Needle in a haystack: Tracking down elite phishing domains in the wild[END_REF] found new phishing samples that were not detected by common phishing detection mechanisms even after one month. We also added the results of this study to Table 4. We trained the classifier on the four public datasets and achieved very high accuracy. When we added the manipulated features in the testing phase, the accuracy degraded significantly and finally became zero. These experiments prove that our proposed attack is sufficient to evade existing classifiers for phishing detection.

Conclusion and Future work

In this work, we explained the limitation of machine learning techniques when adversarial samples are taken into consideration. We introduced the notion of vulnerability level for data instances and datasets based on the adversarial attacks and quantified it. We achieved high accuracy in the absence of this attack using seven different well-studied classifiers in the literature: more than 95% for all classifiers except one that had 82%. However, when we evaluated the bestperforming classifier against the adversarial samples, the performance of the classifier degraded significantly. With only one feature perturbation, the TPR falls from 82-97% to 79%-45% and, increasing the number of perturbed features to four, the TPR fell to 0%, meaning that all of the phishing instances were able to bypass the classifier. We continued our experiments by considering the adver-sary cost in the experiment. We showed that both the number of manipulated features and the total manipulation cost, which can be derived from the difference between original phishing sample and the adversarial sample, are essential. This means that from an attacker point of view, not only changing the minimum number of instances is desired, but also it is important that the adversarial sample has the minimum cost. This is an impressive result and shows the weakness of well-known defense mechanisms against phishing attacks. In future, we want to design robust machine learning models that are immune to adversarial learning attacks.

Fig. 1 .

 1 Fig. 1. Robustness of datasets against adversarial samples

Fig. 2 .

 2 Fig. 2. The manipulation cost for adversarial samples based on number of manipulated features

Algorithm 1 :

 1 Generating the Adversarial Samples

		Result: New Adversarial Samples
		Input : 𝑥 , 𝑠𝑒𝑙𝐹 𝑒𝑎𝑡𝑢𝑟𝑒𝑠
		Output: genSamples
	1 Let 𝐿 and 𝑔𝑒𝑛𝑆𝑎𝑚𝑝𝑙𝑒𝑠 be new array
		/ * Get possible values for selected features	* /
	2 foreach featurePos in 𝑠𝑒𝑙𝐹 𝑒𝑎𝑡𝑢𝑟𝑒𝑠 do
	3	L.append ← 𝑇 [𝑓 𝑒𝑎𝑡𝑢𝑟𝑒𝑃 𝑜𝑠]
	4 end
		/ * Product possible values to generate all combination	* /
	5 L_pr ← 𝑝𝑟𝑜𝑑𝑢𝑐𝑡(𝐿 * )
		/ * Generate new instances based on new feature values	* /
	6 foreach new _ val in L _ pr do
		/ * Making a temporary copy of instance	* /
	7	temp ← 𝐶𝑜𝑝𝑦(𝑥)
	8	for 𝑘 ← 0 𝑡𝑜 𝐿𝑒𝑛( 𝑛𝑒𝑤_𝑣𝑎𝑙 ) do
		/ * Override the feature value with new value	* /
	9	temp[𝑥[𝑘]] = new_val[k]
	10	end
		/ * Adding the new instance to the result array	* /
	11	genSamples.append ( temp )
	12 end
	13 return genSamples
	𝑃 𝑟𝑜𝑑𝑢𝑐𝑡 function over array 𝐿 to calculate all possible values for selected features
	and saves them in 𝐿_𝑝𝑟.

Table 1 .

 1 Number of instances, features, and portion of legitimate and phishing websites in each dataset

	Dataset	Data shape (#) Size Features Legitimate Phishing Instances (%)
	DS-1	2210	7	44.71	55.29
	DS-2	11055	30	55.69	44.31
	DS-3	1250	9	43.84	56.16
	DS-4	10000	48	50.0	50.0

Table 2 .

 2 Evaluation of model against different classifiers with two metrics.

			(a) TPR		(b) ACC
	Cls. DS-1 DS-2 DS-3 DS-4 Avg.	Cls. DS-1 DS-2 DS-3 DS-4 Avg.
	DT	95.25 86.77 84.97 96.14 95.25	DT	94.8 92.1 82.51 95.73 91.29
	GB	96.18 92.25 87.23 97.65 96.18	GB	95.49 94.32 83.76 97.52 92.77
	KNN	95.93 90.61 84.95 93.97 95.93	KNN	94.82 92.21 81.16 93.76 90.49
	RF	96.25 94.25 85.84 97.85 96.25	RF	95.35 95.76 82.89 97.8 92.95
	SVM(l) 95	89.62 86.71 94.93 95	SVM(l) 93.96 92.4 79.16 94.38 89.98
	SVM(r) 93.67 91.88 87.88 95.69 93.67	SVM(r) 93.96 94.14 82.4 95.2 91.43
	Best	96.25 94.25 87.88 97.85	Best	95.49 95.76 83.76 97.8

Table 3 .

 3 The classifier that holds best f1 on each dataset has been selected. TPR and ACC are also reported for comparison

	Metric	DS-1 DS-2 DS-3 DS-4
	Best Classifier GB RF GB RF
	Best f1	95.94 95.17 85.83 97.8
	TPR	96.18 94.25 87.23 97.85
	ACC	95.49 95.76 83.76 97.8

Table 4 .

 4 Comparisons of different approaches in the literature including our proposed approach

	Author	Description	Size ACC
	Niakanlahiji et al. [4]	-Scalable feature-rich framework with a series of new and existing features -Not using third-party services, Language agnostic	22.3K 95%
	Sahinguz et al. [11]	-Real-time detection mechanism based on NLP of URLs, Language independent -Tested on a large dataset, Not using third-party service	73K 97%
	Verma et al. [12]	-Features based on lexical-, distance-, and length-related features of the URL -Using four large datasets	115K 99.3%
	Jiang et al. [5]	-Combined the URL and DNS information, Used a deep neural network with the help of NLP, Automatically extracts hidden features	7M	96%
	Tian et al. [13]	-Studied five types of domain squatting, Using dataset of over 224 million instances that evaded common blacklist registered domains, Using visual and OCR analysis, Found new phishing	234M N/A
	Pereira et al. [6]	-Detecting algorithmically generated domain, Graph-based algorithm to extract the dictionaries that are being used to generate algoritmically domains	80K 99%
	Shirazi et al. [7]	-Studying limitation current approaches: large number of features and bias -Not using third-party services in the datasets , Focused on the domain name, Running at the client-side	2.2 K 97-98%
	Li et al. [8]	-Extract the features from both URL and HTML of the page -Not using third-party services	50K 97%
	Bulakh et al. [25]	-Companies can define their phishing detection mechanism and protect the customers -Can be used as an complimentary service besides other detection approaches	1.3K 96.34%
	Our work	-Evaluate the performance of existing datasets including [7, 21, 23, 24] -Using multiple classifiers and comparing the results	2-10K 81-95%
		-Proposing adversarial sampling attack against the learning model,
	Our work	Showing the feasibility of the attack, Prove the vulnerability of current model,	2-10K 0%
		Modeling the vulnerability level and cost

https://data.mendeley.com/

Acknowledgements

This work is supported in part by funds from NSF Awards CNS 1650573, CNS 1822118 and funding from CableLabs, Furuno Electric Company, SecureNok, and AFRL. Research findings and opinions expressed are solely those of the authors and in no way reflect the opinions of the NSF or any other federal agencies.