Anna Kobusi
email: akobusinska@cs.put.poznan.pl

« Ska

Dariusz Wawrzyniak
email: dwawrzyniak@cs.put.poznan.pl

Replication of Recovery Log an Approach to Enhance SOA Reliability

Keywords: SOA, rollback-recovery, message-logging, replication

Along with development of SOA systems, their requirements in terms of fault-tolerance increase and become more stringent. To improve reliability of SOA-based systems and applications, a ReServE service, providing an external support of web services recovery, has been designed. In this paper we propose to enhance the resilience of ReServE by replication of log with recovery information, and address problems related to deployment of this solution.

Introduction

Service-oriented systems (SOA) are increasingly adopted by industry in various areas of computing. Many web services, especially those found in critical or vital domains (e.g. healthcare, nance, defense, etc.), have stringent requirements in terms of availability and reliability. Since in most cases failures of such services are unacceptable, their dependability has to be ensured [START_REF] Avizienis | Basic concepts and taxonomy of dependable and secure computing[END_REF]. However, building a dependable SOA systems is a dicult task, due to their specic properties. SOA systems are highly dependent on the remote web service components of various characteristics, which are autonomous and loosely-coupled. Web services usually run on heterogeneous platforms, and are hosted by dierent organizations. Such services may be unavailable for an unknown reason, and for an undetermined amount of time. Moreover, their providers may refuse or be unwilling to cooperate with other providers to overcome failures of their services. They also may not be able to take part in fault-tolerance processing because of applied fault-tolerance policies. Therefore, services should not be relied upon, when the fault-tolerant mechanisms are to be provided.

As a consequence, we have proposed ReServE service, which aims in increasing SOA fault-tolerance [START_REF] Brzezi«ski | D-reserve: Distributed reliable service environment[END_REF]. ReServE provides an external support of web services rollback-recovery with the use of a well-known mechanism of messagelogging [START_REF] Elmootazbellah | A survey of rollback-recovery protocols in message-passing systems[END_REF][START_REF] Shahzad | A survey of checkpoint/restart techniques on distributed memory systems[END_REF], and ensures that in the case of failure of one or more system components (i.e. web services or their clients) a consistent state of distributed processing is recovered. The interactions exchanged between clients and services are saved by ReServE in the form of message log, stored in the persistent storage, which is assumed to survive all failures. Such an assumption is dicult to be guaranteed in real-life [START_REF] Rao | The cost of recovery in message logging protocols[END_REF]. Additionally, although a persistent storage supports reliability, due to signicant MTTR (mean time to recovery), its crash results in log unavailability for a substantial period of time. This way, another point vulnerable to crash is introduced, which can reduce system availability. Therefore, in this paper we propose to implement the persistent storage used in ReServE service as a replicated log containing the recovery information. Along with the replication of a message log, also the logic of the module of ReServE, called Recovery Management Unit (RMU), which is responsible for the implementation of a recovery process has to be replicated. Although a general idea of log and RMU replication is straightforward [START_REF] Pedone | Understanding replication in databases and distributed systems[END_REF], and relies on storing messages necessary for recovery of system participants in replicas, its implementation raises several problems, which have to be solved. Among them are: synchronization of logs kept by dierent replicas, handling replicas failures, combining the replication of recovery information with the recovery processing. In this paper we discuss how the above problems may be resolved in the context of ReServE.

The paper is organized as follows. Sections 2 and 3 present system model and general idea of ReServE, respectively. Section 4 discusses the possible approach to replication of ReServE recovery log. Finally, Section 5 concludes the paper and presents the future directions of our work.

2 System model Throughout this paper, a distributed SOA system is considered [START_REF]Reference Architecture Foundation for Service Oriented Architecture -Version 1[END_REF]. It consists of service providers that keep resources, and deliver in the form of provided web services a specied functionality to clients. Web services are autonomous and loosely-coupled. They have a well-dened and standardized interface that denes how to use them. Clients invoke services by sending requests, so service invocation results in a computation, subsequent reply to the requesting client, and possible resource state changes. Service execution may also encompass the collaboration of other services (without compromising the autonomy of each individual service). It is assumed that both clients and services are piece-wise deterministic. Services can concurrently process only such requests that do not require access to the same or interacting resources. Otherwise, the existence of a mechanism serializing access to resources, which uniquely determines the order of operations, is assumed. Communication in the considered system is stateless each request contains all the information necessary to understand the request, independently of any requests that may have preceded it. The considered communication channels are reliable (the reliability is ensured by the retransmission of messages), but they do not guarantee FIFO property. Additionally, the crash-recovery model of failures is assumed, i.e., system components may fail and recover after crashing a nite number of times [START_REF] Avizienis | Basic concepts and taxonomy of dependable and secure computing[END_REF]. Failures may happen at arbitrary moments, and we require any such failure to be eventually detected, for example by a Failure Detection Service [START_REF] Brzezi«ski | Dependability infrastructure for SOA applications[END_REF]. We assume that each service provider may have its own reliability policy and may use dierent local mechanisms that provide fault tolerance. [START_REF] Brzezi«ski | Dependability infrastructure for SOA applications[END_REF]

ReServE architecture

In this Section, the design choices and concepts behind ReServE service are presented. The detailed description of ReServE has already been presented in [START_REF] Brzezi«ski | D-reserve: Distributed reliable service environment[END_REF][START_REF] Danilecki | Re-ServE service: An approach to increase reliability in service oriented systems[END_REF], and is summarized here in order to make a paper self-contained. Due to the fact that interactions between clients and services result in a computation and possible resource state changes, they entail the client-service inter-dependencies. Upon a failure of one of interacting processes, such dependencies may force other processes that did not fail to rollback. Otherwise, states of processes could reect situations impossible in any correct failure-free execution. Due to SOA assumption on autonomy of services, the failure of one process should not inuence the processing of the others. Since service providers do not provide information on the internal implementation of services, it is not known which events introduce inter-process dependencies and result in state changes. Therefore, in general, the recovery of a failed service should be isolated to avoid the cascading rollbacks of other processes. Above observation had an impact on the concept of ReServE functionality. ReServE intercepts the communication between processes and logs all performed interactions (requests of service invocations and the appropriate replies) in a persistent storage of Recovery Management Unit (RMU). The intercepted messages reect the complete history of communication, which is used to recover the consistent system state in the case of failure. However, since in SOA participants of processing may have their private mechanisms providing reliability, their state after the failure may be partially reconstructed with the use of local mechanisms. Therefore, only those messages, the processing of which was not reected in services' (clients') recovered state, should be processed again. The task of RM U is to nd such messages, and reissue them to the service in the same order as before the failure. After re-execution of recovered requests RM U intercepts replies from the service, because they have already been sent to clients and other services during the failure-free execution. RM U module ensures also the idem potency of obtained requests. If it obtains the client's request, to which the response has already been saved in its persistent storage, then such a saved response is sent to the client immediately, without the need of sending the request to the service once again. Thus, the same message (i.e., the message with the same identication number) may be send by a client multiple times, with no danger of multiple service invocations. Another two modules of ReServE service are Service Intermediary Modules (SIM) and Client Intermediary Module (CIM). CIM and SIM serve as proxies for clients and servers and hide the details of rollback-recovery. For this purpose, both modules intercept messages issued by clients and servers, so they allow to fully control the ow of messages in the system. Additionally, SIM monitors the services' status and react in the case of its eventual failure by initiating and managing the service rollback-recovery.

In this Section we propose to replicate the RM U module of ReServE service, instead of using the persistent storage. From the perspective of clients and services, introducing RM U replication is transparent. Each service has a dedicated replica of RM U module, in which it is registered, called a Leader. Analogically, each client has a default RM U replica, called Interceptor. We assume that each request issued by a client is rst replicated, before it is sent to the service. Analogically, the service reply is replicated, before it is sent back to the client.

The crucial issue arising from replication is consistency. Since the replication in the context of this paper concerns log, i.e. a set of requests and replies, and adding elements to a set is commutative, i.e. it does not pose a risk of conicts, thus consistency maintenance boils down to preserving replica completeness. The completeness is important for message safety in the sense of the ability to survive RM U replica crash. We assume that a message is safe if it can be obtained by a given number N of replicas, despite the crash of some log servers. A number N ranges from 1 to | RM U |. When N =| RM U |, all correct replicas hold the message, and the highest level of message safety is achieved. At the same time the system availability (response time), is decreased because before the message is sent to its recipient, rst all N replicas have to acknowledge the fact that they obtained the message. In turn, in the worst case, only one complete replica is required to survive the crash, and to hold the message. In such situation, the level of system reliability is the smallest, but its availability is uttermost, as the message obtained by the RM U replica is immediately passed to its recipient. In the proposed solution it is a role of a Leader to check safety of messages.

The idea of replicating the request among RM U replicas is the following: each time the Interceptor obtains the new request, it adds it to the log M (being a set of requests and replies exchanged between clients and services, and stored by each RM U replica), and broadcasts it to other replicas. The fact of delivering request to a replica is acknowledged by the communication channel, and results in adding such a RM U replica to the set A (a set of RM U replicas that acknowledged obtaining the request). The Leader, after obtaining request from Interceptor, can immediately send a reply to the client and to Interceptor, provided it possess the matching reply in its log M. Otherwise, Leader is responsible for forwarding the request to SIM of the requested service. Since in the considered replication scheme the request is forwarded only after N requests replicas exist, the knowledge on the number of replicas maintained in the system is essential. For this purpose, each RM U replica, after obtaining a request and storing it in its log, sends acknowledment to Leader that updates its set A. But, since the receipt of request can be acknowledged by some RM U replicas either to Interceptor or to Leader (for example in the case of communication channels with a low bandwidth between the RM U replica and Leader ), Interceptor informs Leader , by sending its A to the Leader, which RM U replicas possess a request. In turn, Leader expands its set A on the basis of information obtained from Interceptor. Additionally, Leader broadcasts request to all RM U replicas that did not conrm obtaining this message neither to Interceptor, nor to Leader.

The purpose of the re-broadcast of request is to allow RM U replica that have just recovered from the failure to take part in the replication. In case one RM U replica acts as both the Interceptor, and the Leader, the procedure of updating the set A is simplied. There may be RM U replicas which do not belong to the set, although they possess the request message replica. However, this does not aect the correctness of the proposed solution. After performing the request, service provider returns the reply through its SIM to the Leader. When Leader obtains reply for the rst time, it stores it in the log M, and broadcast it to all RM U replicas to replicate reply along with its corresponding request. Finally, the Leader sends reply to the client. The architecture of replicated ReServE is shown in Fig. 1. Despite reliable communication channels, an Interceptor, Leader, and other RM U replicas can fail, which eectively disturbs the communication between clients and service providers. In order to mask transient communication failures, the client reissues its request when no reply has been received within a given time. Thus, the role of the RM U replica is twofold: keep requests for the purpose of service recovery, and replies for the purpose of client recovery or for ltering duplicated request. In the case of the Leader crash another RM U replica must be elected to take the responsibility for further communication with the service. The Interceptor suspects the Leader crash in two cases. First, when the acknowledgment of obtaining a request broadcasted by the Interceptor is not delivered to Leader, and when the acknowledgment of obtaining by the Leader a request reissued by the client is not delivered. In both cases the Interceptor starts the Leader election procedure. In ReServE service considered until now, we have assumed that each service component can be the subject of failure, except of persistent storage of RM U , where the message log is stored. Since this approach understates the robustness of message log, in this paper we proposed the preliminary concept of the alternative solution, based on the replication of RM U and its log.

Applying the replication mechanism will always introduce an overhead. But, the preliminary performance tests show that in the case of the specic message size, the overall costs of the proposed approach based on the replication are not inferior to the cost associated with the costs of saving messages in the stable storage. Thus, the proposed solution is competitive with the one based on persistent storage, because at similar costs, it increases recovery log availability.

Our future work encompasses the introduction of detailed protocol of recovery log replication, formal proof of its corectness, and detailed empirical evaluation.

Fig. 1 :

 1 Fig. 1: ReServE replication of recovery log

5

  Conclusions and Future Work

This work was supported by the Polish National Science Center under Grant No. DEC-2011/03/D/ST6/01331 and the Grant No. 09/91/DSPB/0571


Eduardo De

Lucena Falcão

Francisco Brasileiro

Andrey Brito
email: andrey@computacao.ufcg.edu.br

José Luis Vivas
email: jlvivas@lsd.ufcg.edu.br

Incentivising Resource Sharing in Federated Clouds

Keywords: cloud and grid computing, cooperation incentives and fairness, peer to peer computing

An important cloud federation enabler is the development of suitable business models, which has so far received relatively little attention from investigators. Few efforts have been dedicated to investigation of business models for enabling cloud federations, especially with regard to distribution issues, in which centralized solutions are usually preferred. Hence, in our work we focus on the use of fully decentralized mechanisms supporting federation of private clouds based on barter mechanisms. We analyze the adaptation of fully decentralized incentive mechanisms previously used in the context of P2P desktop grids, and show that they are not suitable for federated cloud systems because fairness cannot always be guaranteed. We show initial results concerning the use of a mechanism intended to guarantee a higher level of fairness and thereby to promote voluntary participation in a decentralized federation without any central or trusted enforcing authority.

Introduction

Organizations with variable and peaky demand patterns often turn to public clouds (cf. "cloud bursting" [START_REF] Marshall | Elastic site: Using cloudsto elastically extend site resources[END_REF]) in order to meet unexpected or short-term needs. However, during off-peak times resources might become idle, which constitutes an efficiency loss for the organization. An alternative for cloud providers is to participate in a federation for exchanging idle resources. In particular private cloud providers, due to the usually limited amount of owned resources, would greatly benefit from this [START_REF] Gomes | Pure exchange markets for resource sharing in federated clouds[END_REF].

From the architectural perspective, cloud federations can be either centralized or Peer-to-Peer (P2P) [START_REF] Grozev | Inter-cloud architectures and application brokering: taxonomy and survey[END_REF]. In centralized architectures, resource allocation is typically performed by a trusted central entity that is able to prevent free riding and to perform the best matching of consumers and providers. In P2P federations, on the other hand, participants must communicate and negotiate directly with each other. The advantages of decentralized topologies include extensibility, deployment, management, use and growth. The drawbacks, however, include difficulties in discovery, routing, security, and the fact that participants are mostly unknown to each other and cannot be assumed to be trustworthy or collaborative. Moreover, peers should be assumed to be selfish and to have an economic incentive to become free riders.

Similarly, from a market perspective cloud federations can be also classified as centralized or decentralized. In a centralized market system, bids and requests are collected by a central entity or market auctioneer that matches them and decide the best matching of buyers and sellers. In decentralized markets, on the other hand, buyers and sellers must explore the market by themselves and bargain directly with each other. Participants may provide resources in exchange for payment or by bartering. Payment schemes would require the introduction of complex management mechanisms and procedures, whereas bartering schemes can be implemented e.g. in the shape of flexible credit and debit local annotations. The latter would be very suitable for the federation of private clouds, since it is a wholly money-less and distributed scheme that can do without any trusted centralized entities. Obviously, there is a close relation between the architecture and the market structures of private cloud federations. In this work, we propose a lightweight P2P cloud federation infrastructure implementing a decentralized market system.

An important challenge concerns the promotion of cooperation among rational selfish individuals in a decentralized context with no central and trusted enforcing authority. In this kind of system, participants are usually left to themselves and only with limited information about the actual behavior or intention of other actors, may not keep promises, and must rely solely on their own experience, acquired through interactions with each other, in order to decide to what extent they should trust other partners. It is therefore natural to expect that, at first hand, participants will prefer to act as free riders. Moreover, collaborative partners may defect from the federation if they are not satisfied with the results of participation. Some form of individual incentives must therefore be enforced in order to ensure sustained voluntary participation. Our aim is to propose one such mechanism enabling the collaborative peer to make efficient decisions that guarantees both its satisfaction, defined as the ratio between received and requested resources (which should ideally be one), and fairness, defined as the ratio between the amount of resources obtained and the amount of resources provided (which should ideally be approximately one on the long run). The challenge is to find a scheme that guarantees that the levels of both fairness and satisfaction are good enough to ensure that most participants will not defect and free riders will be isolated or kept with a low degree of satisfaction.

For the reasons explained above, incentives or punishment procedures must exist in order to promote cooperation and keep the federation alive. The scheme we propose in this paper leverages the notion of the Network of Favors (NoF) [START_REF] Andrade | Discouraging free riding in a peer-to-peer cpu-sharing grid[END_REF], an incentive mechanism for resource sharing in P2P opportunistic desktop grids. In NoF, each peer uses only its own locally stored interaction history with other peers, which is basically the balance of favors exchanged (total amount of favors a peer A consumed from peer B minus total amount of favors a peer A do-nated to peer B), in order to decide which ones will be given priority in resource requests. We call this notion of NoF the Satisfaction-Driven NoF (SD-NoF). In the SD-NoF a collaborator always supply all of its idle resources to the federation, with the expectation of accumulating credits with other peers, which may be converted to favors in the future. This ensures the best possible levels of satisfaction to collaborators, independently of the level of resource contention. This scheme is suitable for opportunistic desktop grids, since the costs of providing resources, which are intended primarily for in-house consumption, are assumed to be so low that collaborators may disregard fairness and focus only on maximizing satisfaction. However, in the case of private and P2P federated clouds, resources are dedicated, and the associated overhead costs (management staff, energy and space) may not be negligible. Fairness may thus become an important goal, which may be achieved by limiting the amount of supplied resources, thereby isolating free riders more efficiently specially in scenarios with low resource contention. To this end, we introduce a feedback control loop mechanism regulating the amount of supplied resources. Briefly, in order to reward cooperative actors and isolate free riders, a peer in the federation will draw upon its current assessment of fairness in order to define the amount of offered resources, even if it would have at any given moment more idle resources than those offered. By contrast to the SD-NoF, we call this new scheme Fairness-Driven NoF (FD-NoF).

The outcome is a mechanism that more closely conforms to the game theoretical results derived from the notion of reciprocal altruism [START_REF] Trivers | The evolution of reciprocal altruism[END_REF], which shows how selection can operate against cheaters or non-reciprocators (the free riders in the NoF). Altruistic behavior is defined as a behavior that benefits another unrelated individual at a certain cost for the contributing individual. Selection would seem to favor "cheaters", i.e. those individuals that fail to reciprocate favors. However, selection might also discriminate against cheaters if the altruist is able to curtail future altruistic gestures to those individuals, assuming that the sum of the benefits for the altruist of those lost acts outweighs their cost. As a result, the altruist will prefer to exchange altruistic acts with other altruist individuals, not with cheaters.

Feedback Control Loop Mechanism

Andrade et al. [START_REF] Andrade | Discouraging free riding in a peer-to-peer cpu-sharing grid[END_REF] proved that whenever there is resource contention 1 between collaborators (κ), the SD-NoF works well and prioritizes collaborators before free riders. However, only prioritizing collaborators is not enough in scenarios with low resource contention (κ < 1), since in this case all the surplus resources will nevertheless be offered, thus benefiting free riders. To avoid this, collaborators could try to regulate the amount of resources provided, thereby indirectly increasing the value of κ. Collaborators are not able to directly control the value of κ in a decentralized P2P system, and may not even have an interest in doing it. However, contention may be indirectly increased when collaborators try to balance their level of fairness. The simplest way to do it is by decreasing the amount of provided resources. If fairness is close to 1 for each collaborator, the value of contention will also be close to 1. However, the overall level of satisfaction may decrease thereby, and affect not only free riders but collaborators as well.

Each collaborator is then equipped with the feedback control loop mechanism and must define on its own a threshold value τ for the desired minimum level of fairness. The mechanism enhances fairness by controlling the amount of provided resources until its value becomes higher than τ ; once the desired level of fairness has been achieved, the algorithm will instead focus on enhancing satisfaction by increasing the amount of resources provided. Roughly, the feedback control loop mechanism decides whether to decrease or increase the amount of resources provided by a fixed value at each step of the simulation, in this work chosen to be 5% of the total resources capacity, in order to achieve the desired levels of both fairness and satisfaction. By this mechanism, a collaborator could momentarily "leave" the federation, by supplying no resources, until it receives again some favors.

Simulation Model

To assess the behavior of the participants in the FD-NoF, we have built a simulator for a simplified model of a resource sharing P2P federation of cloud providers. The federation consists of a community of n peers, with (1 -f ) • n collaborators and f • n free riders, 0 ≤ f ≤ 1. We are interested in understanding how the set of collaborators perform in relation to the set of free riders. Thus, to eliminate the influence that different kinds of participants might have in the results, we assume that all collaborators have the same capacity and needs, and free riders the same needs. The simulation proceeds in steps. At each step each collaborator can be in a consumer state with probability of π or in a provider state with probability 1 -π. Each collaborator is assumed to have a total resource capacity of C, and the amount of demand requested by any consuming collaborator is D • C, where D ≥ 0. In the SD-NoF, when in a provider state a collaborator will always offer all of its resources, i.e. C. At each step, the resources are given to one or more peers selected from the set of consuming peers according to the balance of favors exchanged (more details of NoF in [START_REF] Andrade | Discouraging free riding in a peer-to-peer cpu-sharing grid[END_REF]). Free riders, in contrast, never provide resources and are always in a consumer state, each one requesting D • C resources at every step.

Scenarios

Our goal is to understand the behavior of the peers in scenarios with low, moderate and high levels of κ, with values ranging over the set {0.5, 1.0, 2.0, 4.0}.

We are interested in the scenarios in which the amount of free riders may seriously affect the level of fairness for collaborators. The higher the percentage of free riders in the system, the more resources, which are provided solely by collaborators, they will collectively consume in total, thereby affecting negatively the level of fairness for collaborators. Thus, we chose f = 0.75 for all scenarios.

The larger the number of participants in the federation, the worse will be the situation to the free riders, even with constant f . This is because the larger the total amount of collaborators, the less will be at each step the deviation from the expected value of the total amount of providers, and hence the probability that a significant amount of resources will be available to free riders, which would imply that the number of providers is a bit higher than average. Then, we fix n to 100 in all scenarios.

Finally, for simplicity we consider C = 1. In summary, our design of experiments will have three parameters with constant values, n = 100, f = 0.75 and C = 1, and two variables with changing values: D and π -which both generates the different levels of κ. We run the simulations in 4000 steps, sufficient for ensuring that the fairness of collaborators achieves stability.

Results and Analysis

In order to evaluate the Feedback Control Loop we simulated the scenarios presented in section 2.2 in both SD-NoF and FD-NoF. Figure 1 shows the average fairness of collaborators (lines) and the individual fairness of each one of them (circles) along the four κ values, at the last step of the simulation in both SD-NoF and FD-NoF, the latter with τ ∈ {0.8, 0.95}. From Figure 1 we may observe that the Feedback Control Loop Mechanism was able to increase the values of fairness in low and moderate κ scenarios. Because f is high (75%), although free rider's individual satisfaction tends to be low (due to the high resource contention between themselves), their collective action strongly affects the fairness of collaborators in SD-NoF, which is on average 0.48 and 0.81 for contention values of 0.5 and 1, respectively (see yellow line on Figures 1a and1b). In this case, FD-NoF increased the fairness of collaborators on average by 60% and 15%, respectively, when τ was set to 0.95, and by 48% and 3%, respectively, when τ was set to 0.8.

Moreover, in scenarios with high values of κ, FD-NoF behaves similarly to SD-NoF, regardless the high f value and τ , which is good since SD-NoF already works fine in these scenarios. One can also notice that the higher is the value of τ , the tougher is the task of the controller. Thus, not surprisingly, when we used τ = 0.95, only few collaborators could indeed approach it when κ = 0.5. Obviously, as κ increases, the Feedback Control Loop Mechanism gets more efficient. Also, smaller values of τ lead to less variability among collaborators.

In summary, FD-NoF achieves its goal: collaborators improve their level of fairness by controlling the amount of supplied resources.

Conclusions and Future Works

In our work we introduce the FD-NoF, an enhancement of the NoF [START_REF] Andrade | Discouraging free riding in a peer-to-peer cpu-sharing grid[END_REF], by defining a Feedback Control Loop mechanism to regulate the amount of resources provided. With the aid of this mechanism collaborators may achieve greater levels of fairness by indirectly increasing the levels of resource contention. FD-NoF behaves similarly to SD-NoF in high resource contention scenarios, which is a positive result since SD-NoF by itself already provides good values of fairness.

As future work, we will implement the FD-NoF as part of the middleware 2 , and deploy and evaluate the performance of the FD-NoF on the federated cloud that comprises the computing infrastructure of the EUBrazilCC project 3 . Moreover, we will refine the FD-NoF in order to accelerate convergence by introducing delegation schemes enabling the donation of services on behalf of a third agent.

  (a) f = 0.75 and τ = 0.8. (b) f = 0.75 and τ = 0.95.

Fig. 1 :

 1 Fig. 1: Average and density of fairness to collaborators along κ ∈ {0.5, 1, 2, 4} with SD-NoF and FD-NoF.

Contention is a metric that characterizes the degree of competition for offered resources, defined as the ratio of requested resources by provided resources.


Olga Tarasyuk
email: o.tarasyuk@csn.khai.edu

Anatoliy Gorbenko
email: a.gorbenko@csn.khai.edu

Alexander Romanovsky
email: alexander.romanovsky@ncl.ac.uk

Vyacheslav Kharchenko
email: v.kharchenko@khai.edu

Vitalii Ruban

The Impact of Consistency on System Latency in Fault Tolerant Internet Computing

Keywords: Internet computing, fault-tolerance, consistency, latency, response time, modelling 1

The paper discusses our practical experience and theoretical results in investigating the impact of consistency on latency in distributed fault tolerant systems built over the Internet. Trade-offs between consistency, availability and latency are examined, as well as the role of the application timeout as the main determinant of the interplay between system availability and performance. The paper presents experimental results of measuring response time for replicated service-oriented systems that provide different consistency levels: ONE, ALL and QUORUM. These results clearly show that improvements in system consistency increase system latency. A set of novel analytical models is proposed that would enable quantified response time prediction depending on the level of consistency provided by a replicated system.

Introduction

Distributed computing has become an industrial trend, indispensable in dealing with enormous data growth. High availability requirements for many modern Internet applications require the use of system redundancy and data replication. Basic fault tolerant solutions such as N-modular, hot-and cold-spare redundancy usually assume a synchronous communication between replicas, which means that every message is delivered within a fixed and known amount of time [START_REF] Lee | Fault Tolerance. Principles and Practice[END_REF]. This is a reasonable simplification for the local-area systems whose components are compactly located, for instance, within a single data centre.

This assumption does not appear to be relevant, however, for the wide-area systems, in which replicas are deployed over the Internet and their updates cannot be propagated immediately, which makes it difficult to guarantee consistency.

The Internet and, more generally, the wide-area networked systems are characterized by a high level of uncertainty, which makes it hard to guarantee that a client will receive a response from the service within a finite time. It has been previously shown that there is a significant uncertainty of response time in service-oriented systems invoked over the Internet [START_REF] Gorbenko | Real Distribution of Response Time Instability in Service-Oriented Architecture[END_REF][START_REF] Gorbenko | Exploring Uncertainty of Delays as a Factor in End-to-End Cloud Response Time[END_REF][START_REF] Bakr | Evaluating the running time of a communication round over the internet[END_REF]. Besides, our experience and other studies [START_REF] Bakr | Evaluating the running time of a communication round over the internet[END_REF][START_REF] Chen | Measuring and Dealing with the Uncertainty of the SOA Solutions[END_REF][START_REF] Potharaju | When the Network Crumbles: An Empirical Study of Cloud Network Failures and their Impact on Services[END_REF][START_REF] Scott | LIFEGUARD: practical repair of persistent route failures[END_REF] show that failures are a regular occurrence on the Internet, clouds and in scale-out data centre networks. When developers apply replication and other fault tolerant techniques in the Internet-and cloud-based systems, they need to understand the time overheads and be concerned about delays and their uncertainty.

In this paper we examine, both in experimental and theoretical terms, how different fault-tolerance solutions [START_REF] Avizienis | Basic Concepts and Taxonomy of Dependable and Secure Computing[END_REF] implemented over the Internet affect system latency depending on the level of consistency provided. The paper discusses the trade-offs between consistency, availability and latency. Although these relations have been identified by the CAP theorem in qualitative terms [START_REF] Brewer | Towards Robust Distributed Systems[END_REF][START_REF] Gilbert | Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services[END_REF], it is still necessary to quantify how different fault-tolerant techniques affect system latency depending on the consistency level. The main contributions of the paper are probabilistic models that can predict the system response time depending on the chosen fault-tolerance technique and/or the selected consistency level, with the probabilistic behaviour of replicas as an input parameter.

The rest of the paper is organized as follows. In Section 2 we discuss the impact of the CAP theorem [START_REF] Brewer | Towards Robust Distributed Systems[END_REF][START_REF] Gilbert | Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services[END_REF] on distributed fault-tolerant systems and examine the tradeoffs between system consistency, availability and latency. Section 3 summarises the results of experimental response time measurements for testbed fault-tolerant systems that have three replicas distributed over the Internet and support different consistency levels. The probabilistic models introduced in Section 4 define the relation between system response time and the consistency level provided. Section 5 evaluates the accuracy of the proposed analytical models by applying them in practice and comparing their results with our experimental data. Finally, some practical lessons learnt from our experimental and theoretical work are summarised in Section 6.

Understanding Trade-offs Between Consistency, Availability and Latency in Distributed Fault-Tolerant Systems

The CAP conjecture [START_REF] Brewer | Towards Robust Distributed Systems[END_REF], which first appeared in 1998-1999, defines a trade-off between system availability, consistency and partition tolerance, stating that only two of the three properties can be preserved in distributed replicated systems at the same time. Gilbert and Lynch [START_REF] Gilbert | Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services[END_REF] view the CAP theorem as a particular case of a more general trade-off between consistency and availability in unreliable distributed systems which assume that updates are eventually propagated. System partitioning, availability and latency are tightly connected. A replicated fault-tolerant system becomes partitioned when one of its parts does not respond due to arbitrary message loss, delay or replica failure, resulting in a timeout. System availability can be interpreted as a probability that each client request eventually receives a response.

In many real systems, however, a response that is too late (i.e. beyond the application timeout) is treated as a failure. High latency is an undesirable effect for many interactive web applications. In [START_REF] Brutlag | Speed Matters for Google Web Search[END_REF] the authors showed that if a response time increases by as little as 100 ms, it dramatically reduces the probability of the customer continuing to use the system.

Failure to receive responses from some of the replicas within the specified timeout causes partitioning of the replicated system. Thus, partitioning can be considered as a bound on the replica's response time. A slow network connection, a slow-responding replica or the wrong timeout settings can lead to an erroneus decision that the system has become partitioned. When the system detects a partition, it has to decide whether to return a possibly inconsistent response to a client or to send an exception message in reply, which undermines system availability.

The designers of the distributed fault-tolerant systems cannot prevent partitions which happen due to network failures, message losses, hacker attacks and components crashes and, hence, have to choose between availability and consistency. One of these two properties has to be sacrificed. If system developers decide to forfeit consistency they can also improve the system response time by returning the fastest response to the client without waiting for other replica responses until the timeout, though this would increase the probability of providing inconsistent results. Besides, timeout settings are also important. If the timeout is lower than the typical response time, a system is likely to enter the partition mode more often [START_REF] Gorbenko | Dependability of Service-Oriented Computing: Time-Probabilistic Failure Modelling[END_REF].

It is important to remember that none of these three properties is binary. For example, modern distributed database systems, e.g. Cassandra [START_REF] Lakshman | Cassandra: a decentralized structured storage system[END_REF], can provide a discrete set of different consistency levels for each particular read or write request. The response time can theoretically vary between zero and infinity, although in practice it ranges between a minimal affordable time higher than zero and the application timeout. Availability varies between 0% and 100% as usual.

The architects of modern distributed database management systems and large-scale web applications such as Facebook, Twitter, etc. often decide to relax consistency requirements by introducing asynchronous data updates in order to achieve higher system availability and allow a longer response time. Yet the most promising approach is to balance these properties. For instance, the Cassandra NoSQL DDBS introduces a tunable replication factor and an adjustable consistency model so that a customer can choose a particular level of consistency to fit with the desired system latency.

The CAP theorem helps the developers to understand the system trade-offs between consistency and availability/latency [START_REF] Abadi | Consistency Tradeoffs in Modern Distributed Database System Design[END_REF]. Yet even though this theorem strongly suggests that better consistency undermines system availability and latency, developers do not have quantitative models to help them to estimate the system response time for the chosen consistency level and to achieve a precise trade-off between them.

Our interpretation of the CAP theorem and the trade-offs resulting from the CAP is depicted in Fig. 1. The application timeout can be considered as a bound between system availability and performance (in term of latency or response time) [START_REF] Gorbenko | Time-Outing Internet Services[END_REF]. Thus, system designers should be able to set up timeouts according to the desired system response time, also keeping in mind the choice between consistency and availability. In the following sections we discuss our practical experience on measuring latency of fault-tolerant service-oriented system depending on the provided consistency level and also introduce analytical models predicting system response time.

Experimental Investigation of the CAP Impact on Fault-Tolerant Service-Oriented Systems

Description of the Testbed Architecture

To investigate the CAP impact on fault-tolerant distributed systems we developed a testbed service-oriented system composed out of the three replicated web services (see Fig. 2). This is a typical setup employed in many fault-tolerant solutions. A testbed web service was written in Java and its replicas uploaded to Amazon Elastic Beanstalk and were deployed in the three different location domains: (i) US West (Oregon); (ii) South America (Sao Paulo) and Asia Pacific (Tokyo). Each web service replica performs a heavy-computational arithmetic calculation such as finding the n digit of Pi when n is a large number and returns the result to the driver. The driver is responsible for invoking each of the replicated web services, waiting for the web services to complete their execution and return response, and, finally, implementing a particular fault-tolerant scheme upon the obtained results.

AWS SDK for Java was used to connect web service replicas on Amazon EC2 from clients (driver) programming code that helps to take the complexity out of coding by providing Java APIs for AWS services.

In our study we investigated the three basic fault-tolerant patterns for web services [START_REF] Gorbenko | Using Inherent Service Redundancy and Diversity to Ensure Web Services Dependability[END_REF] corresponding to different consistency levels (ONE, ALL, QUORUM). In all cases the driver simultaneously forwards client's request to all replicated web services. The consistency level determines the number of replicas which must return a response to the driver before it sends an adjudicated result to the client application:

 ONE (hot-spare redundancy) -when the FASTEST response is received the driver forwards it to the client. This is the weakest consistency level though it guarantees the minimal latency;  ALL (N-modular redundancy) -the driver must wait until ALL replicas return their responses. In this case the response time is constrained by the slowest replica though the strongest consistency is provided;  QUORUM -the driver must wait for the responses from a QUORUM of replica web services. It provides a compromise between the ONE and ALL options trading off latency versus consistency. The quorum is calculated as: (amount_of_replicas / 2) + 1, rounded down to an integer value. As far as in our experiments we use the replication factor of 3, the quorum is 2.

The driver also implements a timeout mechanism aimed to protect clients from endless waiting in case of network or web-services failures or cloud outages.

Response Time Measurement

The driver was implemented as part of the Java client software. The client software was run at a host in the Newcastle University (UK) corporate network. It invoked replica web services several thousand times in a loop using the driver as a proxy.

For the particular client's request we measured the response time of the each web service replica and also times when the driver produces responses corresponding to different consistency levels. The delay induced by the driver itself was negligible in our experiments.

The measurement results obtained for the first 100 invocations are presented in Figs. 3 and4. Table 1 summarizes basic statistical characteristics of the measured data whereas probability density series (pds) of system and replicas response times are depicted in Figs. 5 and6.

As expected, when the system is configured to provide consistency level ONE its latency in average is less than the average response time of the fastest replica. Average system latency in case it provides consistency level ALL is larger than the average response time of the slowest replica. System latency associated with consistency level QUORUM is in the middle.

However, our main observation is that it is hardly possible to make an accurate prediction of the average system latency corresponding to the certain consistency level when the only common statistical measures of replicas response time (i.e. minimal, maximal and average estimates and standard deviation) are known.

This finding resulting from our massive experiments and also confirmed by other researches [START_REF] Rao | Using Paxos to Build a Scalable, Consistent, and Highly Available Datastore[END_REF] show that it is extremely difficult to predict the timing characteristics of various types of wide-area distributed systems, including fault-tolerant SOAs, distributed databases and file systems (e.g. Cassandra, GFS, HDFS), parallel processing systems (e.g. Hadoop Map-Reduce). The dynamic and changing nature of timing characteristics of such systems can be better captured by employing probability density functions.

In the next section we propose a probabilistic modelling approach that addresses this problem. It relies on using probability density functions (PDF) of replica response times to predict system latency at different consistency levels. We propose a set of probabilistic models that allow us to build a combined probability density function of system response time by taking into account provided consistency level and incorporating response time probability density functions for each replica.

When the system is configured to provide consistency level ALL, the probability of returning response to the client at time t is equal to the probability that one of the replicas (e.g. the first one) returns its response exactly at time t, i.e. g 1 (t) while two other replicas return their responses not later than t (by time t), i.e.

) ( ) ( t G t g t 2 0 2   and ) ( ) ( t G t g t 3 0 3   .
So far as we have three replicas, all three possible combinations have to be accounted. As a result, the probability density function of the system response time for consistency level ALL can be defined as following:

) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( t G t G t g t G t G t g t G t G t g t f ALL 2 1 3 3 1 2 3 2 1    . ( 1
)
where g 1 (t), g 2 (t) and g 3 (t) -are response time probability density functions of the first, second and third replicas respectively; G 1 (t), G 2 (t) and G 3 (t) -are response time cumulative distribution functions of the first, second and third replicas respectively. When the system is configured to provide consistency level ONE, the probability of returning a response to the client at time t is equal to the probability that if only one of the replicas (e.g. the first one) returns its response exactly at time t, i.e. g 1 (t), while two other replicas return their responses at the same time or later on, i.e.

) ( ) ( t G t g t 2 2 1     and ) ( ) ( t G t g t 3 3 1     .
Keeping in mind three possible combinations we can deduce the probability density function of the system response time for consistency level ALL as:

        . ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( t G t G t g t G t G t g t G t G t g t f ONE 2 1 3 3 1 2 3 2 1 1 1 1 1 1 1           (2)
Deducing the response time probability density function for the QUORUM consistency level is based on a combination of the previous two cases. The probability of returning response to the client at time t is equal to the probability that one of the replicas returns its response exactly at time t; one of the two remained replicas returns its response by time t and another one responds at time t or later on. Taking into account all possible combinations the probability density function of the system response time for consistency level QUORUM can be deduced as:

        . ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( t G t G t g t G t g t G t G t g t G t g t G t G t g t G t g t f QUORUM 1 2 3 3 2 2 1 3 3 1 3 1 2 2 1 1 1 1            (3)
Using similar reasoning it is possible to deduce response time probability density functions of a system composed of n replicas:

              n i n j j i i ALL t G t G t g t f 1 1 ) ( ) ( ) ( ) ( . ( 4
)                   n i n j j i i ONE t G t G t g t f 1 1 1 1 ) ( ) ( ) ( ) ( . ( 5
)
It is extremely hard to build a general form of the probability density function of the system response time for consistency level QUORUM. However, the general reasoning is as following. The composed probability density function should be presented as a sum of m items, where m is a number of k-combinations of n (k is a number of replicas constituting a quorum). Each of the m items is a product of two factors. The first one defines the probability that a particular combination of k replicas return their responses by time t. Another factor defines the probability that the remaining (n-k) replicas return their responses after t.

Models Validity

In this section we check the validity and accuracy of the proposed models by comparing their prediction with the experimental data presented in Section 3. This check includes the following four steps:

 finding out theoretical distribution laws that accurately approximate the measured replica response times;  applying the proposed mathematical models (1), ( 2) and (3) to deduce probability density functions of the system response time for different consistency levels;  estimating replica and system average response times using the theoretical probability distribution functions;  comparing the theoretical and experimental values of replica and system average response times.

Finding Theoretical Distribution Laws of Replica Response Times

Theoretical distribution laws approximating replica response times can be found in a way described in [START_REF] Gorbenko | Real Distribution of Response Time Instability in Service-Oriented Architecture[END_REF]. It is based on performing a series of hypotheses checks in the Matlab numeric computing environment. The techniques of hypothesis testing consist of the two basic procedures. First, the values of distribution parameters are estimated by analysing an experimental sample. Second, the null hypothesis that experimental data has a particular distribution with certain parameters should be tested.

To perform hypothesis testing itself we used the kstest function: [h, p] = kstest(t, cdf), conducting the Kolmogorov-Smirnov test to compare the distribution of t with the hypothesized distribution defined by matrix cdf.

The null hypothesis for the Kolmogorov-Smirnov test is that t has a distribution defined by cdf. The alternative hypothesis is that x does not have that distribution.

Result h is equal to '1' if we can reject the hypothesis, or '0' if we cannot. The function also returns the p-value which is the probability that x does not contradict the null hypothesis. We reject the hypothesis if the test is significant at the 5% level (if pvalue is less than 0.05). The p-value returned by kstest was used to estimate the goodness-of-fit of the hypothesis. As a result of hypothesis testing we found out that the Weibull distribution fits well the response time of the first (Oregon) and the third (Tokyo) replicas. The response time of the second replica (Sao Paulo) can be accurately approximated by the Gamma distribution.

Deducing Probability Density Functions of the System Response Time

Mathcad has been used at the second stage of our investigation to deduce theoretical distributions of system response times for different consistency levels. It also allows to estimate average system latency and to plot probability density functions. Mathcad worksheet is shown in Fig. 7. It includes seven modelling steps.

At the 1st step we define abscissa axis t and its dimension in milliseconds. Secondly, we set up parameters of replicas response time distribution functions estimated in Matlab and also their shifts on the abscissa axis (i.e. minimal response time values).

At the 3rd and 4th steps the replica response time probability density functions g 1 (t), g 2 (t), g 3 (t) and the corresponding cumulative distribution functions G 1 (t), G 2 (t), G 3 (t) are defined using Mathcad library functions dweibull and dgamma.

At the 5th step we define probability density functions of the system response time corresponding to different consistency levels by combining replicas pdf and cdf according to the proposed equations (1), ( 2) and (3). Probability distribution functions of replicas and system response times are shown in Figs. 8 and9. The bulk of the values of probability density function f ALL (t) is shifted to the right on the abscissa axis as it was expected. The shapes of the f ONE (t) and f QUORUM (t) probability density functions are also in line with the reasonable expectations and experimentally obtained probability density series (see Fig. 6). Finally, at steps 6 and 7 we estimate the system and replicas average response time by integrating their theoretical probability distribution functions.

a1 b1          g2 t ( ) 1 b2 dgamma t min2  ( ) b2 a2          g3 t ( ) 1 a3 dweibull t min3  ( ) a3 b3          G1 t ( ) 0 t t g1 t ( )    d  G2 t ( ) 0 t t g2 t ( )    d  G3 t ( ) 0 t t g3 t ( )    d  fALL t ( ) g1 t ( ) G2 t ( )  G3 t ( )  g2 t ( ) G1t ( )  G3 t ( )   g3 t ( ) G1t ( )  G2t ( )    fONE t ( ) g1 t ( ) 1 G2t ( )  ( )  1 G3t ( )  ( )  g2 t ( ) 1 G1t ( )  ( )  1 G3 t ( )  ( )   g3 t ( ) 1 G1 t ( )  ( )  1 G2 t ( )  ( )    fQUORUM t ( ) g1 t ( ) G2t ( )  g2 t ( ) G1t ( )   ( ) 1 G3t ( )  ( )  g1 t ( ) G3 t ( )  g3 t ( ) G1t ( )   ( ) 1 G2 t ( )  ( )   g2 t ( ) G3 t ( )  g3 t ( ) G2 t ( )   ( ) 1

Accuracy of Mathematical Modelling

Table 2 shows the deviation between the average values of the system and replicas response time estimated practically (see Table 1) and theoretically with the help of the obtained probability distribution functions. These results confirm the significant closeness between actual and modelled timing characteristics. To be sure that not only the average value can be accurately predicted we compare theoretical system probability density functions (see Fig. 9) and practically obtained probability density series (Fig. 6). With this purpose we estimated experimental and theoretical probabilities that system latency at different consistency levels is less than the specified time. The results of this comparison (see Table 3) show a close approximation of the experimental data by the proposed analytical models, especially for the consistency levels ONE and QUORUM. The probabilistic model of the system response time for consistency level ALL gives slightly optimistic prediction, though the average deviation from the experimental data is only 7% -that is close enough.

Conclusion and Lessons Learnt

When employing fault-tolerance techniques over the Internet and clouds, engineers need to deal with delays, their uncertainty, timeouts, adjudication of asynchronous replies from replicas, and other specific issues involved in global distributed systems.

The overall aim of this work was to study the impact of consistency on system latency in fault tolerant Internet computing.

Our experimental results clearly show that improving system consistency makes system latency worse. This finding confirms one of the generally accepted qualitative implications of the CAP theorem [START_REF] Brewer | Towards Robust Distributed Systems[END_REF][START_REF] Gilbert | Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services[END_REF]. However, so far system developers have not had any mathematical tools to help them to accurately predict the response time of large-scale replicated systems. While estimating the system worst-case execution time remains common practice for many applications (e.g. embedded computer systems, server fault-tolerance solutions, like STRATUS, etc.), this is no longer a viable solution for the wide-area service-oriented systems in which components can be distributed all over the Internet. In our previous works [START_REF] Gorbenko | Real Distribution of Response Time Instability in Service-Oriented Architecture[END_REF][START_REF] Gorbenko | Exploring Uncertainty of Delays as a Factor in End-to-End Cloud Response Time[END_REF] we demonstrated that extreme unpredictable delays exceeding the value of ten average response times can happen in such systems quite often. In this paper we have proposed a set of novel analytical models providing a quantitative basis for the system response time prediction depending on the consistency level provided for (or requested by) clients. The models allow us to derive the probability distribution function of the system response time which corresponds to a particular consistency level (ONE, ALL or QUORUM) by incorporating the probability density functions of the replica response times.

The validity of the proposed models has been verified against the experimental data reported in Section 3. It has been demonstrated that the proposed models ensure a significant level of accuracy in the system average response time prediction, especially in case of ONE and QUORUM consistency levels. The proposed models provide a mathematical basis for predicting latency of distributed fault and intrusion-tolerance techniques operating over the Internet. The models take into account the probabilistic uncertainty of replicas' response time and the required consistency level.

The practical application of our work is in allowing practitioners to predict system performance, and in offering them crucial support for the optimal timeout setup and for understanding the trade-off between system consistency and latency. Trading off system consistency against latency requires the knowledge of probability density functions (and parameter values) that accurately approximate replicas' response time. These probabilistic characteristics, which can be obtained by testing or during the trial usage, will need to be corrected at run-time or at tune-time to improve prediction accuracy. It would be possible to replace the response time probability density functions in the proposed models with probability density series. This would make it easier to use the models in practice.

8 Fig. 1 .

 81 Fig. 1. The CAP trade-offs.

Fig. 2 .

 2 Fig. 2. Fault-tolerant service-oriented system.

ResponseFig. 3 .Fig. 4 .

 34 Fig. 3. Response time of different web service replicas.

Fig. 5 .Fig. 6 .

 56 Fig. 5. Probability density series of replicas response times.

Fig. 7 .

 7 Fig. 7. Mathcad's worksheet.

Table 1 .

 1 Response time statistics.

	Response	Replica1	Replica2	Replica3	System consistency level
	Time, ms	(Oregon)	(Sao Paulo)	(Tokyo)	ONE QUORUM ALL
	Minimal	2324	2164	2344	2164 2324	2386
	Average	2428	2434	2588	2342 2449	2660
	Maximal	2821	3371	5573	2509 2830	5573
	Std. deviation 60	228	522	80	72	529
	4	Probabilistic Models of System Response Time
		for Different Consistency Levels

Table 2 .

 2 Accuracy of mathematical modelling.

		Replica1	Replica2	Replica3	System consistency level
		(Oregon)	(Sao Paulo)	(Tokyo)	ONE QUORUM ALL
		Approximating theoretical distributions and their parameters
	distribution	Weibull	Gamma	Weibull
	alpha	113.3578	1.5952	176.8796
	beta	2.3041	164.1599	1.7467
	x-shift	2324	2164	2344
			Average response time, ms
	measured	2428	2434	2588	2342	2449	2660
	modelled	2424	2426	2502	2341	2444	2567
	Deviation, %	0.18	0.34	3.32	0.03	0.19	3.51

Table 3 .

 3 Deviation between theoretical system pdf and pds obtained experimentally.

	Time, ms	pds	Probability that system latency is less than the specified time ONE QUORUM ALL pdf dev.,% pds pdf dev.,% pds pdf	dev.,%
	2175	0.01	0.009	10.00	0	0	-	0	0	-
	2225	0.11	0.116	5.45	0	0	-	0	0	-
	2275	0.23	0.252	9.57	0	0	-	0	0	-
	2325	0.43	0.385	10.47	0.01	0	-	0	0	-
	2375	0.59	0.596	1.02	0.08	0.097	21.25	0	0.003	-
	2425	0.84	0.858	2.14	0.43	0.434	0.93	0.11	0.073	33.64
	2475	0.99	0.975	1.52	0.72	0.752	4.44	0.29	0.263	9.31
	2525	1	0.998	0.20	0.89	0.903	1.46	0.52	0.476	8.46
	2575	1	1	0	0.96	0.961	0.10	0.63	0.643	2.06
	2625	1	1	0	0.96	0.984	2.50	0.72	0.761	5.69
	2675	1	1	0	0.99	0.994	0.40	0.8	0.841	5.13
	2725	1	1	0	0.99	0.998	0.81	0.85	0.892	4.94
	2775	1	1	0	0.99	0.999	0.91	0.88	0.924	5.00
	2825	1	1	0	0.99	1	1.01	0.89	0.945	6.18
	2875	1	1	0	1	1	0	0.91	0.959	5.38
	2925	1	1	0	1	1	0	0.91	0.969	6.48
	2975	1	1	0	1	1	0	0.92	0.977	6.20
	3025	1	1	0	1	1	0	0.94	0.982	4.47
	3075	1	1	0	1	1	0	0.95	0.987	3.89
	Average deviation, %	2.12			2.25			7.12

Acknowledgements

We are grateful to Aad van Moorsel for his feedback on the earlier version of this work and Batyrkhan Omarov for his help with running some of the experiments. Alexander Romanovsky is partially supported by the EPSRC TRAMS-2 platform grant.