A Omar Portillo-Dominguez
email: andres.portillo-dominguez@ucdconnect.ie

Miao Wang
email: miao.wang@ucd.ie

John Murphy
email: j.murphy@ucd.ie

Damien Magoni
email: magoni@labri.fr

Automated WAIT for Cloud-based Application Testing

Keywords: B.8 [Performance and Reliability]: Reliability, Testing, and Fault-Tolerance; D.2.5 [Testing and Debugging]: Testing tools Algorithms, Measurement, Performance Cloud Computing, Performance Testing, Automation, Performance Analysis, Expert Tools

Cloud computing is causing a paradigm shift in the provision and use of software. This has changed the way of obtaining, managing and delivering computing services and solutions. Similarly, it has brought new challenges to software testing. A particular area of concern is the performance of cloudbased applications. This is because the increased complexity of the applications has exposed new areas of potential failure points, complicating all performance-related activities. This situation makes the performance testing of cloud applications very challenging. Similarly, the identification of performance issues and the diagnosis of their root causes are time-consuming and complex, usually require multiple tools and heavily rely on expertise. To simplify these tasks, hence increasing the productivity and reducing the dependency on human experts, this paper presents a lightweight approach to automate the usage of expert tools in the performance testing of cloud-based applications. In this paper, we use a tool named Whole-system Analysis of Idle Time to demonstrate how our research work solves this problem. The validation involved two experiments, which assessed the overhead of the approach and the time savings that it can bring to the analysis of performance issues. The results proved the benefits of the approach by achieving a significant decrease in the time invested in performance analysis while introducing a low overhead in the tested system.

INTRODUCTION

As documented by multiple authors [START_REF] Gao | Cloud Testing-Issues, Challenges, Needs and Practice[END_REF][START_REF] Jogalekar | Evaluating the scalability of distributed systems[END_REF], Cloud computing has proven to be a cost-effective and flexible paradigm by which scalable computing power and software services (e.g., computing infrastructures or software resources) can be delivered. This situation has brought many new business opportunities in the enterprise world. For example, the revenue estimations of Gartner1 indicate that the worldwide Software as a Service (SaaS) revenue (one of the three most well accepted cloud delivery models [START_REF] Gao | SaaS Testing on Clouds. Issues, Challenges, and Needs[END_REF]) is expected to reach US$ 22.1 billion by 2015.

It is also well documented that the Cloud has brought major challenges to the software testing landscape [START_REF] Collard | Performance innovations, testing implications[END_REF][START_REF] Riungu | Research Issues for Software Testing in the Cloud[END_REF]. A particular area of concern is the performance of SaaS applications. This is because Cloud vendors and clients commonly have very demanding quality of service requirements [START_REF] Oldt | Behind-the-Scenes at Salesforce[END_REF]. However, the innate characteristics of a Cloud environment (such as elasticity, scalability and multi-tenancy) have augmented the complexity of the applications, complicating all performance-related activities [START_REF] Gao | SaaS Testing on Clouds. Issues, Challenges, and Needs[END_REF][START_REF] Tsai | Testing the Scalability of SaaS applications[END_REF]. Under these conditions, doing performance testing is complex and time-consuming. A special challenge, documented by multiple authors [START_REF] Angelopoulos | GcLite: An Expert Tool for Analyzing GC Behavior[END_REF][START_REF] Parsons | Detecting Performance Antipatterns in Component Based Enterprise Systems[END_REF], is that current performance tools heavily rely on human experts to understand their output. Also multiple sources are commonly required to diagnose performance problems, especially in highly distributed environments, such as the Cloud. This increases the expertise required to do performance analysis, which is usually held by only a small number of experts inside an organization [START_REF] Spear | Making Performance Analysis and Tuning Part of the Software Development Cycle[END_REF]. Therefore it could potentially lead to bottlenecks where certain activities can only be done by these experts, impacting the productivity of the testing teams [START_REF] Angelopoulos | GcLite: An Expert Tool for Analyzing GC Behavior[END_REF]. This risk is higher in Cloud environments, as vendors commonly promise up to 24x7 availability of their services. While Cloud computing provides the means to deliver these services, human effort is still required to complete certain key tasks, such as the performance testing of those environments [START_REF] Riungu | Research Issues for Software Testing in the Cloud[END_REF].

To simplify the performance analysis and diagnosis, hence increasing the productivity and reducing the dependency on human experts, many researchers have been developing tools with built-in expertise [START_REF] Altman | Performance analysis of idle programs[END_REF][START_REF] Angelopoulos | GcLite: An Expert Tool for Analyzing GC Behavior[END_REF]. However, various limitations exist in these tools that prevent their efficient usage in the performance testing of Cloud environments. The data collection usually needs to be controlled manually which, in an environment composed of multiple nodes to monitor and coordinate simultaneously, is very time-consuming and errorprone due to the vast amount of data to collect and consolidate. This challenge is more complex because the data needs to be processed periodically during the test execution to get incremental results. A similar problem occurs with the outputs, where a tester commonly gets multiple reports, one for each monitored node per data processing cycle. Even though these limitations might be manageable in small testing environments, they prevent the efficient usage of these tools in bigger environments (probably of hundreds of nodes or more), which commonly exist in the Cloud [START_REF] Gao | Cloud Testing-Issues, Challenges, Needs and Practice[END_REF]. Similarly, if a tool requires heavy human effort, this might limit its applicability. On the contrary, automation could encourage the adoption of a technique. This strategy has proven successful in performance testing [START_REF] Dosinger | Continuous integration servers for increasing effectiveness of automated testing[END_REF][START_REF] Shahamiri | A Comparative Study on Automated Software Test Oracle Methods[END_REF], and specially useful in scenarios of continuous testing in the Cloud [START_REF] Jogalekar | Evaluating the scalability of distributed systems[END_REF], which requires to test the SaaS applications whenever a change is detected (e.g., bug fixes or feature enhancements).

This paper proposes a lightweight automation approach to address the common usage limitations of an expert system in SaaS performance testing. During our research development work we have successfully applied our approach to the IBM Whole-system Analysis of Idle Time tool (WAIT) 2 . This publicly available tool is a lightweight expert system that helps to identify the main performance inhibitors in Java systems. Our work was validated through two experiments using real-world applications. The results provided evidence about the benefits of the approach: It drastically reduced the effort required by a tester to use and analyze the outputs of the selected expert tool (WAIT). This usage simplification translated into a quicker identification of performance issues. Also the introduced overhead was low (up to 3% when using a common industry Sampling Interval of 480 seconds). The contributions of this paper are:

1. A novel lightweight approach to automate the usage of expert systems in SaaS performance testing. 2. A practical validation of the approach consisting of a prototype and two experiments. The first one proves that the overhead of our approach is minimal, and the second proves its productivity benefits.

BACKGROUND

Idle-time analysis is a methodology that is used to identify the root causes of under-utilized resources. This approach, proposed in [START_REF] Altman | Performance analysis of idle programs[END_REF], is based on the behavior that performance problems in multi-tier applications usually manifest as idle time of waiting threads. WAIT is an expert system that implements this technique, and it has proven successful in simplifying the detection of performance issues and their root causes in Java systems [START_REF] Altman | Performance analysis of idle programs[END_REF]. WAIT is based on non-intrusive sampling mechanisms available at Operating System level (i.e. "ps" command in Unix) and the Java Virtual Machine (JVM), in the form of Javacores 3 (snapshots of the JVM state, offering information such as threads and memory).

The fact that WAIT uses standard data sources makes it non-disruptive, as no special flags, restart or instrumentation are required to use it. WAIT also requires infrequent samples to perform its diagnosis, so it has low overhead.

2 http://wait.ibm.com 3 http://www-01.ibm.com/support/docview.wss?uid= swg27017906&aid=1

Given its strengths, WAIT is a promising candidate to reduce the dependence on a human expert and the time required for performance analysis in the cloud. However, as with many expert systems that could be used for this purpose, the volume of data generated can be difficult to manage and efficiently process this data can be an impediment to their adoption. For example, the effort required to manually collect data to feed WAIT and the number of reports a tester gets from the WAIT system are approximately linear with respect to the number of nodes and the update frequency of the results. These limitations make WAIT a good candidate to apply our proposed approach.

PROPOSED APPROACH

The objective of this work was to automate the manual processes involved in the usage of an expert system. This logic will execute concurrently with the performance test, periodically collecting the required samples, then incrementally processing them with the expert system to get a consolidated output. This scenario is depicted in Figure 1, where our automation shields the tester from the complexities of using the expert system, so that she only needs to interact with the load testing tool.

Figure 1: Contextual view of the proposed approach

The detailed approach is depicted in Figure 2. To start, some inputs are required: The list of application nodes to monitor; a Sampling Interval to control how often the samples will be collected; a Time Threshold to define the maximum time between data uploads; a Hard Disk (HD) Threshold to define the maximum storage quota for collected data (to prevent its uncontrolled growth); and a Backup flag to indicate if the collected data should be backed up before any cleaning occurs.

The process starts by initialising the configured parameters.

Then it gets a new RunId, value which will uniquely identify the test run and its collected data. This value is propagated to all the nodes. On each node, the HD Threshold and the next Time Threshold are initialized. These thresholds allow the approach to adapt to different usage scenarios. For example, if a tester prefers to get updated results as soon as new data is collected, she could set the HD Threshold to zero. Next each application node starts the following loop in parallel until the performance test finishes: A new set of data samples is collected. After the collection finishes, the process checks if any of the two thresholds have been reached. If either of these conditions has occurred, the data is sent to the expert system (labeling the data with the RunId so that information from different nodes can be identified as part of the same test run). If a Backup was enabled, the is copied to the backup destination before it is deleted to keep the HD usage below the threshold. As certain data collections can be costly (i.e. the generation of a memory dump Java can take minutes and hundreds of megabytes of HD), the Backup could be useful to enable off-line analysis of the collected data. Then updated outputs from the expert system are obtained and the Time Threshold reset. Finally, the logic awaits the Sampling Interval before a new iteration starts. Once the performance test finishes, any remaining collected data is sent (and backed up if configured) so that it is also processed. Lastly the data is cleared and the final outputs of the expert system are obtained.

Architecture. The proposed approach is implemented with the architecture presented in the component diagram4 of Figure 3. It is composed of two main components: The Control Agent is responsible of interacting with the load testing tool to know when the test starts and ends. It is also responsible of getting the runId and propagate it to all the application nodes. The second component is the Node Agent which is responsible for the collection, upload, backup and cleanup in each application node. On each agent, its control logic and Helper functionality (i.e. the calculation of the thresholds in the Node Agent), are independent of the target expert system and load testing tool. On the contrary, the logic that interfaces with the tools needs to be customized. To minimize the code changes, this logic is encapsulated in Wrapper packages which are only accessed through their interfaces. This scenario is presented in Figure 4 which shows the structure of the package Expert System Wrapper. It contains a main interface IExpertSystem to expose all required actions and an abstract class for all the common functionality. This hierarchy can then be extended to support a specific expert system on different operating systems.

These two components communicate through commands, following the Command5 Design Pattern: The Control Agent Prototype. Based on the proposed approach, a prototype has been developed. The Control Agent was implemented as a plugin for the Rational Performance Tester (RPT) 6 , which is a common load testing tool in the industry; the Node Agent was implemented as a Java Web Application, and WAIT was the selected expert system due to its analysis capabilities (discussed in Section 2).

EXPERIMENTAL EVALUATION

Two experiments were performed. The first one evaluated the overhead introduced by our proposed approach. Meanwhile, the second one assessed the productivity benefits that a tester can gain by using the proposed approach. Two environment configurations were used: One was composed of an RPT node, one application node and a WAIT Server node; the other was composed of a RPT node, a load balancer node, two application nodes and a WAIT Server node. All VMs had 2 virtual CPUs, 3GB of RAM, and 50GB of HD; running Linux Ubuntu 12.04L 64-bit, and Oracle Hotspot JVM version 7 with a maximal heap of 2GB.

Experiment 1: Overhead Evaluation. The objective here was to quantify the overhead of the proposed approach and involved the assessment of four metrics: Throughput (tps), response time (ms), CPU (%) and memory (MB) utilisation. All metrics collected through RPT. Also, two realworld applications were used: iBatis JPetStore 4.0 7 which is an e-commerce shopping cart. It ran over an Apache Tomcat 6.0.35. The other application was IBM WebSphere Portal 8.0.18 , a leading enterprise portal solution. It ran over an IBM WebSphere Application Server 8.0.0.5.

Firstly, the overhead was measured in a single-node environment using three combinations of WAIT: The applications alone to get a baseline (BL), the applications with manual WAIT data collection, and the applications with the automated WAIT (AWAIT). For each combination using WAIT, the Sampling Interval (SI) was configured to 480 seconds (a commonly used value) and 30 seconds (minimum value recommended for WAIT). The remaining test configurations were set to reflect real-world conditions: A workload of 2,000 concurrent users; a duration of 1 hour; a HD Threshold of 100MB; and a Time Threshold of 10 minutes.

For JPetStore, each test run produced around 500k transactions. The results presented in Table 1 showed that using WAIT with a SI of 480 seconds had practically no impact in terms of response time and throughput. Furthermore the difference in resource consumption between the two modalities of WAIT was around 1%. This difference was mostly related to the presence of the Node Agent because the uploaded data was very small (around 200KB every 10 minutes). When a SI of 30 seconds was used, the impact on response time and throughput appeared. Since the throughput was similar between the WAIT modalities, the impact was caused by the Javacore generation as it is the only step shared between the modalities. On average, the generation of a Javacore took around 1 second. Even though this cost was insignificant in the higher SI, with 30 seconds the impact was visible. On the contrary, the difference in response time between the two modalities of WAIT was caused by the upload and backup processes (around 4MB of data every 10 minutes) which are steps exclusive to AWAIT. In terms of resource consumption, the differences between the WAIT modalities remained within 1%. For Portal, each test run produced around 400k transactions and the results are presented in Table 2. They show similar trends to the results in Table 1, but a few key differences were identified: First, the impact on response time and throughput were visible even with the SI of 480 seconds. Also, the differences between the results for the two SIs were bigger. As the experimental conditions were the same, it was initially assumed that these differences were related to the dissimilar functionality of the tested applications. This was confirmed after analysing the Javacores generated by Portal, which allowed to quantify the differences in behavior of Portal: The average size of a Javacore was 5.5MB (450% bigger than JPetStore's), its average generation time was 2 sec (100% bigger than JPetStore's), and a maximum generation time of 3 sec (100% bigger than JPetStore's). To explore the small differences between the runs and the potential environmental variations that were experienced during the experiments, a Paired t-Test9 was done (using a significance level of p<0.1) to evaluate if the differences were statistically significant. This analysis indicated that they were only significant when using a SI of 30 seconds. This reinforced the conclusion that the overhead was low and the observation that the SI of 480 seconds was preferable.

A second test was done to validate that the overhead remained low in a multi-node environment over a longer test run. This test used JPetStore and the AWAIT with a SI of 480 seconds. The rest of the set-up was identical to the previous tests except the workload which was doubled to compensate for the additional application node and the test duration which was increased to 24 hours. Even though the results were slightly different than the single-node run, they proved that the proposed approach was reliable, as using AWAIT had minimal impact in terms of performance (0.5% in RTAV G, 0.2% in RTMAX and 1.4% in TAV G). The consumption of resources behaved similarly (an increment of 0.85% in CP UAV G and 2.3% in M EMAV G). A paired t-Test also indicated that these differences, compared with the baseline, were not significant.

In conclusion, the results proved that the overhead caused the automated approach was low, therefore the results of a performance test are not compromised. Due to the impact that the SI and the application behavior might have on the overhead, it is important to consider these factors in the configuration. In our case, a SI of 480 seconds proved efficient in terms of overhead for the two tested applications.

Experiment 2: Assessment of productivity benefits.

Here the objective was to assess benefits our approach Figure 5: Lock contention issue in the WAIT report and the actual source code brings to a performance tester. First, three common performance issues were injected in JPetStore: A lock contention bug (composed of a heavy calculation within a synchronized block of code), an I/O latency bug (composed of an expensive file reading method) and a deadlock bug (composed of an implementation of the classic "friends bowing" example10 ). Then AWAIT monitored the application to assess how well it was able to identify the injected bugs and estimate the corresponding time savings in performance analysis. The set-up was identical to the multi-node test previously described except the duration which was one hour.

The 1st ranked issue was not an injected bug but an issue related to the clustering set-up of Tomcat. The 2nd ranked issue was the lock contention. It is worth noting that both issues were detected since the early versions of the incremental report from the automated tool. Their high frequency (above 96% of the samples) could have led a tester to pass this information to the development team so that the diagnosis could start far ahead of the test completion. The final report reinforced the presence of these issues.

After identifying an issue, a tester can see more details, including the type of problem, involved class, method and source code line. Figure 5 shows the information of our lock contention bug, which was located in the class LockCon-tentionBug, the method generateBug and the line 20. When comparing this information with the actual code, one can see it is precisely the line where the bug was injected (taking a class lock before doing a very CPU intensive logic). The 3rd ranked issue in the report was a symptom of the lock contention bug, suggesting it was a major problem (the issues were correlated by comparing their information, which pinpointed to the same class and method). Finally, the I/O latency bug was identified in 4th place.

The deadlock bug did not appear in this test run, somehow prevented by the lock contention bug which had a bigger impact than planned. As in any regular test phase, the identified bugs were fixed and a new run was done to review if any remaining performance issues existed. Not surprisingly, the deadlock bug appeared in the second test run.

As all injected bugs were identified, this experiment was con-sidered successful. In terms of time, two main savings were documented. First, the automated approach practically reduced the effort of using WAIT to zero. After a one-time installation which took no more than 15 minutes for all nodes, the only additional effort required to use the automated approach was a few seconds spent configuring it (i.e. to change the SI). The second time saving occurred in the analysis of the WAIT reports. Previously, a tester would have ended with multiple reports. Now a tester only needs to monitor a single report which is refreshed periodically. Overcoming the usage constraints of our selected expert system (WAIT) also allowed exploiting its expert knowledge capabilities. Even though it might be hard to define an average time spent identifying performance issues, a conservative estimate of 2 hours per bug could help to quantify these savings. In our experiment, instead of spending an estimated 6 hours analysing the issues, it was possible to identify them and their root causes in a matter of minutes with the information provided by the WAIT report. As seen in the experiment, additional time can be saved if the relevant issues are reported to developers in parallel to the test execution. This is especially valuable in long-term runs, which are common in performance testing and typically last several days.

To summarize the results, they were very promising as it was possible to measure the benefits that a tester can gain by using an expert system through our proposed automation approach: After a quick installation, the time required to use the automated WAIT was minimal. Moreover a tester now only needs to monitor a single WAIT report, which offers a consolidated view of the results.

RELATED WORK

Performance testing. Multiple research works have focused on improving its involved processes. For example, the authors of [START_REF] Ciortea | Cloud9: A software testing service[END_REF] introduced a cloud-based testing harness which uses an execution engine to parallelise the run of test scripts on large shared-clusters of nodes. Meanwhile, the work on [START_REF] Badia | Enabling Large-Scale Testing of IaaS Cloud Platforms on the Grid 5000 Testbed[END_REF] presents an approach to automate the deployment of the major open-source IaaS cloud kits (such as OpenNebula11 ) on Grid 5000 environments. Other efforts have concentrated on evaluating the benefits of cloud testing in practice. For example, the work on [START_REF] Lynch | Testing a Cloud Application: IBM SmartCloud iNotes[END_REF] presents an industrial study case of the new methodologies and tools that have developed to streamline the testing of their cloudbased applications. Finally, other works have focused on automation. However, most of this research has focused on automating the generation of load test suites [START_REF] Albert | Resource-Driven CLP-Based test generation[END_REF][START_REF] Briand | Using genetic algorithms for early schedulability analysis and ST in RT systems[END_REF]. For example, [START_REF] Bayan | Automatic stress and load testing for embedded systems[END_REF] proposes an approach to automate the generation of test cases based on specified levels of load and combinations of resources. Performance Analysis. Other works have focused on addressing the challenges in performance analysis in the Cloud [START_REF] Candea | Automated Software Testing as a Service (TaaS)[END_REF][START_REF] Vijayanathan Naganathan | The Challenges Associated with SaaS Testing[END_REF]. A high percentage of the proposed techniques require instrumentation. For example, the authors in [START_REF] Yang | Perracotta: mining temporal api rules from imperfect traces[END_REF] instrument the source code of the monitored applications to mine the sequences of call graphs under normal operation, information which is later used to infer any relevant error patterns. A similar case occurs with the works presented in [START_REF] Csallner | Dsd-crasher: a hybrid analysis tool for bug finding[END_REF][START_REF] Hangal | Tracking down software bugs using automatic anomaly detection[END_REF] which rely on instrumentation to dynamically infer invariants within the applications and detect programming errors. In all these cases, instrumentation would obscure the performance of an application during performance testing hence discouraging their usage. On the contrary, our proposed approach does not require any instrumentation.

Finally, the authors of [START_REF] Barham | Using magpie for request extraction and workload modelling[END_REF][START_REF] Hoorn | Design and Application of the Kieker Framework[END_REF] present frameworks to monitor software services. Unlike these works, which have been designed to assist on operational support activities, our approach has been designed to address the specific needs of a tester in the performance testing of cloud environments, isolating her from the complexities of an expert system.

CONCLUSIONS

The identification of performance problems in cloud environments is complex and time-consuming. Performance testing is challenging because the complexity of the cloud-based applications has exposed new areas of potential failure points. While the identification of performance issues and the diagnosis of their root causes usually require multiple tools and heavily rely on expertise. To simplify these tasks, this work proposed a novel approach to automate the usage of an expert system in a cloud test environment. A prototype was developed around the WAIT tool and then its benefits and overhead were assessed. The results showed that the introduced overhead was low (between 0% to 3% when using a common industry Sampling Interval ). Also the results showed the time savings gained by applying the approach. In our case, the effort to utilize WAIT was reduced to seconds. This optimization then simplified the identification of performance issues. In our case, all defects injected in JPet-Store were detected in a matter of minutes using WAIT's consolidated outputs. In contrast, a manual analysis might have taken hours.

Figure 2 :

 2 Figure 2: Process flow -automation approach

Figure 3 :Figure 4 :

 34 Figure 3: Component diagram of architecture

Table 1 :

 1 JPetStore -Overhead results

	WAIT	RT AV G	RT M AX	T AV G	CP U AV G	M EM AV G
	Modality & SI	(ms)	(ms)	(tps)	(%)	(MB)
	None (BL)	1889	44704	158	36	1429
	WAIT, 480s	0.0%	0.0%	0.0%	1.1%	3.0%
	AWAIT, 480s	0.0%	0.0%	0.0%	2.0%	3.7%
	WAIT, 30s	1.6%	0.4%	-3.1%	1.47%	4.1%
	AWAIT, 30s	4.4%	0.5%	-4.0%	2.53%	4.4%

Table 2 :

 2 Portal -Overhead results

	WAIT	RT AV G	RT M AX	T AV G	CP U AV G	M EM AV G
	Modality & SI	(ms)	(ms)	(tps)	(%)	(MB)
	None (BL)	4704	40435	98	76	3171
	WAIT, 480s	0.7%	0.6%	-0.1%	0.63%	2.2%
	AWAIT, 480s	3.4%	1.0%	-2.8%	1.13%	4.1%
	WAIT, 30s	14.9%	5.4%	-5.6%	2.23%	5.3%
	AWAIT, 30s	16.8%	9.1%	-5.7%	2.97%	6.0%

http://www.gartner.com/newsroom/id/1963815

https://www.ibm.com/developerworks/rational/library/dec04/ bell/

http://www.oodesign.com/command-pattern.html

http://www-03.ibm.com/software/products/us/en/performance

http://sourceforge.net/projects/ibatisjpetstore/

http://www-03.ibm.com/software/products/us/en/portalserver

http://www.aspfree.com/c/a/braindump/comparingdata-sets-using-statistical-analysis-in-excel/

http://docs.oracle.com/javase/tutorial/essential/concurrency/ deadlock.html

http://opennebula.org/

ACKNOWLEDGMENTS

This work was supported, in part, by Science Foundation Ireland grant 10/CE/I1855 to Lero -the Irish Software Engineering Research Centre (www.lero.ie).


A Omar Portillo-Dominguez
email: andres.portillo-dominguez@ucdconnect.ie

Miao Wang
email: miao.wang@ucd.ie

John Murphy
email: j.murphy@ucd.ie

Damien Magoni
email: magoni@labri.fr

Nick Mitchell
email: nickm@ucd.ie

Peter F Sweeney

Erik Altman
email: ealtman@ucd.ie

Towards an Automated Approach to Use Expert Systems in the Performance Testing of Distributed Systems

Keywords: B.8 [Performance and Reliability]: Reliability, Testing, and Fault-Tolerance; D.2.5 [Testing and Debugging]: Testing tools Algorithms, Measurement, Performance Performance Testing, Automation, Performance Analysis, Expert Tools, Distributed Systems

établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

It is an accepted fact in the industry that performance is a critical dimension of quality and should be a major concern of any software project. This is especially true at enterpriselevel, where the system performance plays a central role in using the software to achieve business goals. However it is not uncommon that performance issues occur and materialize into serious problems in a significant percentage of applications (i.e. outages on production environments or even cancellation of software projects). For example, a 2007 survey applied to information technology executives [START_REF][END_REF] reported that 50% of them had faced performance problems in at least 20% of their deployed applications. Similarly, many research studies have documented the magnitude of this problem. For example, in [START_REF] Jin | Understanding and detecting real-world performance bugs[END_REF] authors found 332 previously unknown performance problems in the latest versions of five mature open-source software suites. This situation is partially explained by the pervasive nature of performance, which makes it hard to assess because performance is practically influenced by every aspect of the design, code, and execution environment of an application. The latest trends in information technology (such as Service Oriented Architecture1 and Cloud Computing2 ) have also augmented the complexity of applications further complicating all activities related to performance.

Under these conditions, it is not surprising that doing performance testing is complex and time-consuming. A special challenge, documented by multiple authors [START_REF] Woodside | The Future of Software Performance Engineering[END_REF][START_REF] Parsons | Detecting Performance Antipatterns in Component Based Enterprise Systems[END_REF][START_REF] Angelopoulos | GcLite: An Expert Tool for Analyzing Garbage Collection Behavior[END_REF], is that current performance tools heavily rely on human experts to be configured properly and to interpret their outputs. Also multiple sources are commonly required to diagnose performance problems, especially in highly distributed environments. For instance in Java: thread dumps, garbage collection logs, heap dumps, CPU utilization and memory usage, are a few examples of the information that a tester could need to understand the performance of an application. This problem increases the expertise required to do performance analysis, which is usually held by only a small number of experts inside an organization [START_REF] Spear | Making Performance Analysis and Tuning Part of the Software Development Cycle[END_REF]. Therefore this issue could potentially lead to bottlenecks where certain activities can only be done by these experts, impacting the productivity of the testing teams [START_REF] Angelopoulos | GcLite: An Expert Tool for Analyzing Garbage Collection Behavior[END_REF].

To simplify the performance analysis and diagnosis, hence increasing the productivity and reducing the dependency on human experts, many researchers have been developing tools with built-in expertise [START_REF] Altman | Performance analysis of idle programs[END_REF][START_REF] Ammons | Finding and Removing Performance Bottlenecks in Large Systems[END_REF][START_REF] Angelopoulos | GcLite: An Expert Tool for Analyzing Garbage Collection Behavior[END_REF]. However, various limitations exist in these tools that prevent their efficient usage in the performance testing of highly distributed environments. Firstly, these tools still need to be manually configured, situation which might impact their accuracy, as their outputs are commonly sensitive to the used configuration. Therefore, if an inappropriate configuration is used, the tools might fail to obtain the desired outputs. Sequentially, the data collection usually needs to be controlled manually which, in an environment composed of multiple nodes to monitor and coordinate simultaneously, is very time-consuming and error-prone due to the vast amount of data to collect and consolidate. This challenge is more complex if the data needs to be processed periodically during a test execution to get incremental results. A similar problem occurs with the outputs, where a tester commonly gets multiple reports, one for each monitored node per data processing cycle.

Even though these limitations might be manageable in small testing environments, they prevent the efficient usage of these tools in bigger environments. To exemplify this problem, let's use the Eclipse Memory Analyzer Tool3 (MAT), which is a popular open source tool to identify memory consumption issues in Java. If a tester wants to use MAT to monitor an environment composed of 100 nodes during a 24-hour test run and get incremental results every hour, she would need to manually coordinate the data gathering of memory snapshots and the generation of the tool's reports. These steps conducted periodically for the 100 nodes every hour, which yields a total of 2400 iterations. Moreover the tester would have to review the multiple reports she would get per hour to evaluate if any memory issues exist. As an alternative, she may concentrate the analysis on a single node, assuming it is representative of the whole system. However it generates the risk of potentially overlooking issues in the tested application. An additional assumption in the above scenario is that an appropriate configuration was used. Otherwise, the tester would have also experienced the risk of potentially overlooking issues.

In addition to these challenges, the overhead generated by any technique should be low to minimize the impact it has in the tested environment (i.e. inaccurate results or abnormal behavior). Otherwise the technique would not be suitable for performance testing. For example, instrumentation 4 is currently a common approach used in performance analysis to gather input data [START_REF] Yang | Perracotta: mining temporal api rules from imperfect traces[END_REF][START_REF] Hangal | Tracking down software bugs using automatic anomaly detection[END_REF][START_REF] Csallner | Dsd-crasher: a hybrid analysis tool for bug finding[END_REF][START_REF] Chen | Path-based failure and evolution management[END_REF]. However, it has the downside of obscuring the performance of the instrumented applications, hence compromising the results of performance testing. Similarly, if a tool requires heavy human effort, this might limit its applicability. On the contrary, automation could encourage the adoption of a technique. As documented by the authors in [START_REF] Shahamiri | A Comparative Study on Automated Software Test Oracle Methods[END_REF][START_REF] Ãűsinger | Communicating continuous integration servers for increasing effectiveness of automated testing[END_REF], this strategy has proven successful in performance testing.

Finally, to ensure that our research is helpful to solve reallife problems in the software industry, we have been working with our industrial partner, IBM System Verification Test team (SVT), to understand the challenges in their testing activities. Their feedback confirms that there is a real need to automate the usage of expert tools so that testers can do analysis tasks in less time and get more meaningful results that can be conveyed with the development teams. This paper proposes an adaptive automation framework that addresses the common usage limitations of an expert system in performance testing. During our research development work we have successfully applied our approach to the IBM Whole-system Analysis of Idle Time tool (WAIT) 5 . This publicly available tool is a lightweight expert system that helps to identify the main performance inhibitors in Java systems. Our work was validated through two experiments. The first experiment evaluated the accuracy of the proposed adaptive framework. The results demonstrated that the adaptive framework is able to configure WAIT, based on a desired usage scenario, without the need of manually tuning the tool. The second experiment assessed the productivity gains that our framework brought to the performance testing process. The results provided evidence about the benefits of the framework, as the effort required by a tester to use and analyze the outputs of the selected expert tool (WAIT) was reduced in 68%. This usage simplification translated into a quicker identification of performance issues; which leads to a reduction in the duration of the performance testing activities of 27%.

The main contributions of this paper are:

1. A novel adaptive framework to automate the usage of expert systems in performance testing. 2. Two novel adaptive policies to self-configure an expert system in performance testing. 3. A practical validation of the approach consisting of the implementation around the WAIT tool and two experiments. The first experiment demonstrates that the accuracy of our framework, and the second experiment demonstrates its productivity benefits. The rest of this paper is structured as follows: Section 2 discusses the background and related work. Section 3 explains the proposed framework, and the prototype. Section 4 describes the assessment performed to identify suitable adaptive policies; while Section 5 presents the proposed adaptive policies. Section 6 describes the performed experiments. Finally, Section 7 shows the conclusions and future work.

BACKGROUND AND RELATED WORK

Idle-time analysis is a methodology to identify the root causes of under-utilized resources. This approach, proposed in [START_REF] Altman | Performance analysis of idle programs[END_REF], is based on the observed behavior that performance problems in multi-tier applications usually manifest as idle time of waiting threads. WAIT is an expert system that implements the idle-time analysis and identifies the main performance inhibitors that exist on a system. Moreover it has proven successful in simplifying the detection of performance issues and their root causes in Java systems [START_REF] Altman | Performance analysis of idle programs[END_REF][START_REF] Wu | A Self-Optimizing Workload Management Solution for Cloud Applications[END_REF].

WAIT is based on non-intrusive sampling mechanisms available at Operating System level (i.e. "ps" command in a Unix environment) and the Java Virtual Machine (JVM), in the form of Javacores 6 (diagnostic feature to get a snapshot of the JVM state, offering information such as threads, locks and memory). The fact that WAIT uses standard data sources makes it non-disruptive, as no special flags, restart or instrumentation are required to use it. It also requires infrequent samples to perform its diagnosis, so it has low overhead. From an end-user perspective, WAIT is simple: A user only needs to collect as much data as desired, upload it to a public web page and get a report with the findings. This process can be repeated multiple times to monitor a system through time. Internally, WAIT uses an engine built on top of a set of expert rules to perform the analysis.

Given its strengths, WAIT is a promising candidate to reduce the dependence on a human expert and reduce the time required for performance analysis. However, as with many expert systems that could be used for testing distributed software systems, the volume of data generated can be difficult to manage and efficiently process this data can be an impediment to their adoption. Similarly, the accuracy of WAIT depends on its configuration and the preferable configuration might vary depending on the application and usage scenario. These characteristics make WAIT a good candidate to apply our proposed framework.

Automation in Testing. The idea of applying automation in the performance testing domain is not new. However, most of the research has focused on automating the generation of load test suites [START_REF] Albert | Resource-Driven CLP-Based test case generation[END_REF][START_REF] Bayan | Automatic stress and load testing for embedded systems[END_REF][START_REF] Zhang | Automated test case generation for the stress testing of multimedia systems[END_REF][START_REF] Briand | Using genetic algorithms for early schedulability analysis and ST in RT systems[END_REF][START_REF] Chen | Yet Another Performance Testing Framework[END_REF][START_REF] Garousi | Traffic-aware stress testing of distributed systems based on uml models[END_REF][START_REF] Wang | Model-based load testing of web applications[END_REF]. For example [START_REF] Bayan | Automatic stress and load testing for embedded systems[END_REF] proposes an approach to automate the generation of test cases based on specified levels of load and combinations of resources. Similarly, [START_REF] Chen | Yet Another Performance Testing Framework[END_REF] presents an automation framework that separates the application logic from the performance testing scripts to increase the re-usability of the test scripts. Meanwhile [START_REF] Wang | Model-based load testing of web applications[END_REF] presents a framework designed to automate the performance testing of web applications and which internally utilizes two usage models to simulate the users' behaviors more realistically.

Other research efforts have concentrated on automating specific analysis techniques. For example, the authors of [START_REF] Yu | Practical isolation of failure-inducing changes for debugging regression faults[END_REF] present a combination of coverage analysis and debugging techniques to automatically isolate failure-inducing changes. Similarly, the authors of [START_REF] Li | Dynamically Validating Static Memory Leak Warnings[END_REF] developed a technique to reduce the number of false memory leak warnings generated by static analysis techniques by automatically validating and categorizing those warnings.

Finally, other researchers have proposed frameworks to support different software engineering processes. For example, the authors of [START_REF] Hoorn | ÃČ â ȂŽ ÃĆ Âĺ INFORMATIK INSTITUT F UR Continuous Monitoring of Software Services : Design and Application of the Kieker Framework CHRISTIAN-ALBRECHTS-UNIVERSIT AT Continuous Monitoring of Software Services : Design and Application of the Kieker Framework[END_REF][START_REF] Barham | Using magpie for request extraction and workload modelling[END_REF] present frameworks to monitor software services. Both frameworks monitor the resource utilisation and the component interactions within a system, but target different technologies ( [START_REF] Hoorn | ÃČ â ȂŽ ÃĆ Âĺ INFORMATIK INSTITUT F UR Continuous Monitoring of Software Services : Design and Application of the Kieker Framework CHRISTIAN-ALBRECHTS-UNIVERSIT AT Continuous Monitoring of Software Services : Design and Application of the Kieker Framework[END_REF] focuses on Java and [START_REF] Barham | Using magpie for request extraction and workload modelling[END_REF] on Microsoft technologies). Unlike these works, which have been designed to assist on operational support activities, our proposed framework has been designed to address the specific needs of a tester in the performance testing, isolating her from the complexities of an expert system.

6 http://www-01.ibm.com/support/docview.wss? uid=swg27017906&aid=1 Performance Analysis. A major research trend has focused on identifying performance bugs and their root causes. For example, the work on [START_REF] Xiao | Xusheng. Context-Sensitive Delta Inference for Identifying Workload-Dependent Performance Bottlenecks[END_REF] proposes an approach to predict the workload-dependent performance bottlenecks (WDPBs) through complexity models. The work on [START_REF] Yu | SimRacer: an automated framework to support testing for process-level races[END_REF] presents a technique to detect processes accessing a shared resource without proper synchronization; while the authors of [START_REF] Barr | Collecting a Heap of Shapes[END_REF] analysed the memory heaps of real-world object-oriented programs to provide insights to improve memory allocation and program analysis techniques.

A high percentage of the proposed performance analysis techniques require some type of instrumentation. For example, the authors in [START_REF] Yang | Perracotta: mining temporal api rules from imperfect traces[END_REF] instrument the source code of the monitored applications to mine the sequences of call graphs under normal operation, information which is later used to infer any relevant error patterns. A similar case occurs with the works presented in [START_REF] Hangal | Tracking down software bugs using automatic anomaly detection[END_REF][START_REF] Csallner | Dsd-crasher: a hybrid analysis tool for bug finding[END_REF] which rely on instrumentation to dynamically infer invariants within the applications and detect programming errors; or the approach proposed by [START_REF] Chen | Path-based failure and evolution management[END_REF] which uses instrumentation to capture execution paths to determine the distributions of normal paths and look for any significant deviations in order to detect errors. In all these cases, the instrumentation would obscure the performance of an application during performance testing hence discouraging their usage. On the contrary, our proposed framework does not require to instrument the tested applications.

Furthermore, the authors of [START_REF] Jiang | Automated performance analysis of load tests[END_REF] present a non-intrusive approach which automatically analyzes the execution logs of a load test to identify performance problems. As this approach only relies on load testing results, it can not determine root causes. A similar approach is presented in [START_REF] Malik | Pinpointing the subsys responsible for the performance deviations in a load test[END_REF] which aims to offer information about the causes behind the issues. However it can only provide the subsystem responsible of the performance deviation. On the contrary, our approach allows the applicability of the idle-time analysis in the performance testing domain through automation, which allows the identification of the classes and methods responsible for the performance issues.

ADAPTIVE AUTOMATED FRAMEWORK

The objective of this work was to automate the manual processes involved in the usage of an expert system to improve a tester productivity by decreasing the effort and expertise needed to use an expert system.

Figure 1: Proposed framework -Contextual view

This scenario is depicted in Figure 1, where our proposed framework executes concurrently with a performance test, shields the tester from the complexities of properly configuring and using the expert system, so that she only needs to interact with the load testing tool.

The following sections describe our proposed framework, its supporting architecture, and our implemented prototype.

Adaptive Framework. Our proposed adaptive framework is depicted in Figure 2. As a self-adaptive system is normally composed of a managed system and an autonomic manager [START_REF] Salehie | Self-adaptive software: Landscape and research challenges[END_REF], our framework plays the role of the autonomic manager. Therefore, it controls the feedback loops which adapts the managed system according to a set of goals. Meanwhile, the expert system and the application nodes under test play the role of the managed systems. As defined by multiple authors [START_REF] Robertson | Introduction: the First International Workshop on Self-Adaptive Software[END_REF][START_REF] Weyns | Do external feedback loops improve the design of self-adaptive systems? a controlled experiment[END_REF], self-adaptation endows a system with the capability to adapt itself autonomously to internal changes and dynamics in the environment to achieve particular quality goals in the face of uncertainty. In order to achieve that goal, our framework follows the well-known MAPE-K adaptive model [START_REF] Kephart | The vision of autonomic computing[END_REF]. This model is composed of 5 components: A Monitoring component to obtain information from the managed systems; an Analysis component to evaluate if any adaptation is required; then a component to Plan the adaptation, and a component to Execute it. Finally a Knowledge component supports the others in their respective tasks.

The key element of our proposed framework is its policy base, which fulfills the role of the Knowledge component and defines the pool of available adaptive policies. Each expert system requires to have at least two policies: A data gathering policy (to control the collection of samples), and an upload policy (to control when the samples are processed by the expert system). Additionally, an expert system might have other policies available (i.e. to back up the obtained samples, or correlate the expert system outputs).

From a configuration perspective, the tester needs to provide the Information Base, which is composed of all the input parameters required by the chosen policies. For example, an upload policy might require a Time Threshold to process the samples periodically, or a data gathering policy could use a Sampling Interval to coordinate the collection of samples.

From a process perspective, the autonomic manager has a core process which coordinates the other MAPE-K components. The process is depicted in Figure 3. It is triggered when the performance test starts. As an initial step, it gets a new Control Test Id, value which will uniquely identify the test run and its collected data. This value is propagated to all the nodes. Next all application nodes start the following loop (in parallel) until the performance test finishes: A new set of data samples is collected following a data gathering policy. After the collection finishes, the process checks if any upload policy has been fulfilled. If it does, the data is sent to the expert system (labeling the data with the Control Test Id so that information from different nodes can be identified as part of the same test run). Similarly, updated results are retrieved from the expert system in order to be consolidated. Additional policies might also be executed depending on the configuration. For example, as certain data collections can be costly (i.e. the generation of a memory dump in Java can take minutes and require hundreds of megabytes of hard disk), a back-up policy could be used to enable further off-line analysis of the collected data. This core process continues iteratively until the performance test finishes. When that occurs, all applicable policies are evaluated one final time before the process ends. Furthermore, any exceptions are internally handled and then reported.

Architecture. The proposed framework is implemented with the multi-agent architecture presented in Figure 1. This is composed of three types of agents: The Control Agent is responsible of interacting with the load testing tool to know when the test starts and ends. It is also responsible of evaluating the adaptive policies and propagate the decisions to the other nodes. The second component is the Application Node Agent which is responsible to perform any required tasks in each application node. Similarly, the Expert System Agent is responsible of interfacing with the expert system. These components communicate through commands, following the Command 7 Design Pattern: The Control Agent invokes the commands, while the other agents implement the logic in charge of executing each concrete command. An example of these interactions is depicted in Figure 4: Once a tester has started a performance test (step 1), the Control Agent propagates this action to all the Application Node Agents (steps 2 to 4). Then each Application Node Agent performs its periodic tasks (steps 5 to 9) until any of the configured upload policies is fulfilled and the data is sent to the expert system (steps 10 and 11). These steps continue iteratively until the test ends. At that moment, the Control Agent propagates the stop action (steps 21,22 and 24). At any time, the tester might choose to review the intermediate results of the expert system (steps 12 to 14) until getting the final results (steps 25 to 27).

Prototype. Based on our proposed framework, a prototype has been developed in conjunction with our industrial partner IBM. The Control Agent was implemented as a plugin for the Rational Performance Tester (RPT) 8 , which is a common load testing tool in the industry; the Application Node Agent was implemented as a Java Web Application; WAIT was the selected expert system due to its analysis capabilities (discussed in Section 2). Finally, the Expert System Agent was implemented in PHP in order to extend the web interface of WAIT (which is developed in that technology).

Additionally, two policies were implemented: One data gathering policy which uses a constant Sample Interval (SI) during the complete test execution. Similarly, an upload policy which uses a constant Upload Time Threshold (UTT).

ASSESSMENTS OF WAIT TRADE-OFFS

To understand which adaptive policies would be more adequate for WAIT, an assessment of its performance trade-offs were done. It involved two tests. The first one evaluated the overhead introduced by the data gathering process of WAIT in the application nodes. The second one measured the resource utilisation in the WAIT server during the processing of those samples.

All tests were done in an environment composed of eight VMs: Five application nodes, one WAIT server, one load balancer and one load tester (using RPT 8.5). All VMs had the following characteristics: 2 virtual CPUs, 3GB of RAM, and 50GB of HD; running Linux Ubuntu 12.04L 64-bit, and Oracle Hotspot JVM version 7 with a heap of 2GB. The application nodes also ran an Apache Tomcat 6.0.35.

The DaCapo 9 benchmark 9.12 was chosen because it offers a wide range of different application behaviours to test. For each benchmark, its smallest Sample Size was used. In order to invoke the Dacapo benchmarks from within a RPT HTTP test script, a wrapper JSP was developed. It allowed the execution of any DaCapo benchmark via an input parameter. Finally, a 24-hour test duration was chosen to reflect more realistic test conditions.

The next sections describe the results of the performed tests.

Overhead in the Application Nodes. This test involved the assessment of the throughput (transactions per second) and the identified bugs during the data gathering process of WAIT. These metrics were collected through RPT and the WAIT report, respectively.

As the SI controls the frequency of samples collection from the monitored application (which is the main potential cause of overhead introduced by WAIT), a broad range of values was tested (7, 15, 30, 60, 120, 240, 360, 480, 720 and 960 seconds). Purposely, the smaller value in the range (7 seconds) was chosen to be smaller than the minimum recommended value for WAIT (30 seconds). Similarly, the largest value in the range (960 seconds) was chosen to be larger than 480 seconds (a SI commonly used in the industry). As the UTT has minimum impact in the performance of the application nodes (only controlling when the samples are sent to the WAIT server), a constant value of 30 minutes was used.

The obtained results showed that there is a relationship between the selection of the SI and the performance cost of using WAIT. This behaviour is depicted in Figure 5, which summarizes the results of the tested configurations. It can be noticed how the throughput decreases when the SI decreases. This performance impact is mainly caused by the Javacore generation process which pauses the JVM during its execution. Even though its cost was minimum when using higher SIs, its performance impact gradually became visible (especially when using SIs below 30 seconds, which is the minimum recommended SI for WAIT). On the contrary, the number of identified bug increases when the SI decreases. This positive impact is a direct consequence of feeding more samples to the WAIT server, situation which allows WAIT 9 http://dacapobench.org/ to do a more detailed analysis of the monitored application. A second round of analysis concentrated on the most critical issues identified by the WAIT report (those ranked with a frequency above 80% and which would be the most relevant for a tester) to assess if the previously described behaviours were also observed there. That analysis confirmed the presence of similar behaviours, as depicted in Figure 6. Overhead in WAIT Server. This test involved the assessment of the CPU (%) and memory (MB) utilization in the WAIT Server during the processing of the samples. These metrics were collected using the "top" command.

As both SI and UTT influence the number of samples that are sent to the WAIT server to be processed (as the lower the SI, the more samples to be processed. Similarly, the higher the UTT, the more samples to be processed), a range was chosen for each parameter. For the SI, the following 3 values were used: 30, 240 and 480 seconds. For the UTT, the following 3 values were used: 5, 30 and 60 minutes.

The results showed how the resource utilisation in the WAIT server is related to the number of samples processed in parallel; which is a function of both the SI and UTT. For example, the experimental configuration which used a SI of 240 seconds and an UTT of 30 minutes, reported similar resource utilisations than the configuration which used a SI of 480 seconds and an UTT of 60 minutes, as both combinations fed the same number of samples (per upload iteration) to the WAIT server.

Even though the CPU and memory utilisations showed simi-lar trends, the WAIT server proved to be more CPU-intensive (with its CP UAV G and CP UMAX exceeding the 90% utilisation in most of the tested configurations). On the contrary, the WAIT server was considerable less memory-intensive (with its maximum M EMAV G around 23MB, and the maximum M EMMAX around 28MB). These results are presented in Figures 7 (CPU) and 8 (memory). Conclusions. The performed tests proved that the selection of SI influences the performance impact that using WAIT might provoke in the monitored application. Similarly, the selection of UTT influences the resource utilisation in the WAIT server. These behaviours make the selection of both settings good candidates to become adaptive policies.

ADAPTIVE POLICIES

The following sections describe our proposed policies, which are based on the outcomes of the previously described assessment.

Accuracy-Target Data Gathering Policy. This adaptive policy was designed to balance the performance tradeoffs (between precision and performance) caused by the selection of the SI.

The detailed policy is depicted in Figure 9, where each step is mapped to the corresponding MAPE-K element. This policy requires two user inputs: The response time threshold, which is the maximum acceptable impact to the response time (express as a percentage); and the warm-up period, which is the time after which all the types of transactions have been executed at least one time (hence contributing at least once to the average response time of the test run). Two additional parameters are required. As their values are specific for each expert system, they would be retrieved from our policy base: The minimum SI that should be used for The process starts by waiting the configured warm-up period. Then it retrieves the average response time (RTAV G) from the load testing tool. This value becomes the response time baseline (RTBL). After that, the process initialises the SI of all the application nodes with the minimum SI. This strategy allows to collect as many samples as possible, un-less the performance degrades below the desired threshold. Next, an iteratively monitoring process starts (which lasts until the performance testing finishes): First, the process waits the current SI (as no performance impact caused by the expert system might occur until the data gathering occurs). Then, the new RTAV G is retrieved and compared against the RTBL to check if the threshold has been exceeded. If so, it means that the current SI is too small to keep the overhead below the configured threshold. In this case, the SI is increased by the value configured as ∆SI. Finally, the new SI is propagated to all the application nodes, which start using it since their next data gathering iteration.

Finally, it is worth mentioning that this policy was inspired on the scenario (commonly experienced in the industry) where a test team aims to minimise the number of required performance tests due to budget or schedule constraints. This might be achieved by allowing certain level of known overhead in the tested application in order to identify as many performance bugs as possible, in addition to the normal results obtained from a performance test.

Efficiency-Target Upload Policy. To minimise the resource utilisation caused by the selection of UTT, the following adaptive policy was designed. The detailed policy is depicted in Figure 10, where each step is mapped to the corresponding MAPE-K element. It requires two user inputs:

The initial UTT to be used; and the ∆UTT, which indicates how much the UTT should change when an adjustment is required. An additional parameter would be retrieved from our policy base (as its value is specific to each type of resource): The target of maximum utilisation (ToMU). As documented in [START_REF] Rolia | Automating enterprise application placement in resource utilities[END_REF], the objective of this target is to retains certain unused capacity to provide a soft assurance for quality of service. In the case of the CPU, this target is 90%.

The process starts by initialising the UTT of all the application nodes with the initial UTT. Then the process awaits until all the application nodes have uploaded their samples one time. This step is done to make sure that the UTT is only modified if required. After all the nodes have uploaded their results, the process retrieves the CP UAV G of the shared service during the processing of those samples, as well as the average duration of the CPU peak usage (CP U -DAV G). Then the CP UAV G is compared with the ToMU. If the ToMU has been exceeded, new UTTs are calculated.

As a first strategy, a different UTT is calculated for each node to prevent that all the nodes upload their results at the same time. To respect as much as possible the initial UTT, the calculation of the new UTTs is based on the current UTT (by iteratively subtracting or adding the CP UAV G -D from the current UTT until all nodes have a new UTT). For example, if we have 5 nodes, a current UTT of 30 minutes and a CP U -DAV G of 1 minute, the new UTTs of the nodes would be 28,29,30,31 and 32. Finally, the new UTTs are propagated to all the application nodes, which start using them since the next upload iteration.

In case a subsequent adjustment is required (meaning that only splitting the UTT was not enough to bring the CP UAV G below the ToMU), the current UTT is decreased (by the value configured as ∆UTT), before the calculation of the

EXPERIMENTAL EVALUATION

Two additional experiments were performed to evaluate the performance of our proposed adaptive framework. The first experiment evaluated the accuracy of the implemented adaptive policies, while the second experiment assessed the productivity gains that our framework brought to the performance testing process. The next sections describe these experiments, their results, and the main threats of validity.

Assessment of Adaptive Policies. The objective of this experiment was to evaluate if the adaptive policies fulfilled their purpose of addressing the identified trade-offs without the need of manual intervention from the tester's side. The experimental set-up was equal to the one used in the assessments of the WAIT trade-offs (Section 4) with except of the selection of SI and UTT, as the adaptive policies took the place of the manual configurations of these parameters.

The adaptive policies used the following configurations: For the accuracy policy, a 20% threshold was defined. This configuration was suggested by IBM SVT to reflect real-world conditions. Additionally, a warm-up period of 5 minutes was found to be enough for all the test transactions to be completed at least one. Finally, the minimum SI and the ∆SI were set to 30 seconds. Regarding the efficiency policy, the initial UTT was set to 30 minutes (as it is a time range commonly used in the industry to monitor performance tests); and the ∆UTT was set to 10 minutes. Finally, the CPU maximum utilisation threshold was set to 90%.

The obtained results were compared against the results from the previously performed assessment of the WAIT trade-offs (Section 4). The first part of our analysis focused on evaluating the accuracy policy. In terms of performance overhead, the results demonstrated that the accuracy policy worked, as it was possible to finish the test with the overhead caused by WAIT within the desired threshold. This was the result of increasing the SI once the threshold was exceeded (during the test execution) to reduce its performance impact. In our case, this adjustment involved that the SI increased two times, moving from its initial value of 30 seconds to a final value of 90 seconds. Regarding bug coverage, the number of bugs found with the adaptive policy was higher than the number of bugs found with the corresponding static SI of 90 seconds. This was the result of using other (smaller) SIs during the test, situation which provoked that the bug coverage was higher (compared to a SI of 90 seconds) during certain periods of the test. The same analysis was done considering only the critical bugs, and similar behaviours were observed. These results are presented in Figures 11 (for the overall bugs) and 12 (for the critical bugs).

The second part of our analysis concentrated on evaluating the efficiency policy. The obtained results showed that the efficiency policy achieved its goal of decreasing the utilisation in the shared service (the WAIT server in this scenario). In our case, the CP UAV G and CP UMAX of the first round of uploads (which occurred before any adjustment) were 90.7% and 95.0%, respectively. As these metrics exceeded the target of maximum utilisation (ToMU), the efficiency policy adjusted the UTTs of the nodes after the first round of uploads. After the UTT adjustment, the CP UAV G and CP UMAX decreased to 65.7% and 75.0% (re- maining below the ToMU during the rest of the test). This behaviour is depicted in Figure 13 which visually shows the CPU utilisation for one of the performed experiment using our efficiency policy. The first peak is the CPU utilisation before the UTT adjustment, while the second set of peaks is the CPU utilisation after the adjustment. In conclusion, the results of this experiment demonstrated that our proposed adaptive policies achieved their respective goals: The accuracy policy kept the performance trade-offs within the desired threshold. Meanwhile, the efficiency policy decreased the resource utilisation of the shared service, minimising the possibility of its saturation.

Assessment of benefits in testing productivity. Here the objective was to assess the benefits our framework brings to a performance tester in terms of reduced effort and time.

The experimental set-up was similar to the one used in the previous experiment with three exceptions: First, the iBatis JPetStore 4.0 10 application was selected in order to test our framework with a different application behaviour. Second, the number of application nodes was increased (to 10 nodes) to test our framework in a bigger test environment.This experiment also involved modifying the source code of JPet-Store to inject five performance issues.

Two types of runs were performed: The first type involved a tester trying to identify the injected bugs using WAIT manually (M-WAIT). A second type of run involved using WAIT through our automation framework (A-WAIT). In both cases, the tester did not know the number or characteristics of the injected bugs.

The results of these experiments are summarized in Table 1.

After comparing the results of both runs, two time savings were documented when using the automated WAIT: First, the effort required to identify bugs was decreased (68% less than the manual WAIT). This time saving was the result of simplifying the analysis of the WAIT reports: Instead of having multiple reports (one per node) that needed to be analysed and manually correlated, the tester using the automated WAIT only needed to keep monitoring a single report which incrementally evolved. The second saving involved the time required by the tester to identify all the injected bugs. By using the automation framework, it was possible to feed WAIT incrementally during the performance test execution (in contrast to manual WAIT, where the tester needed to wait until the end of the performance test). This behaviour allowed the tester using the automated WAIT to easily get intermediate results during the test run.

In our case, all the bugs were identified by the tester using the automated WAIT after the first hour of test execution. Therefore, she was able to start the analysis of those bugs in parallel to the rest of the performance test execution (which the tester kept monitoring). A direct consequence of this second time saving was that the overall duration of the performance testing activity decreased 27%. For the tester using the automated WAIT, the activity practically lasted only the planned 24-hour duration of the performance test, plus some additional time required to review the final consolidated WAIT report. It is also worth mentioning that both testers were able to identify all the injected bugs with the help of the WAIT reports. An additional observation from this experiment is that the time savings gained by the automated framework are directly related to the duration of the test and the number 10 http://sourceforge.net/projects/ibatisjpetstore/ of application nodes in the environment. This expected behaviour is especially valuable in long-term runs, which are common in performance testing and typically last several days. The same situation occurs with the performance testing of highly distributed environments, as the obtained time savings will be higher under those conditions.

To summarize the experimental results, they allowed to measure the productivity benefits that a tester can gain by using an expert system through our proposed automation framework. In particular, two time savings were documented: The effort required to identified bugs was reduced (68% in our case), as well as the total duration of the activity (27% in our case). A direct consequence of these time savings is the reduction in the dependence on human expert knowledge and a reduced effort required by a tester to identify performance issues, hence improving the productivity.

Threats to Validity. Like any empirical work, there are some threats to the validity of these experiments. First the possible environmental noise that could affect the test environments. To mitigate this, all tests were done in an unloaded environment and multiple runs were executed for each identified combination. Another threat was the selection of the tested applications. Their limited number implies that not all types of applications have been tested and wider experiments might be needed to get more general conclusions. However, there is no reason to believe that the presented framework cannot be applied to other usage scenarios. Additionally, a key motivation to select DaCapo was that it offers a wide range of application behaviours, hence helping to mitigate this risk.

CONCLUSIONS AND FUTURE WORK

The identification of performance problems in highly distributed environments is complex and time-consuming. Even though researchers have been developing expert systems to simplify this task, various limitations exist in those tools that prevent their effective usage in performance testing. To address these limitations, this work proposed a novel adaptive framework to automate the usage of an expert system in a distributed testing environment. A prototype was developed around the WAIT tool and then its benefits were assessed. The results showed the time savings gained by applying the proposed framework. In our case, the effort required to identified bugs was reduced 68%, while the total duration of the performance testing activity was reduced 27%. The results also demonstrated that the adaptibility capabilities of our framework fulfilled their purpose of simplifying the configuration of the expert system. This was done by addressing the identified WAIT trade-offs without the need of manual intervention from the tester's side. Thus, the approach was shown to simplify the usage of an expert system and to reduce the time required to analyze performance issues, thereby reducing the costs associated with performance testing. Future work will focus on investigating how best to extend the adaptive capabilities of our framework (i.e. adding data analytic policies to post-process the outputs of the expert systems).

Figure 2 :

 2 Figure 2: Adaptive automation framework

Figure 3 :

 3 Figure 3: Automation framework -Core Process

Figure 5 :Figure 6 :

 56 Figure 5: Bug Coverage vs. Throughput

  -UT[5]-3 SI[4]-UT[5]-6 SI[8]-UT[30]-18 SI[4]-UT[30]-37 SI[0.5]-UT[5]-50 SI[4]-UT[60]-75 SI[0.5]-UT[30]-300 SI[0.5]-UT[60]

Figure 7

 7 Figure 7: WAIT Server -CPU utilisations

Figure 8 :

 8 Figure 8: WAIT Server -Memory utilisations

Figure 9 :

 9 Figure 9: Accuracy-Target Data Gathering Policy

Figure 10 :

 10 Figure 10: Efficiency-Target Upload Policy

Figure 11 :Figure 12 :

 1112 Figure 11: Overall Bug Coverage vs. Throughput

Figure 13 :

 13 Figure 13: WAIT Server -CPU utilisation

Table 1 :

 1 M-WAIT and A-WAIT Comparison

				M-WAIT
	Metric	M-WAIT (hr)	A-WAIT (hr)	vs. A-WAIT
				(%)
	a.Duration of
	Performance testing	33.2	24.1	-27%
	activity (b+c)
	b.Duration of Performance Testing	24.0	24.0	0%
	c.Effort of Performance Analysis (d+e)	8.5	4.8	-44%
	d.Effort of Bug Identification	6.5	2.1	-68%
	e.Effort of Root Cause Analysis	2.0	2.0	0%

http://msdn.microsoft.com/en-us/library/aa480021.aspx

http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf

http://www.eclipse.org/mat/

http://msdn.microsoft.com/enus/library/aa983649(VS.71).aspx

http://wait.ibm.com

ACKNOWLEDGMENTS

This work was supported, in part, by Science Foundation Ireland grant 10/CE/I1855 to Lero -the Irish Software Engineering Research Centre (www.lero.ie).