Giuliano Casale
email: g.casale@imperial.ac.uk

Marco Gribaudo
email: gribaudo@elet.polimi.it

Giuseppe Serazzi
email: serazzi@elet.polimi.it

Tools for Performance Evaluation of Computer Systems: Historical Evolution and Perspectives

The development of software tools for performance evaluation and modeling has been an active research area since the early years of computer science. In this paper, we offer a short overview of historical evolution of the field with an emphasis on popular performance modeling techniques such as queuing networks and Petri nets. A review of recent works that provide new perspectives to software tools for performance modeling is presented, followed by a number of ideas on future research directions for the area.

Introduction

Since the early years of computing, software tools have been used to evaluate and improve system performance. This has been soon recognized as fundamental in a number of phases of a computer system's life-cycle, namely design, sizing, procurement, deployment, and tuning. However, due to the inherent complexity of the systems being evaluated and the novelty of the computing field, effective performance evaluation tools took several years to appear on the market. Simulation was the first technique used extensively for evaluating the performance of hardware logic of single components initially, and of entire systems later, see [START_REF] Nielsen | Computer simulation of computer system performance[END_REF] for a review. The introduction of simulation languages in the 60s, such as Simscript [START_REF] Markowitz | Simscript: a simulation programming language[END_REF] and GPSS [START_REF] Ibm | General purpose systems simulator iii user's manual[END_REF], was a milestone since several tools oriented to the simulation of computer systems and networks appeared shortly afterwards on the market. In the early 70s, two simulation packages oriented to computer performance analysis, namely Scert [START_REF] Herman | Scert: a computer evaluation tool[END_REF][START_REF] Herman | The use of a computer to evaluate computers[END_REF] and Case, were among the first to reach commercial success. It must be pointed out that, due to its dominant position in the computer market from the 60s to the 80s, almost all tools were developed for modeling systems and network technologies developed by IBM. Features of all generations of IBM systems, such as 360s and MVS, were deeply analyzed through simulation models and with other new analysis techniques that were becoming available. Other types of tools such as hardware monitors [START_REF] Carlson | A user's view of hardware performance monitors[END_REF], i.e., electronic devices connected to the system being measured with probes and capable of detecting significant events from which performance indexes can be deduced, were also used in the 70s. These did not reach a great diffusion due to their high costs, the difficulty of use, and the huge effort required to adapt them to different systems and configurations.

In those years, models started to emerge as a new way to evaluate single components and system architectures. Among the various problems approached were the evaluation of time-sharing supervisors, I/O configurations, swapping, paging, memory sizing, and networks of computers. The commercial interest in simulation modeling tools declined once efficient computational algorithms for analytical modeling appeared thanks to the pioneering work of Buzen [START_REF] Buzen | Computational algorithms for closed queueing networks with exponential servers[END_REF]. Analytical techniques became rapidly popular because of their relatively low cost, general applicability, and easy and flexibility of use with respect to simulation. Such techniques are still popular today and have been the subject of several books and surveys [START_REF] Kleinrock | Queueing Systems[END_REF][START_REF] Ferrari | Measurement and Tuning of Computer Systems[END_REF][START_REF] Haring | Performance Evaluation: Origins and Directions[END_REF][START_REF] Stewart | Introduction to the Numerical Solution of Markov Chains[END_REF]. BEST/1 [START_REF]Best/1 -design of a tool for computer system capacity planning[END_REF] was the first tool implementing analytical techniques being marketed commercially with great success. Rapidly, tens of tools for analytical modeling appeared on the market. Over the years, as soon as a new analytical technique has been discovered a new tool implementing it has been developed. Thus, we have now performance evaluation tools based on Queuing Networks, Petri Nets, Markov Chains, Fault Trees, Process Algebra, and many other approaches. Hybrid and hierarchical modeling techniques have been introduced in the 70s and 80s to analyze very large and complex systems. Starting from the 90s, due to the increase of the state spaces needed to represent models of modern systems, simulation has become again a fundamental tool for model evaluations. This has been also a consequence of the dramatic increase of computational power in the last two decades, which has made simulation a more effective computational tool than in the past. Several tools were designed specifically to solve particular class of problems. For example, SPE.ED [START_REF] Smith | Performance engineering evaluation of corba-based distributed systems with spe[END_REF] is a tool focused on the solution of the problems typical of Software Performance Engineering [START_REF] Smith | Performance Solutions: A Practical Guide to Creating Responsive, Scalable Software[END_REF]. More recently, in the security domain, the ADVISE method has been introduced to quantitatively evaluate the strength of a system's security [START_REF] Lemay | Adversarydriven state-based system security evaluation[END_REF].

In spite of this long historical evolution, there is a lack of surveys covering the history and current perspectives of the performance tool area. The aim of this paper is to fill this gap and provide an up-to-date review and critique of current software tools for performance modeling. We point to [START_REF] Casale | Tools for computer performance modeling and reliability analysis[END_REF] for a special issue on popular open source tools developed in academia in recent years. In this work, we first offer an overview of recent developments, many of which not covered in [START_REF] Casale | Tools for computer performance modeling and reliability analysis[END_REF], focusing in particular on Markov chains (Section 2), Queueing Networks (Section 3), Petri Nets (Section 4), Fault Trees (Section 5) and Process Algebras (Section 6). In Section 7 we instead discuss trends and new perspectives in software performance tools architectures. Finally, Section 8 gives final remarks and concludes the paper.

Markov Models

Due to limited space, we here give only a brief overview on tools for Markov modeling and we focus next on higher-level modeling languages such as queuing networks or Petri nets.

Markov chains have been extensively used since the beginnings of performance evaluation as the fundamental technique to analyze stochastic models. The power of Markov chains derives from the ease of conditioning probabilities, which depends only on the current active state of the chain. In addition to basic discrete-time Markov chains (DTMCs) and continuous-time Markov chains (CTMCs), the performance evaluation community has intensively investigated the use of absorbing processes, such as phasetype (PH-type) distributions, to represent the statistical properties of measurements and for transient analysis of performance models. Although PH-type distributions and Markov-modulated processes are very active research areas, we here focus only on DTMCs and CTMCs.

Due to their historical importance, many tools exist for the analysis of DTMCs and CTMCs which have been developed both by performance engineers and numerical experts. A comprehensive review of modern numerical techniques for the analysis of Markov chains can be found in [START_REF] Stewart | Introduction to the Numerical Solution of Markov Chains[END_REF]. Popular tools include MARCA3 , Mobius4 , SHARPE 5 , SMART 6 , and PRISM 7 . Such tools include exact and approximate Markov chain solvers, such as the Kronecker-based solution methods proposed in [START_REF] Buchholz | Complexity of memory-efficient kronecker operations with applications to the solution of markov models[END_REF]. Advanced techniques for state space generation and storage are also available such as multiway decision diagrams (MDDs), matrix diagrams, and symbolic state-space generation. MDD are an extension of the binary decision diagrams (BDD), a data structure capable of detecting redundancy and similarity in the state space of a model, allowing to reduce significantly the memory requirement to store the states. A discussion on such techniques can be found in [START_REF] Chung | A comparison of structural formalisms for modeling large markov models[END_REF].

Queuing Network Models

Queuing network models (QNMs) have been intensively used for the last three decades to study the effects of resource contention on scalability of computer and communication systems [START_REF] Baskett | Open, closed, and mixed networks of queues with different classes of customers[END_REF][START_REF] Lavenberg | A perspective on queueing models of computer performance[END_REF]. In their basic formulation, a QNM is composed by a set of resources visited by jobs belonging to a set of classes. Each job places a service demand, following some statistical distribution, at each visited resource, and the busy period of a resource depends on the contention placed by other jobs that simultaneously request service. The objective of the study is to compute performance metrics such as server utilizations or job response time distributions. Due to the lack of analytical solutions for general models, a number of approximation methods have been defined in the past, but there is still a lack for widely-applicable analytical approximation tools. In this context, simulation has become important in many practical applications to estimate performance metrics of QNMs, although analytical tools remain fundamental in several contexts, such as optimization studies which require the fast solution of hundreds of thousands models.

Queuing network modeling has a long history and has been addressed by several commercial packages such as BEST/1 [START_REF]Best/1 -design of a tool for computer system capacity planning[END_REF], RESQ [START_REF] Sauer | The research queueing (RESQ) package, version 2: Introduction and examples[END_REF], QNAP [START_REF] Potier | The markovian solver of QNAP2 and examples[END_REF], CSIM [START_REF] Schwetman | CSIM Reference Manual[END_REF], and a variety of academic tools such as Tangram-II8 , JINQS 9 , SHARPE 10 , Java Modelling Tools 11 , LQNS 12 , and several others 13 . A recent collection of research papers on some of the above academic tools can be found in [START_REF] Casale | Tools for computer performance modeling and reliability analysis[END_REF]. It is interesting to point out that, although the networking community has traditionally relied on queueing theory, popular tools such as NS-2 14 have been used quite rarely to simulate QNMs. Indeed, NS-2 and other networking tools are well suited for the description of network components and protocols, but this is usually a level of detail that is excessive for the abstractions used in QNMs. More recently, the OmNet++ framework has tried to invert this trend by publishing several tutorials for QNM analysis 15 . In spite of the large number of tools available, the techniques used for QNM simulation are quite similar: they all implement the classic discrete-event simulation paradigm, where a calendar of events, often based on a priority queue, is maintained in order to process chronologically arrival and departure of jobs from the resources. A variety of papers and books provide help to the developer of such tools to implement the most complex tasks, such as statistical analysis, transient filtering, rare event simulation, and implementation of preemptive disciplines such as processor sharing [START_REF] Fishman | Statistical analysis for queueing simulations[END_REF][START_REF] Heidelberger | A spectral method for confidence interval generation and run length control in simulations[END_REF][START_REF] Sauer | Computer Systems Performance Modeling[END_REF][START_REF] Pawlikowski | Steady-sate simulation of queueing processes: A survey of problems and solutions[END_REF].

More recently, new interesting techniques have been integrated in academic and commercial tools in order to analyse QNMs. We here try to survey for the first time these emerging ideas.

Ψ 2 is a tool 16 for steady-state analysis of QNMs that is based on perfect simulation theory. The fundamental ideas of this new simulation approach is to consider the Markov process underlying the queuing network and first identify a set of representative events such as job arrivals or end of service. A transition function Ψ (x, e) is then defined to represent the evolution of the current network state x as a function of each possible event e. The perfect simulation technique applies in its original form to the case where all events e are monotonous, i.e., such that for each pair of states (x, x ) for which a partial ordering x ≤ x exists it is Ψ (x, e) ≤ Ψ (x , e) for all events e. If such monotonicity condition is satisfied, a case which can be verified for large classes of queuing networks, Ψ 2 can simulate the model efficiency by an adaptation of the coupling-from-the-past (CFTP) algorithm. This algorithm involves an iteration that estimates steady state by randomization of the recent trajectories of the system prior to reaching the steady-state. The computational costs of the techniques grows linearly with the state space size, therefore significantly improving over the cubic or quadratic costs of a direct numerical solution of the infinitesimal generator.

Opedo17 is a recent tool for the optimization of performance and dependability models. This tool shows a rare case of a complex framework built upon open-source modeling tools such as OmNet++, Java Modelling Tools, APNN 18 , and the techniques developed in papers such as [START_REF] Buchholz | Complexity of memory-efficient kronecker operations with applications to the solution of markov models[END_REF]. The fundamental idea is to define a black-box interface to describe the output of existing modeling tools and develop a numerical framework for parameter optimization that is based only on black-box descriptions. Opedo uses a number of nonlinear search techniques to estimate a local optimum, such as pattern search and response surface methodologies, or a global optimum, such as evolutionary algorithms and Kriging methods. Integrated frameworks of this type appear promising especially in the context of software performance engineering where the first studies for large automatic software tuning based on performance models have recently appeared [START_REF] Martens | Automatically improve software architecture models for performance, reliability, and cost using evolutionary algorithms[END_REF]. Such frameworks automatically search for a set of design parameters that can ensure desired levels of responsiveness in an application.

Mathworks SimEvents 19 is a commercial extension of the MATLAB/Simulink simulator to support QNMs. Simulink has traditionally focused on simulation of continuoustime dynamical systems based on a number of ODE integrators, therefore the integration of SimEvents inside this framework allows to combine discrete simulation models with continuous-state simulation. Another interesting feature is that the tool description proceeds through the block diagram notations that are popular in control theory, therefore strongly emphasizing the input/output behavior of each component in the simulation. Another advantage of such tool over existing QNM simulators is that it can natively combine finite-state machines and flow charts which are useful for integration with hardware system and complex process models, respectively. Finally, another advantage is the robustness of the Simulink simulator, which is used in real-time critical industrial applications and therefore is affected by very few software bugs due to the high maturity level of the tool.

Another direction explored recently is the idea of considering fast queuing network approximations at the stochastic process level by means of linear programming. An advantage of these approaches over simulation is that linear programming can accurately describe hundreds of thousands or even millions of state probabilities. In the lp-rBm technique in [START_REF] Saure | A linear programming algorithm for computing the stationary distribution of semi-martingale reflecting brownian motion[END_REF], a queuing network can be described as a multidimensional reflected Brownian motion (rBm), which is extremely powerful to represent non-exponential distributions. Linear programming is used to approximate the equilibrium of the rBm which is not available in closed-form. The MAPQN Toolbox20 applies to closed models with general service time distributions. A number of necessary balance equations between the state of the queue is formulated, leaving equilibrium probabilities as unknown. This returns estimates that are provable bounds on the exact solution.

The wide availability of tools for QNMs suggests that much has been already done in support of the development and application of these models outside pure research. However, a number of additional extensions may be considered that are still lacking in the performance community. First, most tools seem to lack a software regression support in order to validate successive releases on a set of representative models. While these regressions are easy to define, it is harder to find in the literature detailed published solutions for reference models, especially for models with a mixtures of complex features (e.g., non-preemptive multiclass priorities, forking, finite capacity regions). This appears a limitation that the literature should address, since individual groups are cur-rently not sharing their best practices and useful case studies with the rest of the community.

Next, with the exception of few packages such as SMART, Java Modelling Tools, or Opedo, it appears that analytical results have been poorly integrated and exploited in current tools, possibly with the exception of the class of product-form models. While there exist indeed limitations to the accuracy of some approximations, it is a contradiction in terms that the largest body of work of the performance modeling community is at all effects marginalized from the software implementation and distribution. Larger research families, such as the linear algebra or parallel computing communities, have addressed these problems by creating public repositories to share standard implementations of important algorithms. Unfortunately, no similar experience has been attempted (at least to the best of the authors' knowledge) in the performance evaluation community. New recent attempts are trying to correct this issue 21 , however more cooperation is needed in our community to promote the success of such initiatives.

Petri Nets

Petri Nets (PN) are a graph based formalism, capable of visually describing system characterized by parallelism and synchronization. A Petri Net can be seen as a bipartite graph, where nodes are dived into two classes called places and transitions. For an historical review of Petri Nets, the reader can refer here 22 . Applications of Petri Nets to performance evaluation, mainly rely on their stochastic version (SPN -Stochastic Petri Nets and its generalization (GSPN -Generalized Stochastic Petri Nets). For a tutorial on GSPNs, the interested reader can refer to [START_REF] Kartson | Modelling with Generalized Stochastic Petri Nets[END_REF]. A large number of tools are available for GSPNs, e.g., GreatSPN 23 , SMART, PIPE2 24 .

Petri Nets are usually analyzed in steady state or in transient, either by discrete event simulation or by numerical techniques. In the latter case, the state space of the model is computed and its temporal evolution is mapped to a CTMC. Performance indexes are then obtained from the transient or steady state solution of the obtained CTMC.

Beside steady state and transient analysis, the bipartite graph structure of the model allows several analysis to be performed without explicitly generating the state space. Such analysis allows the determination of invariants, bounding properties, and ability to fire transitions. Petri nets are supported by several tools, each one having its own characteristics for what concerns the analysis techniques and for the capability of verifying different types of structural properties. A reference to the tools supporting PNs analysis can be found here 25 .

Throughout the years, several new types of PNs have been devised to simplify the study of computer systems. Each type of PN has its own benefits and it is supported by some specific tools. In the following we will briefly summarize some of the PN families that are currently used to address real-world modelling problems.

The GreatSPN tool supports Stochastic Well-formed Nets (SWNs) [START_REF] Chiola | Stochastic well-formed colored nets and symmetric modeling applications[END_REF], an important extension to Colored Petri Nets (CPNs) [START_REF] Jensen | Coloured Petri Nets: Basic Concepts, Analysis Methods, and Practical Use[END_REF]. CPNs improves the concept of marking of place by adding attributes to the tokens. Attributes are called colors, and belong to specific classes called types. Each token has associated a set of types that defines its attributes. When a transition fires, it removes some of the tokens from its input places, and collects their attributes into variables. At the same time, the firing of a transition inserts tokens into its output places. The attributes of the generated token are computed as functions of the variables collected from the input places. CPNs are important because they allow to use colors to model different types of objects and to model object-dependent behavior in a compact way. SWNs are CPNs where the functions that changes the color of the tokens have special forms. SWN have several interesting properties that allows some analysis to be performed on a reduced symbolic representation of the state space of the model. This allows to significantly reduce the size of the state space, thus increasing the size of the model that can be considered.

As observed in Section 2, the SMART tool has been one of the first tools to encode the state space of the CTMC underlying a GSPN using the MDD and to encode the transition matrix using the Matrix Diagram technique. When applied to PNs, the tool can exploit some of the structural properties of the networks to better organize the MDD levels, and to significantly reduce the time required to compute the state space of the model.

The TimeNET26 tool has been one of the first tools to support Non-markovian Stochastic Petri Nets (NMSPNs) [START_REF] Trivedi | Non-markovian petri nets[END_REF]. These type of PN allow transitions to fire following general non-exponential firing time distributions. In this case transitions are characterized by an extra parameter, the memory policy, used to define what happens when a transition, after being disabled, becomes enabled again. Three different policies are possible: prd (preemptive repeat different) when a new sample for the distribution is computed every time, prs (preemptive resume) when the transition continues its activity by firing after the remaining time, and pri (preemptive repeat identical) when, after each time a transition gets enabled, it restarts its activity but maintains the sampled firing time. Non-exponential transition can be solved by approximation as PH-type distributions, or by explicitly considering a memory variable in either the time domain or in the transformed domain.

The Oris27 and Romeo28 tools support Timed Petri Nets (TPNs). TPNs assigns intervals to timed transitions. Each transition fires after a time that belongs to the associated interval. Nothing is assumed about the distribution of the firing time of a transition, for this reason TPN allows non-determinism, and are particularly suited for Real-time applications. TPN tools transform a TPN model in a set of possible evolution region, each one described by a Difference Bounds Matrices (DBM). Performance indexes are then computed directly from the DBM set.

Fluid Stochastic Petri Nets, Continuous Petri Nets and Hybrid Petri Nets, add a new kind of place, the fluid place which contains a continuous marking. The three formalisms are very similar and differs only for small technical details. Even if fluid formalisms have been widely studied in the literature, very few tools actually consider them. One example is the FSPNedit tool [START_REF] Gribaudo | Fspnedit: a fluid stochastic petri net modeling and analysis tool[END_REF], which allows for both simulation and numerical analysis of FSPNs. Analytical solution of FSPNs is performed by computing transforming the model into a set of partial derivatives differential equations, and then by computing performance indexes from the solution of the PDEs. Simulation is performed using the time-scale transformation, since dependency on fluid values makes the system non-homogeneous.

Although MDD-based technique have significantly reduced the memory requirements for encoding the CTMC underlying a GSPN, allowing models with billions of states to be stored in few kilobytes and to be generated in fractions of seconds, the probability vector still have to be encoded directly. This actually limits the maximum number of states and thus the complexity of the models that can be addressed. Some research has already been done on techniques to encode the probability vector, but none has provided satisfactory results yet.

For what concerns the use of non-exponential transitions, the current approach tends to increase significantly the state space, limiting thus the number of non-Markovian activities that can be included in a model. Several techniques have been devised to describe the state space of a non-Markovian system using MDD. So a solution to the encoding of the probability vector should also help in allowing the use of an extended number of non-exponential transitions in NMSPNs models.

Fault Trees

Fault trees (FTs) is a formalism specifically devised for reliability analysis, and originally created at Bell Labs in the 60s. A fault tree contains a root node called the top event, and several leaves called basic event. Basic events are connected to the top event by arcs that traverse a series of intermediate nodes called gates. Gates usually correspond to boolean operations (the classical and, or and not), but might also contain extended primitives like the "m out of n". Basic events of a FT usually represent the occurrence of a faulty condition (such as the breaking of a component). The top event determines the state of the entire system, which might be compromised whenever one or more of its components fails. Due to their simplicity and their popularity, there exists many tools that can address the solution of FTs. A short list of available tools can be found here 29 .

In most cases, the user can assign a probability distribution to the basic event and the tool computes the probability distribution of the top event. If the basic events are independent, the exact distribution can be computed with simple algebraic operations. Difficulties arises when considering correlation among events, repair from faulty states, and cascade of events. In such case FTs are usually analyzed resorting to discrete event simulation, or by mapping them to other formalism such us GSPNs.

One of the most recognized tool is SHARPE, already introduced in Section2, which can perform several different analysis over given FTs. SHARPE also supports other similar formalisms like Reliability Block Diagrams, and Reliability graphs. The former characterize processes with a block diagram that explicitly shows the introduced redundancy. The latter describe systems with a graph where the failure rates are associated with edges. In this case, the required condition is that there exists a path from one node (called the source) to another node (called the sink). SHARPE solves the proposed models analytically by characterizing the FT with exponential polynomial distributions, and then by exploiting the analytical properties of such distributions.

The tool RADYBAN [START_REF] Montani | Radyban: A tool for reliability analysis of dynamic fault trees through conversion into dynamic bayesian networks[END_REF] exploits the analogies between fault trees and another probabilistic formalism: the Bayesian Networks (BAs). BAs are used to represent uncertain knowledge in probabilistic environments, and can be suited to perform reliability analysis. It is possible to prove that BAs can be more powerful than FTs, and that they can be suited to model more advanced features like noisy gates (that is gates that do not perform their and, or, not task deterministically).

Process Algebras

Process Algebras are a class of performance evaluation formalisms that describes models using a simple text-based representation. Even if the term was coined in the 80s, studied that lead to the definition of this formalism started in the early 70s. A nice historical introduction to Process Algebra can be found here 30 . In particular the modeling technique split a system into several interacting components. Each component can perform a set of actions, and then evolve to perform other activities. Usually the evolution of each component is represented by a very simple grammar such as:

S ::= α.S 1 | S 1 + S 2 | C S , (1)
where α.S is the prefix operator that tells that component S evolves to component S 1 after performing action α, S 1 + S 2 is the choice operator that tells that component S can evolve to either S 1 or S 2 , and C S is a constant used to address a sequence of components. Components can then be composed in models, using another very simple grammar such as:

P ::= P 1 L P 2 | P/L | S.
Operator P 1 L P 2 is the cooperation of P 1 and P 2 over the set of actions L. In order to perform one action in set L, the two components P 1 and P 2 have to synchronize, and the action is executed simultaneously. Operator P/L is called hiding, and simply prevents the resulting component to synchronize on actions belonging to the set L, by making such actions private (or internal).

In performance evaluation, particular dialects of Process Algebra that associate timing to events are used. Two common timed extensions of Process Algebra are PEPA (Performance Evaluation Process Algebra) and EMPA (Extended Markovian Process Algebra). For example, PEPA modifies the grammar presented in Equation 1 to S ::= (α, r).S 1 | S 1 + S 2 | C S , by adding a rate r to actions (that now are denoted as (α, r)). Each action is executed after an exponential distributed time with rate r.

Process algebras are supported in several tools such as ipc/Hydra31 , PEPA -Workbench 32 , Two towers 33 , and Mobius. Usually analysis is performed by enumerating the states that can be reached by the model (exploiting symmetries and creating symbolic states to reduce the size of the state space), and by creating a CTMC or a Generalized Semi-Markov Process (GSMP) to study the evolution of the model.

The PEPA-Workbench tool, beside offering the possibility to analyze a model using CTMC or discrete event simulation, it allows the use of new approximations based on fluid interpretation and differential equations [START_REF] Hillston | Fluid flow approximation of pepa models[END_REF].

Several application-domain specific derivation of Process Algebra have been produced. For example the tool BioPEPA-workbench 34 , supports an interesting extension of PEPA called BioPEPA, that defines a grammar that is suited for describing the processes that models the chemical reaction happening in biochemical system.

Architectures, Trends and Expectations

Several important trends are leading the current researches, such as the conjunction of qualitative (mostly model checking) and quantitative analysis, and the scalability and the parallelization of tools. Due to space constraint, instead of briefly considering several aspects, we focus on a single specific trend: tool inter-operability.

Following basic software engineering principles, the internal structure of modern performance tools is often organized around a clear separation of concerns. Separate software modules implement scientific algorithms, user interfaces, managers for performing repeated cycles of experiments, and primitives for generating, storing, and possibly simulate the models. Both in academia and industry, such modularization helps in separating and organizing the activity of scientific programmers (or students) involved in the development of the different parts of the code. On the other hand, this has been hardly combined with software reuse, since most performance groups opted to develop their own libraries instead of creating a public framework for sharing their work with the community.

We believe that such practices does not follow modern trends of software engineering, especially of the open source community that has promoted in recent years the sharing and reuse of software artifacts. In particular, major steps have been done towards software integration by means of standardized programming libraries (e.g., the Java Platform) and data exchange languages (e.g., XML). These technologies create interesting opportunities also to improve the way performance tools are defined.

A proposal for leveraging on these technologies that we describe in this section is to define a new family of performance meta-tools that could help the integration of the software artifacts available in the performance community. The general structure of a performance meta-tool, referred to as p-platform, is outlined in Fig. 1. Each layer of a pplatform describes a typical concern of a performance modeling study and we propose to organize the interaction between different submodules by means of layers, communicating through standardized meta-languages. In particular we suggest the use of XML as a possible implementation of a meta-language used to describe the interfaces between the layers. The ability of integrate different performance tools into a public framework would substantially improve the robustness and scale of current performance tools. It would also give the ability to users to select the components that best fit the goals of the performance study. The number of components required in the analysis is not fixed and depends on the objectives. Furthermore, the component of a layer can be skipped, for example one may evaluate a Markov model without using any high-level modeling language. According to the p-platform description, main steps of a performance study would be:

1. analysis of the intended use of the model based on the study's objectives; identification of the best technique to be used to describe the problem and its characteristics; 2. identification of the solution algorithm required to produce the type of results needed, e.g., equilibrium or transient values, exact or approximate solutions; 3. design of the experiment to be undertaken through the manager module, e.g, whatif, single run, optimization technique; 4. selection of the computing infrastructure to be used to run the numerical or simulation algorithms, single server, cluster, cloud, web, etc.

Although the principles outlined are simple, to the best of the authors' knowledge integration via XML has been poorly adopted by current performance modeling tools. The only notable exceptions are the Java Modelling Suite which coordinates data exchange between modules using XML files, the ongoing performance interchange format project PMIF [START_REF] Smith | Performance model interchange format (pmif 2.0): Xml definition and implementation[END_REF][START_REF] Smith | Performance model interchange format (pmif 2): A comprehensive approach to queueing network model interoperability[END_REF], and the Petri Net Markup Language PNML 35 which is supported by a growing number of PN tools. A possible explanation for this is that the majority of the tools have been developed started from the 80s, therefore according to the software engineering principles of the time. We believe that open release of the source code through open platforms such as Sourceforge would represent a first step in the right direction of helping external groups provide ideas, report bugs, and discuss in forums the issues we have outlined in this section.

Conclusions

In this paper, we have reviewed past and present efforts towards implementing software tools to support performance evaluation activities. Our analysis has revealed the area to be still very active, with a number of new simulation and analysis techniques still being proposed for classic models such as queuing networks and Petri nets. We have also argued that the recent advent of standardized data exchange languages such as XML opens new opportunities towards integrating existing community efforts into larger performance evaluation frameworks. To support this idea, we have outlined a performance meta-tool architecture, named p-platform, that provides high-level intuition on the basic blocks needed to define such frameworks.

http://www4.ncsu.edu/ billy/MARCA/marca.html

http://www.mobius.illinois.edu/

http://people.ee.duke.edu/ kst/

http://www.cs.ucr.edu/ ciardo/SMART/

http://www.prismmodelchecker.org/

http://www.land.ufrj.br/tools/tangram2/tangram2.html

http://www.doc.ic.ac.uk/ ajf/Research/manual.pdf

http://people.ee.duke.edu/ kst/

http://jmt.sourceforge.net

http://www.sce.carleton.ca/rads/lqns/

http://web2.uwindsor.ca/math/hlynka/qsoft.html

http://www.isi.edu/nsnam/ns/

http://www.omnetpp.org

http://psi.gforge.inria.fr/

http://www4.cs.uni-dortmund.de/Opedo/

http://www4.cs.uni-dortmund.de/APNN-TOOLBOX/

http://www.mathworks.com/products/simevents/

http://www.cs.wm.edu/MAPQN/

http://www.perflib.net

http://www.informatik.uni-hamburg.de/TGI/PetriNets/history/

http://www.di.unito.it/ greatspn/index.html

http://pipe2.sourceforge.net/

http://www.informatik.uni-hamburg.de/TGI/PetriNets/tools/quick.html

http://www.tu-ilmenau.de/fakia/TimeNET.timenet.0.html

http://www.stlab.dsi.unifi.it/oris/

http://romeo.rts-software.org/

http://www.enre.umd.edu/tools/ftap.htm

http://www.win.tue.nl/fm/0402history.pdf

http://www.doc.ic.ac.uk/ipc/

http://www.dcs.ed.ac.uk/pepa/tools/

http://www.sti.uniurb.it/bernardo/twotowers/

http://homepages.inf.ed.ac.uk/stg/software/biopepa/

http://www.pnml.org/

Acknowledgement

The authors wishes to thank the anonymous reviewer for the valuable comments that helped to improve the quality of the paper.


Carlos Juiz
email: cjuiz@uib.es2putxi@uib.cat

Ramon Puigjaner

From the origins of performance evaluation to new Green ICT performance engineering

Keywords: Performance evaluation of telephony, computer systems and communication networks, Performance modelling, Queuing theory, Queuing network theory, Simulation, Benchmarking, Green ICT

This paper intends to present an overview of the evolution of performance evaluation since its first steps the Erlang works for modelling telephone networks, based on simple queues until the present current challenges in Green ICT that will require the development of new paradigms and mathematical tools, and rapidly passing across the modelling works of Khintchine and Pollaczeck; Jackson; Baskett, Chandy, Muntz and Palacios; Buzen; Reiser and Lavenberg; and many others, and benchmarking standards that have produced solutions to the problems appearing in these hundred of years. Finally, we analyze some of the challenges of computer performance evaluation appearing today, mainly those related to the energy consumption and sustainability, globally known as Green ICT.

Introduction

It is well known that you cannot manage what you cannot measure and also that before installing a complex system it is better to have an estimation of what will be its performance behaviour. In the domain of information and telecommunication technologies (ICT), the first case of application of the rules of thumb we mentioned was based on the Erlang works on queuing theory to plan the capacity of the telephone system [START_REF] Erlang | The Theory of Probabilities and Telephone Conversations[END_REF], [START_REF] Erlang | Solution of some Problems in the Theory of Probabilities of Significance in Automatic Telephone Exchanges[END_REF], [START_REF] Erlang | Telephone Waiting Times[END_REF]. Since then a lot of techniques and tools have been developed for predicting the behaviour of ICT systems. These techniques have evolved to be able to tackle the new challenges that have appeared during the last hundred years.

This paper is organized as follows: in the next section the first works of Erlang applied to telephone systems are presented. Section 3 rapidly explores the improvements of the queuing theory experienced until the moment in which computer appears. Section 4 reviews the different models and benchmarks developed to predict the performance of computer systems. In section 5 models and benchmarks developed for communications systems are presented. Section 6 presents the evolution of operational drivers of success for systems and how Performance Engineering is evolving through these drivers. Section 7 presents the challenges of analyzing the performance of new types of ICT systems or more precisely the challenges of new concepts in performance evaluation. And finally section 8 concludes.

Erlang Works

Even if the works of Andrei Andreyevich Markov obtained his results on Markov chains (refined later by Norbert Wiener) before those of Agner Karup Erlang, we prefer to start our history with Erlang because his works are more related to ICT systems. Erlang was a Danish mathematician that worked for long time at the Copenhagen Telephone Company where he had the opportunity to analyze the phenomena related to the new invention that was the telephone. Among his works they are specially important the analysis allowing to represent the telephone communications like a Poisson distribution [START_REF] Erlang | The Theory of Probabilities and Telephone Conversations[END_REF] and the determination of how many circuits were needed to provide an acceptable telephone service (B and C Erlang formulae) and how many telephone operators were needed to handle a given volume of calls [START_REF] Erlang | Solution of some Problems in the Theory of Probabilities of Significance in Automatic Telephone Exchanges[END_REF], [START_REF] Erlang | Telephone Waiting Times[END_REF]. It is necessary to remember that at the beginning of the twentieth century human operators and cord boards were used to switch the telephone calls by means of jack plugs. The main limitation of the Erlang works is the assumption that both the service time and the interarrival time should be exponentially distributed. These results were intensively used for more than sixty years. However, Erlang was not a pure researcher but a hands-on one. To verify his assumptions he did not refuse to conduct measurements that obliged him to climb into street manholes. He can be considered as the father of the queuing theory and the performance evaluation of ICT systems. He was also an expert in the history and calculation of the numerical tables of mathematical functions, particularly logarithms. He devised new calculation methods for certain forms of tables.

Other Advances before the Computers

The next important result in queuing theory came from two mathematicians that simultaneously break one of the limitations of the Erlang assumptions: the service time could have any type of distribution. This result was obtained in 1930 by the Austrian-French Félix Pollaczek and the Russian Aleksandr Yakovlevich Khinchin working independently one of the other. In this case the result was not induced by a technical need but was of academic interest. Other interesting results, useful in some computer models, were those obtained by Alan Cobham who analyzed the behaviour of a queuing system when there are customers with different priorities. His initial work can be found in [START_REF] Cobham | Priority Assignment in Waiting Line Problems[END_REF].

The need of predicting the performance of computers obliged scientists and engineers to develop new techniques and tools adapted to the new challenges.

Modelling

Although the initial step for tackling the problem of computer system performance modelling was initially thought for modelling a computer system performance, the pioneering works of Jackson [START_REF] Jackson | Jobshop like Queueing Sysems[END_REF] and Gordon and Newell [START_REF] Gordon | Closed Queueing Systems with Exponential Servers[END_REF] with the algorithmic complement of Buzen [START_REF] Buzen | Computational Algorithms for Closed Queueing Networks with Exponential Servers[END_REF] prepared the arrival of the work of Baskett, Chandy, Muntz and Palacios [START_REF] Baskett | Open, Closed and Mixed Networks with Different Classes of Customers[END_REF] on product formnetworks. This work established a complete basis for modelling the performance of a computer system allowing an analytical process, even if the possibility of analytical processing was restricted by an important set of conditions. This work was completed by the proposal of convenient algorithms like those of Reiser and Kobayashi [START_REF] Reiser | Queueing networks with multiple closed chains: Theory and computational algorithms[END_REF] and Reiser and Lavenberg [START_REF] Reiser | Mean Value Analysis of Closed Multichain Queueing Networks[END_REF].

In parallel with these works, Buzen y Denning [START_REF] Denning | The Operational Analysis of Queueing Network Models[END_REF], [START_REF] Buzen | Operational Treatment of Queue Length Distributions and Mean Value Analysis[END_REF] developed a different approach for tackling the same problem and arrived to similar numerical results; the operational analysis. This approach has a more physical basis without the strong mathematical and statistical conditions of the queuing network approach.

In order to facilitate the use of these results to engineers and researchers, several tools were developed. Among them we can find BEST-1 developed by BGS, the company created by J.P. Buzen, RESQ developed by IBM and QNAP2 developed by INRIA.

In order to free the modelling process of the constraints of the product form networks, a number of approximated methods were developed, like those of Courtois [START_REF] Courtois | Decomposability: Queueing and Computer Systems Applications[END_REF], Marie [START_REF] Marie | Modélisation par Réseaux de Files d'Attente[END_REF], Gelenbe and Mitrani [START_REF] Gelenbe | Analysis and Synthesis of Computer Systems[END_REF] and many others. Some of these approximated algorithms were included in some of the previous tools. Some of the above mentioned tools also included simulation capabilities to analyze queuing network models not able to be processed by any other technique.

Benchmarking

However in many cases computer customers need to compare the computing capacity of different systems. Initially this capacity was evaluated simply by the execution time of a typical instruction, habitually the addition instruction. With the increasing complexity of computer systems (clever architectures, sophisticated operating systems, etc.) this simple comparison was no more valid. Computer customers proposed the execution of a sample of their real workload on the different systems they had to compare. Sophisticated techniques were developed to characterise the system workload by a reduced number of programmes extracted from this workload.

The next step was to standardise these sets of programmes depending on the type of workload we were interested in. The definition of these standards was reached by consensus like in the case of LINPACK [16], a collection of Fortran subroutines that analyze and solve linear equations and linear least-squares problems for the scientific computation or established by organizations participated by computer manufacturers and customers, like in the case of SPEC [17].

The first benchmarks proposed by SPEC were devoted to batch environments, then the target was a conversational system and currently cover most of the typical working environments of computer systems. An aspect particularly interesting is the possibility of scaling the benchmarks in order to adapt the size to the particular needs of the customer. Also SPEC organization publishes results of its benchmarks with different sizes running over a variety of systems.

Performance Evaluation of Networks

When computers were no more able to work isolated the need of connection initially with terminals and then with other computers appears. The complexity of the systems increased and consequently the complexity of the tools and techniques used to evaluate their performance.

Modelling

When the computers ran in an isolated way it was quite easy to build analytical models able to be processed with the help of appropriated tools. However, with the increasing of the networked systems complexity (based on wired or wireless LANs, or in ad-hoc networks, or in sensor networks, etc.), frequently it was very hard to find an enough accurate model for representing such systems and able to find the solution analytically. Fortunately, the increasing computer speed allowed building simulation models representing with enough accuracy the behaviour of the corresponding systems and having the possibility of obtaining results with small enough confidence intervals in computing times reasonably short.

Among the different simulation engines there two that are used very frequently: OPNET [18] and ns-2 [19].

Benchmarking

In terms of standard benchmarks, and restricting our comments to SPEC [17], the variety of benchmarks proposed by this institution has been increased to cover the most typical systems including a network in their structure. To cover these needs benchmarks for analyzing conversational systems, transactional systems, web systems (an e-commerce sales systems), etc. have been developed and proposed to the community of computer systems users.

The classical drivers of operational success at ICT have been always: Performance, Dependability, Quality, Flexibility and Cost [START_REF] Slack | Operations and Process Management: Principles and practice for strategic impact[END_REF]. Some of them seem to be clearly non-functional and other functional, but all five are closely related. In fact, the cost has the key of selecting particular solutions at datacenters for the other four operational drivers. Lately, other two drivers have been emerged as important as the first four: Security (some authors included it into Dependability) and Energy (classically included it in Cost). In Sections 2-5 we have overviewed the evolution, just considering Performance, then considering Performance and Quality or Performance and Dependability, and other combinations of operational drivers for Performance Engineering and Performance Evaluation of software, hardware, networks, etc. In last ten years, a lot of research effort has been put on Performance and Security issues, but the issue of this paper is about the new opportunities in the combination of Performance and Energy efficiency, which is part of Green ICT arena. That is, most of old studies in Performance Evaluation were done with a single driver in mind: Performance, but nowadays, there is a shift between this single objective to the consideration of multiple, particularly, Performance and Energy.

Turning to the drivers of success, it is clear that taking only one of them usually produce the detriment of some of the others. For example, maximizing the performance may not guarantee quality of service or even do more inflexible a particular solution, while maximizing security can reduce yields by overexposure to controls, and these situations always increase the cost. Consequently, the challenge of the new performance engineers is to find a balance among these factors. Similarly, one of the challenges is to find the Green ICT energy efficiency acceptable performance, i.e. the energy cost control while maintaining the performance necessary for operation. In the next section, we try to explain the factors that can influence the performance engineering and energy, and the opportunities offered by these factors for research.

Green ICT and Performance

The prevailing international scientific opinion on climate change is that human activities resulted in substantial global warming from the mid-20th century, and that continued growth in greenhouse gas concentrations [48] caused by human-induced emissions would generate high risks of dangerous climate change [START_REF]Chapter 9: Projections of Future Climate Change[END_REF]. Even this debate is controversial, what it is clear is that some human activities, as the use of ICT, are producing huge greenhouse gases.

With the overall improvement in the quality of connections and the spread of the wireless Internet, the web applications, easily updatable and often "in the cloud and free", are gaining ground rapidly to traditional software. But not only distributed applications of cloud computing are growing, traditional web applications based on client-server paradigm are also growing, despite the alleged gratuity of some popular web applications.

Data center power consumption continues to grow at an alarming pace [START_REF] Epa | Report to congress on server and data center energy efficiency[END_REF][START_REF] Mankoff | Some computer science issues in creating a sustainable world[END_REF]. There are five distinct industry segments accounting for most of a data center's energy-efficiency: (1) servers and storage systems, (2) power conditioning equipment, (3) cooling and humidification systems, (4) networking equipment, and (5) lighting/physical security. Therefore, sustainable data centers require the multidisciplinary collaboration of mechanical engineers, electrical engineers, computer scientists, and others. Particularly, the hardware infrastructure of the data center consists of hundreds (or thousands) of servers hosting revenue-generating services, interconnected with each other and the outside world via networking equipment, and relying on storage devices for persistent data. These hardware elements are managed by a datacenter-wide software stack that spans the platforms and virtualization layers [START_REF] Banerjee | Sustainable Data Centers: Enabled by Supply and Demand Side Management[END_REF].

On one hand, data centers that host web applications and user files are huge warehouses with a lot of servers, additional electronic equipment and complex cooling systems. While computers, mobile phones and telecommunications networks are today primarily responsible for energy consumption, energy spending caused by data centers that maintain the quality of internet services in operation is growing faster. But giving more capacity to data centers usually means more servers and consequently all five factors mentioned above consume more energy [START_REF] Katz | Tech Titans Building Boom[END_REF].

On the other hand, Performance engineering is the set of roles, skills, activities, practices, tools, and deliverables applied at every phase of the Systems Development Lifecycle, which ensures that a solution will be designed and implemented to meet performance requirements [START_REF] Menasce | Performance by Design: Computer Capacity Planning by Example[END_REF]. The increasing number of Internet users and systems connected to it, especially due to the popularization of applications and Web-based services, produces continuously capacity problems. In these growing web systems, if no action is taken, a loss in quality of service is produced and, ultimately, worsens the performance of any web application. Additionally, this growth of web systems, may be in a slightly irregular shape and unplanned in most cases, due to time-to-market pressures [START_REF] Menasce | Capacity Planning for Web Services: metrics, models, and methods[END_REF]. Variables that could be affected by these changes in system workload are those perceived by the user and that constitute the quality of service (Quality of Service, QoS, among which is singular importance, the response time) as those need to know the server's responsibility to provide quality of service desired (which shall include the use, overhead and the throughput) [START_REF] Molero | Evaluación y Modelado del Rendimiento de Sistemas Informáticos[END_REF]. The performance improvement of these services can be done at different layers, from server side to applications, from network protocol to routing, etc. [45], [START_REF] Killelea | Web Performance Tuning[END_REF]. When even the techniques of performance and modelling of web-based computer systems and distributed has not reached a sufficiently scientific mature, a new challenge directly related to this branch of research emerges: the relationship between performance and energy cost is i.e., the energy efficiency of systems; this section is aimed by these combination of two topics and the future challenges for performance engineers due to this combination.

State of the art

The solutions to achieve better energy efficiency in sharing distributed resources will address different topics. The first is at the computer architecture level, techniques have been proposed to reduce processor energy usage through multi-core processors [START_REF] Kumar | Single-ISA heterogeneous multi-core architectures: The potential for processor power reduction[END_REF], designing energy proportional hardware [START_REF] Barroso | The case for energy proportional computing[END_REF], and increasing the number of performance and sleep states [START_REF] Nedevschi | Reducing network energy consumption via sleeping and rate-adaptation[END_REF]. Benchmarking techniques to quantify the energy costs of different computer architectures have been proposed [48].

Even though, the current possibilities of on-the-fly adaptation of the underlying hardware behaviour will be studied: for instance even it is possible to decrease the voltage of the processors to slow them down and save energy, there will be the need to characterize the electrical consumption of the systems, as a function of the workload of the system. This implies first to have a means to measure or estimate the energy consumption volume (by actual power-consumption measurement equipments), and then to model the behaviour of the computations related to the obtained performances. However, the price to be paid is reducing the system components longevity. Actions have to be taken at different levels of decision to improve the energy-efficiency of the distributed web servers:

• At the networking level, the idea would be to decrease the overall usage of the network. For instance, moving large amount of data will operate on routes consuming less energy and the placement of the data (or the replication of the data) will be optimized to minimize the equipment usage; complementarily, approaches to minimizing the time to transfer the data, and to reducing the data to be transmitted (e.g., by using forecasting techniques) will be investigated.

The performance forecasting for several network layers [START_REF] Gilly | Analysis of burstiness monitoring and detection in an adaptive Web system[END_REF], [START_REF] Alcaraz | Handling HTTP flows over a DiffServ framework[END_REF] should provide some skills performance and power tools development in order to know how applications are using the network.

• At the server level, in order to maximize energy efficiency, whether in a single system or over an ensemble of systems, users and data center operators need to understand the relationship between resource usage and system-level power consumption. This understanding enables such optimizations as consolidating workloads on as few machines as possible and turning others off [START_REF] Chen | Energy-aware server provisioning and load dispatching for connection-intensive internet services[END_REF], [START_REF] Heath | Energy conservation in heterogeneous server clusters[END_REF], [START_REF] Pinheiro | Load balancing and unbalancing for power and performance in cluster-based systems[END_REF], since current hardware is highly inefficient at low utilization [START_REF] Barroso | The case for energy proportional computing[END_REF]. While several types of full-system power models have been proposed, often in the context of enabling a particular optimization, they have not been systematically compared over a variety of software workloads and hardware platforms. Performance engineering will contribute to these efforts building several power models for servers executing web different workloads in extensive studies including regression models and also simulation. Other techniques include multiserver power coordination [START_REF] Raghavendra | No power struggles: A unified multi-level power management architecture for the data center[END_REF], hard disk spin-down [START_REF] Narayanan | Write off-loading: Practical power management for enterprise storage[END_REF], synchronization aware multithreading [START_REF] Park | Managing energy-performance tradeoffs for multithreaded applications on multiprocessor architectures[END_REF], and compiler driven optimizations [START_REF] Xie | Compile-time dynamic voltage scaling settings: opportunities and limits[END_REF]. Another interesting issue is server virtualization, where hardware elements are managed by a datacenter-wide software stack that spans the platforms and virtualization layers. However, adding a new software layer also adds additional overhead and risk of over utilization of hardware with clear consequences in energy but also savings. Performance engineers will also study how performance and energy varies with virtualization of servers in comparison with the non-virtual ones.

• At the application level, context awareness of the applications should be favored: it aims at developing applications that are aware of the power state of the computers and the currently set power policy, such that they are tuned appropriately, and respond to changes dynamically [START_REF] Kusic | Power and performance management of virtualized computing environments via lookahead control[END_REF]. Developing energy-aware data mining techniques will support energy efficiency at the applications level. For instance, clustering/classifying of data sources, identifying usage patterns and cluster users will help to decrease the energy consumed in the network. This last level is very attractive to performance research for several reasons. First, this layer has the most information on the actual user impact of performance and energy tradeoffs, enabling more aggressive performance sacrifice in unimportant areas compared to lower layer techniques. For instance, an application may throttle processor frequency to the minimum value required to satisfy user perceptible delay behavior. Second, application specific optimizations can be made at this layer such as changing the algorithm used, accuracy of computation (e.g. changing from double precision to single), or quality of service provided [START_REF] Im | Energy optimization for latency-and quality-constrained video applications[END_REF]. Third, energy usage at the application layer may be made dynamic [START_REF] Flinn | Energy-aware adaptation for mobile applications[END_REF]. For instance, an application hosted in a data center may decide to turn off certain low utility features if the energy budget is being exceeded, and an application on a mobile device may reduce its display quality [START_REF] Iyer | Energy-adaptive display system designs for future mobile environments[END_REF] when battery is low. This is different from system layer techniques that may have to throttle the throughput resulting in users being denied service.

Some green ICT research challenges for performance engineering

Many of the current green performance activities represent isolated optimizations; others attempt to standardize approaches and solutions and to promote a common understanding of the issues and complexities. While recognizing their vital importance, the literature is replete with such solutions and experiences. The new challenges will be how to reduce power consumption in a collaborative manner not only as individual efforts on devices but holding an optimal quality of service. These new performance challenges will propose realistic energy-efficient alternate models and also study new solutions for benchmarks. While much effort is nowadays put into hardware specific solutions to lower energy consumptions, the need for a complementary approach is necessary from the application viewpoint. Performance Engineering will characterize the energy consumption and energy efficiencies of applications at servers. Then based on the current hardware adaptation possibilities and interconnection networks, performance engineers will characterize the trade-off between energy savings and functional and non-functional parameters, particularly the web system performance. Most of the current effort consists of building performance and power models for servers from standard benchmarking, subsequently deriving new simulation engines to derive performance and energy results and finally propose a new benchmark models for servers taking into account energy issues.

For this reason, we propose an incomplete list of Green ICT performance topics putting together computing technologies, networking, optimization, and their performance evaluation and power issues. Our proposed research opportunities list focuses on several integrated energy and ICT related issues, from simulation to software applications, from networking to data centres, from performance engineering to consumption models, etc. trying to transit from research on only ICT functional issues to green ICT considerations, together.

The current open list of research topics can be started with the first volume of results in COST IC0804 [58], where there are included the results of this European action. However, our position is that there are research issues very appealing for the performance and power research advance. Here there is our open and incomplete list:

Energy consumption/performance models: models and tools where system administrator concerns about energy-efficiency of ICT usage are considered together with performance. Energy-Efficient algorithms: Dynamic Speed scaling and power-down client integration but holding resilience of systems. Dynamic performance management: programming models for the cloud rely on the use of tasks scheduled across large data centres, taking advantage of the parallelism/performance and power. Etc. For sure that there are many more, in fact, all performance engineering issues are directly or indirectly related to power consumption, so that we may select almost every performance issue to be studied from the energy-efficiency viewpoint. Thus, we may review the performance engineering research from the very beginning but now considering power consumption as a primary factor for the new Systems Development Lifecycle.

Performance evaluation concept has evolved along with he changes experienced by the changes in the technology: hardware, software and networks, and in the goals of the end users with respect to the systems they were using.

We have overviewed the evolution, just considering Performance evaluation, then considering other engineering factors as Performance and Quality or Performance and Dependability, and other combinations of operational drivers for Performance Engineering of software, hardware, networks, etc. In last ten years, a lot of research effort has been put on Performance and these new issues, but the topic of this paper is about the new opportunities in the combination of Performance and Energy efficiency, which is part of Green ICT arena. That is, most of old studies in Performance Evaluation were done with a single driver in mind: Performance but nowadays, there is a shift between this single objective to the consideration of multiple, particularly, Performance and Energy. We have seen how the tools and techniques used to evaluate or predict the computer performance have evolved and the new challenges derived from Green ICT will oblige to find new tools and techniques addressed to overcome the current challenges as they are suggested in the last part of this paper.

  New simulation and benchmarking and monitoring engines considering devices, networks and their corresponding energy consumption Energy Efficient Computing Centres: this will be addressed at different levels, i.e. at a data centre level, workload and server management, respectively. Energy Efficient Networks: at multiple levels, including networking devices, communication and control protocols, network architectures design, and network performance. Green Software Design: considering the compiler level on algorithmic and data structures optimizations for grid systems. Also, at application level to investigate what are the key aspects in the applications that specifies the performance requirements/power consumption that user expects from those services Design for Energy Awareness: empowering social in their use of products and services by increasing visibility of energy choices during operation.

	ICT Management Green Metrics: indicators of performance/energy from data
	centre monitoring to ICT governance/management
	Power-aware middleware for data centres: more intelligent power management
	for virtualised machines, power-aware virtualisation and benchmarking.
	Power-efficient routing: design of packet classifiers for high speed and low
	power consumption.
	Energy-efficient bandwidth allocation: making minimum consumption of energy
	resources in network equipment.
	Energy and distributed computing: placement of mobile agents, workload
	placement, virtual machine migration, P2P resource allocation, etc, providing
	means to develop energy-aware allocation strategies.
	Eco Data Base Management Systems: managing the energy consumed during
	query processing and considering query aggregation in workload.

Acknowledgements

This work has been partially funded by the Spanish Ministry of Science and Innovation and the FEDER funds, both in the framework of the projects TIN2007-60440 and TIN2009-11711.