We present next some fundamental aspects of cloud computing in order to establish a common, basic vocabulary.

I

Cloud computing has become a prominent feature of the modern computing landscape. Industry analysts predict that almost % of total IT expenditure will shi to the cloud by , while the global cloud market will reach billion in annual revenues in , attaining a compound annual growth rate of . % between and [ , ]. Cloud computing is a model in which providers own and manage con gurable computing resources (e.g., networks, servers, storage, applications, services), and customers access them on demand, typically on a pay-per-use basis. e model brings bene ts to both providers and customers [ ].

Providers pool and share computing resources among multiple customers, which leads to increased resource utilisation and reduced costs of resource provision. Customers obtain and release resources on demand while paying only for actual resource use, which leads to reduced costs of using resources. To optimise the value that providers and customers draw from the cloud, these actors must continuously manage cloud resources and applications in response to changes in their operating environments. is document presents my research activities conducted between and within the MYRIADS project-team (IRISA and Inria Rennes -Bretagne Atlantique) focused on cloud management. e overall goal of these activities was to develop methods and tools to assist cloud actors in managing cloud resources and applications in line with their objectives.

is chapter is structured as follows. First, Section . presents an overview of cloud computing concepts and related technology and business aspects. Section . then discusses the main research challenges that guided my research. Finally, Section . summarizes my research activities and outlines the remainder of this document.

. . C is document adopts the de nition of cloud computing provided by NIST [ ]:

Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of con gurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management e ort or service provider interaction.

According to NIST, the essential characteristics of the cloud computing model are as follows:

• On-demand self-service-cloud customers can provision computing capabilities, such as server time and network storage, automatically, without requiring human interaction.

• Broad network access-capabilities are available over the network and accessed through heterogeneous client platforms, such as mobile devices and workstations.

• Resource pooling-provider resources are pooled to serve multiple customers.

• Rapid elasticity-capabilities can be rapidly provisioned and released in response to demand.

• Measured service-resource usage is continuously monitored, controlled and reported.

e services provided by cloud providers are classi ed as follows:

• So ware-as-a-Service (SaaS)-customers access applications running on the provider's infrastructure; customers have no control over these applications or the infrastructure.

• Platform-as-a-Service (PaaS)-customers deploy on the provider's infrastructure applications created or acquired by customers; customers have control over the deployed applications, but not over the underlying infrastructure.

• Infrastructure-as-a-Service (IaaS)-customers provision fundamental computing resources (e.g., processing, storage, and network resources) in order to deploy and run arbitrary so ware, including applications and operating systems.

We use the general term cloud system (or cloud) to describe a system that o ers cloud services. Note that cloud systems may o er services to one another. For example, a SaaS system may use services provided by a PaaS system that in turn uses services provided by an IaaS system.

A cloud provider (or provider) is a person or organisation that owns and operates a cloud to provide cloud services. A cloud customer (or customer) is a person or organisation that uses a cloud. For example, a cloud customer may deploy cloud applications on a PaaS or IaaS system (but not a SaaS).

We distinguish the following models of deploying and using clouds:

• Private cloud-a cloud available for exclusive use by a single organisation.

• Community cloud-a cloud available for use by a speci c community of consumers.

• Public cloud-a cloud open to use to the public.

• Hybrid-cloud-a combination of multiple, independent clouds enabling data and application portability. Hybrid clouds may be used to support cloud bursting; that is, allowing applications that run in a private cloud to burst out into public clouds, when needed. e related concept of multi-cloud (or, cross-cloud) refers to a customer using multiple clouds at the same time.

.

. T B A

e main enabling technology of cloud computing is virtualisation, which enables the creation of virtual resources by multiplexing physical resources. A common form of virtualisation creates virtual machines (VMs) that emulate physical machines and execute operating systems. VMs are composed of virtual resources (e.g., memory, CPU, storage). VMs and virtual resources are typically the computing resources delivered by IaaS clouds.

Virtualisation brings several technical bene ts. First, it facilitates sharing physical infrastructures among multiple users while isolating users from each other. Second, it introduces exibility in resource management. For example, it enables modifying the resource allocations of VMs in a ne-grained manner (e.g., modifying the memory of a VM) as well as migrating VMs between physical machines. ird, it enables encapsulating applications in isolated packages with customised so ware stacks.

A core feature of cloud computing is the delivery of computing resources as a service.

Customers select cloud services based on the provided qualities of service (QoS) and prices.

QoS typically covers non-functional service characteristics, such as performance, availability, and reliability. We use the term Service Level Objective (SLO) to express requirements on QoS (e.g., the monthly uptime percentage for a service should be greater than . %).

Naturally, customers prefer that cloud systems provide strong guarantees on provided QoS.

Such guarantees may be included in Service-level Agreements (SLAs), which are contracts that formalize the terms of the provider-customer interaction. SLAs include the SLOs that the provider promises to meet and the price that the customer agrees to pay. e SLAs may also specify penalties paid by the provider when SLOs are violated.

ere are various ways in which cloud services are priced. e most popular pricing model is the pay-per-use (or pay-as-you-go) model, in which the charge paid by the customer depends on service usage (e.g., paying per second of use of a VM instance). Another model is subscription pricing, in which the charge is xed in advance for a given subscription period.

Static pricing refers to models in which prices are xed or change at a slow rate. Dynamic pricing refers to models in which prices uctuate, for example, based on supply and demand.

An example is Amazon EC Spot Instances pricing, in which customers bid a maximum price and run VM instances as long as the uctuating spot price is lower than their bid and capacity is available [ ].

.

R C

Next I outline the main challenges that have guided my research activities. e challenges are divided into two parts, focusing on automated management on behalf of providers and of customers.

. . P Cloud providers require automated resource management systems that continuously control the allocation of virtual or physical resources to customer requests. e main challenges in designing such systems are as follows:

S Resource management must deal with the changing environment in which the cloud system operates while e ectively pursuing provider objectives.

is environment includes uctuating workloads imposed by customers, varying levels of performance provided by underlying infrastructures (e.g. shared resources, networks), and varying QoS and prices provided by external cloud services. Providers have multiple objectives that must be considered simultaneously (e.g., maintaining SLAs and minimising costs)

and are subject to various constraints (e.g., resource capacities). e main challenge is thus ensuring that the objectives are satis ed despite the environment uidity.

Depending on the type of cloud, provider objectives may take various forms. For public cloud providers, the main objective is optimising pro t, which requires some combination of increasing revenues and reducing the costs of using underlying resources. Another important objective is maintaining the SLAs with customers. SLA violations typically reduce short-term pro t because of incurred penalties, but they also reduce the long-term attractiveness of the cloud service. For private cloud providers, a typical objective is achieving a resource allocation with desired properties, such as economic e ciency. Economic e ciency here means that the resources are allocated to the customers that value them the most (more precisely, the total value delivered to customers is maximised).

S SLO To increase the value of their service, providers should respond to the diversity and evolution of customer needs. is is a particular concern for PaaS providers. Indeed, these providers aim to deliver a complete development and hosting environment for applications while the application types popular with their customers are constantly evolving (e.g., web, data analytics, machine learning applications). An important challenge in designing PaaS management systems is thus supporting extensibility with respect to application types.

Di erent application types are associated with di erent QoS properties and methods of ensuring that these properties are satis ed. Along with application types, the management system should thus also be able to support associated QoS properties and methods. For example, the system should be able to support HPC applications along with SLOs related to completion time, web applications along with SLOs related to response time, or streaming applications along with SLOs related to latency and throughput.

S

Providers are operating a shared infrastructure that serves multiple customers. Depending on the cloud system, the size of this infrastructure may vary from a few machines to the multiple, globally distributed data centres of enterprise cloud providers. erefore, an important concern for the management system is its scalability; that is, its ability to handle growing numbers of resources and customers. Scalability can be increased, for example, by applying e cient decision-making algorithms or by applying decentralised management structures that involve multiple decision-making units, each having only partial information on the system.

. . C

Cloud customers require automated application management systems that continuously control the deployment of their applications on cloud services. We focus, speci cally, on PaaS and IaaS customers since SaaS customers have no direct control over applications. e main challenges in designing such management systems are as follows: S Modern distributed applications have complex structures imposing multiple constraints on the required underlying infrastructure (e.g., resource requirements of individual components, requirements for sharing resources between components). ese constraints are subject to change as the workloads and requirements of end users change. At the same time, an immense range of options is available for hosting an ap- ing SLAs that include QoS expectations (e.g., response time constraints), prices, and penalties to be paid when these expectations are violated. At the same time, being commercial entities, the providers have the objective to increase pro tability. To assist such providers with delivering their services, we proposed management solutions that take into account SLAs and infrastructure costs and take actions to increase provider pro t while coping with changing workloads.

Speci cally, the chapter presents two solutions. We discuss each solution in turn in the following.

. to integrate them into a real platform, and to perform a detailed evaluation of their impact on the provider pro t.

is work has led to two main contributions [ , , , , , ]. First, we proposed an approach for automatically managing cloud resources to increase the pro t of SaaS providers.

e approach enables processing service workloads reliably and cost-e ectively, taking into account operational costs and multiple, simultaneous SLAs with di erent customers, with each SLA de ning expected QoS, prices, and penalties. Second, the work proposed a prototype, called Qu DS, that applies this approach to support developing and managing SaaS applications.

In the following, I discuss the system model underpinning the approach and the mechanisms provided by the approach. I then provide an overview of the prototype and the evaluation before concluding with a brief discussion.

. . S S S M

e approach considers a SaaS system that integrates an application and makes it accessible to customers. e application is based on the master/worker pattern with an adjustable number of workers. Each customer has its own SLA (contract) with the provider, specifying a price, the expected levels of QoS for executing customer requests, and the penalties paid by the provider when these levels are violated. Each SLA de nes a price that depends on the SLA type and the required duration of the contract. e prices are de ned in a cost-based manner; that is, the more resources are required, the higher the prices are. e SLA also de nes penalties that are paid by the provider to customers when SLOs are violated. ere are two kinds of penalties: contract rescission penalties for rescinding ongoing contracts, and request cancellation penalties for failing to process individual requests.

R M

Resource management seeks to control the number of ongoing contracts and requests in order to increase the provider pro t. e proposed approach separates the assignment of resources to each customer (termed resource booking) and the assignment of resources to each customer request (termed resource allocation). To assist in these tasks, the approach employs two mechanisms, namely, under-provisioning (that is, booking less resources than required) and contract rescission (that is, canceling ongoing contracts).

Resource booking is triggered by new contract proposals. It determines the amount of resources to be booked using an under-provisioning policy (e.g., booking % of required resources). In case of resource shortage (e.g., in private cloud settings), resource booking decides whether to rescind an ongoing contract and accept the proposal or reject the proposal. e decision takes into account the estimated pro t of accepting the proposal and the penalty for contract rescission, aiming to increase the provider pro t. .

. Q DS P

To show that it is feasible to build a management system for SaaS providers following the approach and to enable the evaluation of the approach, we built a research prototype, the show the generated pro t for two customer pro les (heterogeneous and safe) and for three penalty policies (half, full, and double reimbursement). e heterogeneous pro le contains an equal number of contracts of each SLA type, while the safe pro le contains only contracts with strong reliability requirements (i.e., of safe SLA type). e half reimbursement policy means that the penalty that providers pay is half the contract price.

We observe that the higher the fault rate, the lower the pro t is since high fault rates increase the chances of incurring SLA violations and penalties. Similarly, as expected, the higher the penalty, the lower the pro t is. Moreover, the pro t is lower for the heterogeneous customer pro le since this contains contracts that are more sensitive to faults. In all cases, using work replacement (FT is on) increases the provider pro t. which applies the approach and provides support for batch and MapReduce applications.

In the following, I present the system model underpinning the approach and then describe the main policies suggested by the approach. Next, I outline the application of the approach to a speci c type of applications-namely, computational applications-the Meryn prototype, and the evaluation, concluding with a brief discussion.

. e approach is generic and independent from the types of supported applications and it relies on one background assumption. Namely, for each supported application type, there is a corresponding QoS translator plug-in able to translate the resource con gurations and, potentially, workload conditions, to provided QoS levels and vice versa. is enables predicting the QoS impact of modifying resources and estimating the required resources for achieving given QoS levels. Such translation capabilities can be based on application-type speci c knowledge, performance measurements on the provider platform, workload models, and performance models. Although improving prediction accuracy can improve the provider pro t, the approach remains independent of the achieved accuracy level.

.

. R S P

We developed three policies for sharing the private resources between the di erent groups and allocating resources to each newly arrived application.

B P e basic policy is naive. It statically partitions the private resources between the groups.

When a new application arrives for a group and the group has insu cient private resources, the application will be hosted on public resources.

A P e advanced policy enables the transfer of resources from one group to another one. When a new job arrives in a peak period, its corresponding group rst checks if it can get available private resources from other groups before renting additional resources from a public cloud.

O P e optimisation policy aims to nd the most cost-e ective solution for executing a newly arrived application. Speci cally, when the available private resources in all groups are not su cient for hosting the new application, the policy compares the cost of three options for getting the missing resources: ( ) renting them from a public cloud, ( ) obtaining them from running applications, or ( ) waiting for private resources to become available. e rst option is similar to the one used in the advanced policy. e second option, called donating option, may have a QoS impact on the a ected applications. Note that this option is applicable only to applications that can use variable amounts of resources during their lifetime. e third option, called waiting option, may have a QoS impact on the new application.

e policy is implemented in a decentralised, auction-based manner. In essence, the group corresponding to the new application receives bids from the other groups, representing the estimated penalties associated with obtaining resources from these groups using both the donating and waiting options. e corresponding group then selects the option with the lowest bid and compares this with the option of renting public cloud resources. e pseudocode of the optimisation policy is shown in Algorithm .

. . C A

We applied the approach to computational applications, de ned as applications running for a nite duration without any human intervention (e.g., batch applications). Computational applications may be rigid, that is, applications that require a xed amount of resources, or elastic, that is, applications that can use variable amounts of resources at runtime. e work assumed that the QoS translator plug-in for computational applications relies on a simple, linear speedup performance model, where the application runtime is inversely proportional to the used number of resources.

e following subsections describe how SLAs are de ned and how the waiting and donating bids are calculated for computational applications.

C SLA e SLA for computational applications speci es a deadline and a price. e deadline is a limit for the completion time of the application. e PaaS system calculates the deadline and price using the QoS translator plug-in. Speci cally, the deadline is calculated as the application's predicted runtime and the price as the cost of the resources for this time period.

e system proposes several SLA classes (i.e., high, medium, low) to customers by combining di erent values for deadline and price. For example, the high SLA class has the shortest deadline, corresponding to the minimal possible application's runtime, and the highest price;

inversely, the low SLA class has the longest deadline and lowest price. Having multiple SLA classes increases the number of choices given to customers (e.g., best price or best QoS). If the customer does not agree with any proposed SLA class, the customer may propose either the deadline or the price, and the system lls in the missing value, if feasible.

If the system fails to complete the application before its agreed deadline, the provider pays a penalty. e penalty depends on the delay, that is, the di erence between the application's actual completion time and the deadline. We de ned three revenue functions that determine how the penalty increases with delay, that is, a linear, a bounded linear and a step revenue function.

C B

In this section, we discuss how donating and waiting bids are calculated for computational applications. In essence, the waiting bid corresponds to the estimated penalty of the new application due to waiting private resources to become available. e donating bid corresponds to the sum of the estimated penalties of the applications involved in donating the required resources.

W B

Using the QoS translator plugin, the group calculates the remaining time for each application to nish its execution and release its resources. e group's waiting time is the earliest such time when the released resources are greater than or equal to the required resources. e group's waiting bid is then the estimated penalty for delaying the new application by this waiting time.

D B

A group can impose on an application three alternative forms of donating resources: total lending, partial lending, and giving. With total lending, the application is suspended, it lends all its resources to the new request, and resumes execution once the request terminates. With partial lending, the application lends a subset of its resources to the request, continues running with the remaining resources, and gets back the resources once the request terminates. With giving, the application gives a subset of its resources to the request and continues running with the remaining resources. Rigid applications support only total lending, while elastic applications support all three forms of donation. Each form of donation impacts the remaining time of the application, possibly incurring a delay and an associated penalty. e remaining time is estimated using the QoS translator plugin and the penalty using the revenue function.

e group's donating bid is calculated as follows. For each group application and each applicable form of donation, we calculate the estimated delay and associated penalty. We then use a heuristic to calculate a subset of applications and associated donation forms that reduces the sum of penalties while providing the required resources and satisfying the QoS violation thresholds. e group's donating bid is then this sum of penalties.

. . M P

To validate the proposed pro t optimisation approach, we developed a prototype, called

Meryn, a cloud-bursting PaaS system that supports batch and MapReduce applications VCs may exchange private resources among them or burst into public cloud resources according to the policies described in Section . . . e customers submit their applications through a common and uniform interface, independently of the type of their applications.

e architecture of Meryn is shown in Figure . . e main components are the client manager, a cluster manager for each VC, an application controller for each hosted application, and the resource manager. e client manager receives application submission requests, acts as the intermediary between the customer and the corresponding cluster manager, and . . E e objective of our evaluation was to compare the generated provider pro t with each of the three policies: basic, advanced, and optimisation. For this evaluation, we used both simulations and experiments with the Meryn prototype. e used metrics include the pro t of a workload, the completion time of a workload, the number of deadline violations, the average delay, and the VM usage proportions (i.e., the proportion of VMs obtained through the local VC, a di erent VC, donating applications, waiting for the termination of applications, or the public cloud).

Simulations and experiments used the same setup. e Meryn prototype has the two previously mentioned VCs, one for MapReduce applications and one for batch applications. We evaluated two variants of the optimisation policy (opt and opt ) with di erent QoS violation thresholds combined with three revenue functions (e.g., Lopt is opt with Linear revenue function, Bopt is with Bounded linear, and Sopt is with Step revenue function).

e opt variant has higher QoS violation thresholds than opt and is thus more aggressive in impacting applications (speci cally, it can impact up to % of the applications with a delay of up to % of the deadline). For illustration, we present a subset of the evaluation results. Full details can be found in [ ]. advanced and Sopt policies. We see that the Sopt optimisation policy generates the most pro t, that is, from . % to . % more pro t than the advanced policy. is is because of the highly variable overhead of managing VMs in experiments, resulting in more cases of application delays and penalties.

Figure . shows the VM usage proportions associated with each method of obtaining VMs (i.e., local, VC, donating, waiting, public cloud) for di erent workloads and policies in experiments. We see that the optimisation policy uses public clouds to a smaller extent than the basic and advanced policies, opting, if possible, for alternative sources of VMs. Figure .   shows the completion times for the three workloads. We see that the optimisation policy produces slightly longer completion times than the basic and advanced policies (an increase of up to . % for workload and the Sopt policy). is increase is due to the incurred delays of the impacted applications and the required time for calculating the bids and transferring VMs between VCs.

Table . focuses on the QoS impact of the optimisation policy on applications and shows that the percentage of impacted applications is between % and . % and the average delay of delayed applications is between % and . % of the deadline. ese values remain below the QoS violation thresholds.

In summary, the results showed that the optimisation policy generates more provider pro t than the basic and advanced policies. As expected, the policy causes some applica- tions to miss their deadlines and slightly increases workload completion times. However, the number of impacted applications does not exceed the prede ned thresholds. Moreover, the results of simulations and experiments agree with each other, validating the evaluation.

. . D e main novelties of this work were the support for cloud bursting, enabling applications to use simultaneously private and public cloud resources, and the support for multiple application types, such as batch and MapReduce applications. For simplicity, the work only considered the allocation of homogeneous, coarse-grained resources (i.e., VMs with xed capacities). A more ne-grained, exible resource allocation model will be described in Chapter .

Moreover, the work did not consider dynamically adding resources to running applications VM types, cloud services, and components must then be instantiated and con gured appropriately to produce the initial application deployment.

At runtime, the application deployment will likely need to be adapted to accommodate environmental changes, such as changes in the workload, in application requirements, or in the availability, performance, and cost of underlying cloud services. is adaptation may involve, for example, increasing the number of web server instances or migrating instances across clouds, and must ensure that the application satis es customer requirements as well as feasible.

ere is a plethora of so ware platforms aiming to reduce this deployment and management complexity by applying self-adaptation techniques [ , , , , ]. ese platforms typically support the following four tasks (see Figure . ): ( ) modelling, ( ) deciding, ( ) executing, and ( ) monitoring.

Modelling involves specifying the deployment requirements of the application in a declarative, cloud provider-independent language. For example, this may include specifying the required operating system and geographic location of the VMs hosting individual application components. Deciding involves deriving the initial application deployment as well as deciding when and how to adapt the deployment at runtime. Executing involves applying the deployment decisions, dealing with the heterogeneity of clouds. Finally, monitoring in-volves collecting information about the running application (e.g., performance data), which is used by the deciding task, forming the typical closed-loop structure of adaptive systems.

My work on automated application management can be separated into three parts:

• development and evaluation of the adapter component for adapting multi-cloud applications, part of the PaaSage open-source application management platform;

• development and evaluation of a cost-e ective approach for adaptive application deployment that builds on and specialises the PaaSage platform;

• development and evaluation of an application management platform based on SPLs (So ware Product Lines) including a proactive policy for vertical scaling.

Each part is discussed in the following sections. To address cloud heterogeneity and facilitate application management, the project adopted a model-driven engineering approach. e project de ned a modelling language, called • Pro ler: receives a cloud provider-independent application description (e.g., application components and requirements on virtual hardware) and a set of cloud provider descriptions (e.g., VM types and prices for Amazon EC ), all written in the CAMEL language, and formalizes the problem of deploying the application on available cloud resources as a constrained optimisation problem. e optimisation problem includes variables (e.g., the number of component instances), constraints (e.g., two components must be hosted in the same VM) and an optimisation objective (e.g., minimising cost).

• Reasoner: solves the optimisation problem using an extensible set of solvers. e reasoner generates a cloud provider-speci c deployment model (e.g., application deployment on EC ).

• Adapter: receives a proposed deployment model, validates that it is acceptable, and adapts the currently running application deployment to reconcile it with the proposed model. e adapter is also responsible for monitoring and dynamically adapting the application deployment, e.g., when performance deteriorates, or cloud provider prices change.

• Executionware: enacts the deployment of application components on selected cloud providers and sets up monitoring of required metrics.

A e purpose of the adapter is to transform the currently running application con guration into the target con guration in an e cient and safe way [ ]. As shown in Figure . , the adapter is composed of three components: the plan generator, the adaptation manager, and the application controller.

e plan generator receives as input the current and the target application con gurations and generates a plan representing the required recon guration actions and their order. e adaptation manager performs three tasks: it validates that the recon guration plan is acceptable, it applies the plan to the running system in an e cient and safe way, and it maintains an up-to-date representation of the current system state. e application controller compo-nent monitors the running application and its execution context in order to detect changes that make the current deployment unacceptable. Based on the monitored information, the component triggers the execution of the full recon guration process. To handle delays and transient failures, the application controller applies simple fault-tolerance tactics, such as retrying recon guration actions or issuing redundant recon guration requests. One of the challenges in performing this optimisation is considering not only the predicted bene ts of adapting the deployment (e.g., improved performance) but also the predicted costs (e.g., delays for provisioning new virtual machines). For example, when the workload is rapidly changing, the management system should refrain from performing complex, multi-cloud recon guration actions. e reason is that the costs are unlikely to be recovered if the application deployment does not remain stable.

Most existing application management solutions do not consider adaptation costs in making adaptation decisions [ , , , , ]. A limited number of research solutions do con-sider adaptation costs [ , , , ], but these are solutions operated by cloud providers rather than customers, and they do not support multi-cloud application management.

To respond to this challenge, we proposed an approach for adaptive deployment that explicitly considers the adaptation cost in deciding when and how to adapt the deployment [ ].

e approach builds on the PaaSage architecture described in Section . . . e rest of this section describes the approach and then outlines how it was practically implemented and validated.

C D O

e approach assumes that a running application generates revenue for the application owner depending on the performance obtained by application users (e.g., depending on meeting response time guarantees). In turn, the owner pays cloud providers for the resources used by the application. e goal is then to optimise the pro t, which is the di erence between revenue and resource cost. e approach assumes that the instantaneous pro t can be derived from a pro t function that takes as input the current performance characteristics (e.g., response time) and the cost rate of used resources.

e approach proposes a continuous deployment optimisation process composed of two tasks: ( ) generating a proposed deployment that optimizes pro t under the current runtime conditions, ignoring any recon guration costs, and ( ) deciding when and how to recon gure the running system based on the proposed deployment, taking into account recon guration costs and bene ts. In PaaSage, these tasks are the responsibility of the reasoner and adapter respectively.

Regarding the rst task, the reasoner receives as input the pro t function and the current application workload and provides to the adapter a proposed deployment model and its predicted, instantaneous pro t. Internally, the reasoner maintains a performance model and a cost model, enabling it to estimate the performance and cost for a given application deployment and workload. e reasoner then searches for the deployment that maximizes the pro t for the current workload.

Regarding the second task, the adapter produces a recon guration plan consisting of the tasks (e.g., creation, update, and deletion of VMs) necessary for transforming the current deployment to the proposed one. Given the proposed recon guration plan, the adapter validates that executing this plan is actually bene cial. is validation decision relies on weighing the potential recon guration bene ts against recon guration costs. Bene ts depend on the proposed deployment and on the time that the system remains in this deployment. Costs depend on the transient system state during recon guration and the time that this recon guration lasts.

Concretely, validation is based on the following information: ( ) the estimated recon guration duration (T rc ), ( ) the estimated time until a new recon guration is initiated (called stability interval or W), ( ) the current pro t (P ), ( ) the predicted pro t during recon guration (P ) and the predicted pro t for the proposed con guration (P ). e adapter performs the recon guration if the associated pro t is higher than the pro t of doing nothing over the stability interval W, i.e.,

T rc * P + (W -T rc ) * P > W * P ( . )
e recon guration duration (T rc ) can be estimated using historical information from previous executions of recon guration tasks. e stability interval (W) can be approximated as the time during which the application workload remains relatively stable (i.e., varies within a speci c workload band), which can be predicted using historical information on workload evolution.

A I e approach was implemented by modifying and extending the reasoner and adapter components (see Fig. . ). e pro t function that we used is:

Pro f it = ⎧ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎩ reward -cost if response time < target -cost if response time ≥ target ( . )
Cost is the cost rate of used resources; reward is the revenue when the SLO is satis ed (i.e., the response time is less than the given target).

We have extended the PaaSage reasoner with a solver that relies on manual, o -line proling of the application to nd the best deployment. Speci cally, we gather performance samples from multiple executions of the application with representative deployments and workloads and use those samples to predict the performance for similar deployments and workloads. e cost of a deployment is estimated in a straightforward way using the pricing information of cloud providers.

e adapter includes a recon guration validator implementing the approach presented earlier, and predictors for the recon guration duration and stability interval. e reconguration duration is predicted using historical information on past recon guration tasks.

e stability interval is predicted by applying an autoregressive moving average lter on past measured values of this interval.

. e EPA workload, comprising an increasing and a decreasing phase, is played twice and an idle period of . minutes is inserted between the two plays. Scenario is a case when the recon guration validator is switched o , whereas the validator is switched on in Scenario .

Response time, throughput, and pro t results for the two scenarios are shown in Fig. . .

With the rst workload increase, both scenarios scale out the application tier, resulting in the same behaviour for the rst minutes. e di erence occurs at the th minute when the workload decreases. Scenario performs a scale-in while Scenario performs no action.

Indeed, the turned-on validator in Scenario denies the proposed recon guration as the predicted stability interval is small with respect to the recon guration interval. As a result, with the second workload increase, Scenario shows worse performance than Scenario .

e reasoner in Scenario eventually recognizes the increase and suggests a scale-out that brings the response time back to normal at around the th minute. When the workload . . SPL-A M P P V S is work focused on producing a complete application management solution using a modeldriven approach, similar to that followed by PaaSage. It was performed in the context of Carlos Ruiz Diaz's PhD thesis [ ], a student at the University of Guadalajara, whom I cosupervised. e main novelty of the work was the integration of a proactive adaptation policy to perform vertical VM scaling. is section rst outlines the management approach proposed by this work, and then describes the proactive adaptation policy.

SPL-M A

e application management approach [ ] is similar to that of PaaSage, but relies on a less expressive modelling language. As a result, the approach makes it easier for application owners to specify their deployment requirements, but it supports a more limited range of adaptation scenarios (e.g., it lacks support for distributing application components across di erent clouds). Speci cally, modelling relies on SPL (So ware Product Lines) techniques. Although related research had already applied SPL techniques to facilitate application deployment [ , ], support for dynamic adaptation was lacking.

Following SPL principles, the con guration choices supported by IaaS clouds are represented as feature models; speci c application deployments are represented as feature congurations. Models of the deployment are maintained at runtime and are used to facilitate adaptation. e architecture derived from this approach, called XIPE, supports all application management tasks (modelling, deciding, executing, monitoring) and is depicted in module adapts the amount of allocated memory of the VM. e adaptation policy operates as follows. Initially, the system considers only memory predictions. When a memory increase is predicted, the system takes into account RT predictions. When the RT is predicted to violate the SLO speci ed by the application administrator, adaptation is triggered. A er the adaptation, the system returns to the initial state. e adaptation policy was evaluated using experiments with a web application under a real workload trace in a private cloud environment.

e policy was compared to di erent con gurations of the default OpenStack horizontal scaling solution. e results showed that the adaptation policy provides similar or better performance than the OpenStack solutions with signi cantly reduced resource usage [ ].

. . D is section presented work aiming at facilitating taking decisions about application deployment. We rst presented the adapter tool, part of the PaaSage platform, that enables adaptation of multi-cloud applications. We then discussed an approach for adaptive application deployment that exploits the PaaSage technology. e approach explicitly weighs predicted adaptation costs against adaptation bene ts based on predictions about recon guration duration and workload evolution. Finally, we presented an application management platform that includes a proactive adaptation policy that modi es the VM memory size. Restructuring simulation applications to become cloud-based applications presents several challenges. First, the applications must be decomposed into services that can be deployed on separate machines, potentially in di erent clouds, and independently scaled. Second, automated support for application deployment and management must be provided in order to reduce the complexity exposed to scientists. is should include automated support for elasticity to optimise the application performance and cost. Indeed, simulation applications typically require di erent amounts of resources at di erent times. For example, the initialisation phase may require fewer compute resources than the main processing phase, thus making it more e cient to scale up the resource allocation at runtime. Finally, handling failures must be supported, which is essential for avoiding data loss and ensuring the correct .

. D FF SE F

e DiFFuSE framework is structured in three layers (see Figure . ). Seen from the bottom up, the layers are increasingly speci c to the domain of epidemic simulations. At the bottom is a cloud deployment and execution platform, capable of running any modular application across clouds. Next is the resource coordination layer that supports building services that exchange data using a speci c set of provided mechanisms. At the top is the epidemic simulation services layer that supports building epidemic simulation applications as sets of cooperating services. e three layers are outlined in the following.

E S S

is layer provides reusable design and code for commonly required functionality in epidemic simulation applications, such as controlling experiments, executing simulation runs, and managing the ow of simulation data. is functionality is provided as a collection of services that can be easily customised and used by developers. e services fall into three main categories.

• Data Services: retrieve data from databases and provide these data as well as related meta-data to other services.

• Computing Services: perform the main computations of the simulation, either sequentially or in parallel, using single-computer parallelisation (e.g., using OpenMP).

Computing services include services that execute simulation runs (called worker services) as well as services that generate and process the nal results. Here we cover a single experiment that validated that DiFFuSE helps reduce data transfer times in multi-cloud deployments. In such deployments, components of the simulation application are located at di erent, geographically distributed data centres. As a result, network characteristics have a strong e ect on application performance. In the experiment, data services were deployed in an OpenStack-Grid' private cloud and worker services in the Amazon EC public cloud. e workers were con gured to be in the same EC availability zone in order to reduce network latency.

When a worker is initialised, the worker obtains data from data services, which causes tra c of about GBs from the private cloud to the public cloud over the wide-area network connection. is would normally be repeated for every worker that joins the simulator with a negative e ect on the simulator performance. To reduce this negative e ect, we applied the DiFFuSE replication feature, enabling the worker that rst obtains the data to provide replicas of the data to other workers. In this way, data transfer tra c was restricted within the boundary of the public cloud.

To evaluate the replication feature, we compared data transfer times for the BVDV simulator when data replication was enabled (E) and disabled (D) for di erent numbers of workers ( , , , workers) processing a total of simulation runs. S is chapter examined work on automated application management on behalf of customers.

First, it discussed methods and tools for supporting application recon guration across multiple clouds, drawing on a model-driven approach. en, it concentrated on the special case of epidemic simulation applications and presented a solution for restructuring, deploying, and executing such applications in cloud environments. e next chapter describes a management solution that coordinates the customers and the provider of a private cloud.

A R M P C

To jointly satisfy the objectives of customers and the provider, we need to coordinate their interactions through appropriate rules and incentives. is is a main theme of this chapter.

Indeed, in the previous two chapters, we examined cloud management from the perspective of the provider or the customer. In each case, we did not consider how the other party behaves in pursuing its objectives. For example, in Chapter , we did not consider how customers negotiate SLAs, or how they react to contract rescissions, request cancellations, or SLO violations. In this chapter, we take a holistic approach that examines how all actors interact and pursue their own objectives within a private PaaS system.

e practical problem that motivated this work arose in the context of a collaboration with EDF (Électricité de France) R&D. is organisation needed to share its private infrastructure among scientists and engineers, who wanted to execute diverse applications (e.g., batch, bag-of-tasks, MapReduce applications) with various performance objectives (e.g., meeting deadlines). Using public cloud resources was excluded owning to security constraints. We identi ed two key requirements that a resource management system for this organisation should satisfy. First, given that the capacity of the infrastructure is limited, the system should distribute resources to the customers that value them the most. Indeed, traditional resource management systems for clusters (e.g., batch schedulers [ , ]) rely on priority queues or fair sharing policies, providing no incentive to customers to relax their demand during contention periods, eventually leading to resources not being put to their best use. Second, the system should provide the exibility to support a variety of applications, customer-de ned SLOs, and policies for achieving these SLOs.

To meet these requirements, we proposed a novel application and resource management solution for private PaaS clouds. In our solution, the provider seeks to increase economic e ciency by virtualising the infrastructure and distributing ne-grained virtual resources to customers through a market-based mechanism, namely an auction. Customers seek to satisfy the SLOs of their applications through employing controllers that dynamically adapt the applications' resource demand.

Many resource management systems focused on providing SLO support to applications, typically using dynamic controllers based on performance models [ , , , , ]. Unlike our solution, these systems target speci c application types and lack the exibility to support a wide range of applications and SLOs. Other related research focused on sharing a cluster between applications requiring di erent programming frameworks

• First, we proposed a market-based approach that allocates virtualised resources to applications while enabling each application to individually and dynamically adapt its resource demand based on customer needs. e approach also provides adaptation policies for vertical and horizontal application scaling.

• Second, we proposed a prototype platform, called Merkat, that applies the approach to support generic and extensible application and resource management for private clouds. e prototype platform was deployed and evaluated within EDF R&D.

e remainder of this chapter is structured as follows. Section . presents the system model underpinning the approach. Section . describes the implementation of the auction mechanism, and Section . describes the proposed adaptation policies. A brief overview of the Merkat prototype is provided in Section . , followed by a discussion of the evaluation in Section . . Finally, Section . makes concluding remarks.

.

P S S M

In our approach, we assume a private PaaS system hosting applications of di erent types supplied by customers (see Figure . ). Each application runs in its own virtual platform, e bene ts of using this market-based mechanism are threefold. First, the mechanism provides incentives to customers to use resources judiciously given that customers are sel sh, thus, increasing the economic e ciency of allocations. Second, it provides a generic way to control resource demands, making it easier to support diverse kinds of applications, SLOs, and policies for meeting those SLOs. Finally, supporting ne-grained, dynamic allocation helps to improve the utilisation of the infrastructure.

.

A I

We describe next how the provider implements the proportional-share auction. is implementation follows four steps: (i) VM bid submission; (ii) VM resource allocation; (iii) VM placement; (iv) price computation.

VM B S

To provision a VM, the application controller submits a bid (b c pu ,b memor y ) with values for each VM resource. e initial bid can be computed based on the current price or past price history, and the bid can be modi ed to cope with price uctuations or changes in resource demand.

VM R A

e VM Scheduler periodically computes resource allocations for the VMs by considering their bids and resource utilisation limits (a max_c pu , a max_memor y ) that specify the maximum resource usage for a VM. e resource allocation computation is performed in two steps:

(i) the VM Scheduler computes the VM allocation considering the entire infrastructure as a single physical node; (ii) the VM Scheduler corrects the allocations to cope with the fact that the infrastructure capacity is partitioned among nodes.

When considering the entire infrastructure as a single physical node, the VM Scheduler computes the resource allocations as follows. Given a set of bids b j for VMs j ∈ { ..n}, and a resource with a capacity of C units, the resource allocation for j will be:

al loc j = b j ∑ n b k ⋅ C, ( . )
Since the infrastructure capacity is partitioned between nodes, there are situations when resulting allocations cannot be enforced. For example, consider physical nodes each with a capacity of CPU units ( CPU unit represents % of CPU time). We need to allocate

VMs that have a bid of credits, and VMs that have a bid of credits. Using Equation . ,

. Auction Implementation the VM Scheduler computes ideal allocations (al loc ideal j ) for the VMs as follows: . CPU units for the rst VMs and . CPU units for the last two VMs. It is practically impossible to enforce these allocations on three physical machines. To solve this issue, the VM Scheduler can place the rst VMs on the same physical node, and the last VMs on separate nodes.

e VM Scheduler can then recompute the allocations using the capacity of individual nodes in Equation . . In the example, the resulting, real allocations (al loc real j ) are: . CPU units for the rst VMs and CPU units for the last VMs. e allocation di erence is called allocation error and is de ned as follows: for the applications running in the VMs, the load balancing process tries to make a trade-o between the number of performed migrations and the VM allocation error. For example, it may not make sense to migrate a VM when its allocation error is %. To select the VMs to be migrated at each scheduling period, the VM scheduler relies on an algorithm based on a tabu search metaheuristic method [ ].

Algorithm explains the load balancing process. e algorithm receives the list of current nodes, nodes, the list of running VMs, vms, the list of VMs to be started at the current scheduling period, newvms, and three thresholds: (i) maximum number of iterations performed by the algorithm to obtain a better placement than the current one, N iter ; (ii) maximum allocation error supported for the VMs in their current placement, E max ; (iii) maximum number of migrations required to reach a better placement, M max . Based on this information the algorithm computes the new placement of VMs on nodes and outputs a migration plan, composed of the VMs to be migrated, and a deployment plan, composed of the VMs to be started.

P C

e resource price is computed as the sum of all bids divided by the total infrastructure capacity. If this price is smaller than a prede ned reserve price, then the reserve price is used. Customers are charged for each running VM at each scheduling interval with a credit amount equal to the product of the resource price and the resource allocations.

Algorithm

. A P e application controller applies policies to adapt the application on behalf of the customer.

To illustrate this adaptation, we discuss two policies-one for vertical scaling and one for horizontal scaling-that modify the application resource demand to meet customer SLOs under budget constraints. e policies run periodically and use two performance thresholds, upper and lower, as a trigger: when the performance metric traverses a threshold, the policy takes actions that change the application resource demand. e vertical scaling policy adapts the bids of each VM and uses suspend/resume mechanisms to avoid high price periods. is policy can be applied for applications with a static number of VMs, such as MPI applications, with customer-provided time constraints. e horizontal scaling policy adapts both the number of VMs and their bids. is policy can be applied for elastic applications, such as task-processing frameworks (e.g., Condor, Hadoop).

In general, the policies behave as follows:

• When the SLO is met, the policy reduces the resource demand of the application, thus saving the budget to run other applications.

• When the SLO is not met and the budget allows it, the policy increases the resource demand of the application.

• When the SLO is not met and the budget is insu cient, the policy may suspend the application, or stop the application, or reduce its resource demand.

e policies are outlined next. Full details can be found in [ ].

. . V S P is policy generates the resource bids for the next period based on the application performance metrics, the current allocations, and the budget. e policy decreases the resource bids if the performance metric drops below the lower threshold (e.g., the application is expected to nish sooner than its deadline), or if the allocation per VM for one resource reaches the maximum. If the performance metric is above the higher threshold (e.g., the application is expected to miss the deadline), the policy increases the resource bids. If the current budget is not enough to meet the SLO, the policy takes a decision based on the application and SLO types. Speci cally, if it is advantageous to suspend the application execution (e.g., in the case of best-e ort or deadline-driven batch applications), the policy suspends the application, resuming it when the price drops. If it is not advantageous to suspend the application execution (e.g., for interactive applications, or applications for which the customer might be satis ed with partial results delivered at a deadline), the policy recomputes the bids in a way that favors the resource (CPU, memory) with a small allocation (i.e., the resource that represents a bottleneck in the application's progress).

.

. H S P

is policy generates the number of VMs and the bids for the next time period based on the performance metrics, the current resource prices, and the budget. e policy decreases or increases the number of VMs depending on whether the SLO is met is not. e policy ensures that the resulting number of VMs is less than or equal to an upper bound on the number of VMs. is upper bound is calculated as the maximum number of VMs permitted by the application budget so that all VMs receive their maximum resource allocation (i.e., a max_c pu ,a max_memor y ) at the current resource prices.

. M P

To validate our proposed resource and application management approach, we built a prototype, called Merkat, whose architecture is shown in Figure . . Merkat is composed of three main services: e VM Scheduler is in charge of allocating resources to running VMs and computing their node placement using Algorithm . e Virtual Currency Manager applies virtual currency distribution policies and manages customer and application budgets. e

Virtual Platform Manager acts as the entry point for customers to use the system.

To submit an application, the customer uses a virtual platform template that speci es the con guration of the VMs (e.g., VM disk images), the type of application controller, budgeting . E

We conducted a set of experiments to evaluate the extent to which the approach supports customers in meeting their SLOs. We also wanted to validate the exibility of the approach in supporting di erent application types and SLOs. To that end, we performed both simulations and testbed experiments described in turn next.

. . S L T

To test Merkat with large workloads, we used simulation since this would have been impractical to do in a real testbed. Speci cally, we implemented the algorithms of Merkat in

CloudSim [ ], an event-driven simulator implemented in Java. e applications are batch applications composed of a xed number of tasks running in parallel. To express the diversity of customer requirements, we de ned three types of customers:

• Full deadline customers: e customer wants the results by a speci c deadline; if the application fails to terminate before the deadline, the customer is unsatis ed.

• Partial deadline customers: e customer wants the results by a speci c deadline; the customer values partial results at the deadline.

• Full performance customers: e customer wants the results as soon as possible, but can also accept a bounded delay.

Each customer type is associated with a utility function that measures the value obtained by the customer (called customer satisfaction) as a function of the application execution time.

Each type is also associated with an adaptation policy, which specialises the vertical scaling policy described in Section . . to re ect customer requirements. e workload was based on the HPC N trace [ ], containing a log of applications submitted to a Linux cluster with nodes. is trace was chosen because of the information regarding memory requirements of applications. Since the trace had no information on deadlines, we assigned synthetic deadlines to applications, between . and times the application execution time.

We assigned budgets inversely proportional to this deadline factor.

e experiments showed that Merkat can accommodate di erent types of customers while providing good customer value and with limited VM management overhead (migration and suspend/resume operations). Full results can be found in [ ]. Here we focus on the comparison of the total satisfaction provided by Merkat compared to two traditional scheduling policies, namely, FCFS (First Come, First Served) and EDF (Earliest Deadline First). FCFS is the policy typically applied by IaaS cloud managers. EDF is a policy typically applied to minimise the number of missed deadlines. It is important to note that cloud systems cannot practically apply EDF or similar algorithms without limiting their support to a prede ned set of customer goals (e.g., in the case of EDF, meeting deadlines). We chose EDF in order to compare our exible approach with a specialised, deadline-driven system. We observe that (i) the system outperforms FCFS in all cases, as FCFS does not consider application valuation or SLO in its decisions; (ii) when the contention is not high, the system outperforms EDF in terms of customer satisfaction; and (iii) when the system is highly loaded, EDF performs better. e performance gap between the mechanism and EDF in case of high contention can be explained as follows. With EDF, the central scheduler takes consistently good scheduling decisions; it sorts applications by their deadline and executes the application with the smallest deadline rst. us, more applications with smaller deadlines get to run on time, providing higher satisfaction. With Merkat, applications do not consistently take the best allocation decisions since they adapt independently with only limited information. is leads to a performance degradation, the result of allowing applications to behave sel shly (i.e., the price of anarchy [ ]).

. . T E

For the experiments on a real testbed, we used three applications with di erent SLOs and associated policies. Speci cally, we used an MPI application with a vertical scaling policy that attempts to meet customer deadlines, the Condor framework with a horizontal scaling pol- icy that attempts to regulate the number of queued tasks, and the Torque framework with a horizontal scaling policy that attempts to regulate the wait time in the queue per task. e experiments validated the exibility of Merkat in supporting di erent applications, SLOs, and policies. ey also showed that the applications can adapt to changes in operating conditions (e.g., price increases in contention periods) as well as to changes in customer requirements (e.g., changes in deadlines) [ ].

In a particular experiment, we compared Merkat with Maui, a popular batch scheduler in HPC environments [ ]. Speci cally, we deployed Merkat and Maui on a cloud of compute nodes of Grid' and ran MPI applications over a time interval of hours. e applications had to processes, each process in one VM. We considered full-deadline customers, and generated the workload based on a Lublin [ ] model. In Merkat, each application ran with the vertical scaling policy. Maui used FCFS and back lling to schedule applications.

Table . summarizes the results of this experiment. e results showed that, when using Merkat, the number of applications that nish before their deadline increases by .

percentage points compared to using Maui. Using Merkat also increased the customer satisfaction compared to Maui (for Merkat, the total satisfaction was credits, compared to a negative satisfaction of -credits for Maui). ese results provided evidence of the e ectiveness of Merkat in managing real clouds. To further validate the results, we com-pared Merkat with its CloudSim-based simulation on the same workload trace. We found a di erence of percentage points in the number of applications that meet their deadlines, with more applications meeting their deadlines in the simulation. We also see a di erence in the total satisfaction and the number of migrations. e reason for the di erence is that Merkat/CloudSim relies on a simple application model in which performance interference is not accurately modelled. When running real applications on Merkat, Merkat's algorithms have to cope with higher variability in performance metrics, which leads to higher variability in bids and resource allocations, and thus a higher number of migrations.

. Increasing the proximity to users promises multiple bene ts, including highly responsive services, reduced bandwidth consumption, support for user mobility, and enhanced privacy.

However, realising these bene ts poses several research challenges. First, compared to traditional cloud computing, fog computing exploits a wider heterogeneity of resources (e.g., sensors, mobile devices, gateways, micro data centres, cloud data centres) and network links (e.g., wireless access technologies, backbone links). is diversity makes it more di cult to de ne common interfaces and abstractions for accessing these resources and models for reasoning about their capabilities. Importantly, this diversity also expands the range of possible application deployments, making it more di cult to make application placement decisions.

In the context of the FogCity project [ ], we have initiated work on a fog resource management scheme that takes into account fog node heterogeneity. e scheme is based on a bargaining game and seeks to satisfy the QoS preferences of customers (e.g., application delay and location preferences) while maximising node utilisation.

Second, compared to traditional cloud execution environments, fog environments are more dynamic and subject to unpredictable changes. is results from user and device mobility (e.g., personal devices, cars, drones), resource volatility (e.g., battery shortage), and topology changes (e.g., fog nodes joining, leaving, failing). e higher rate of environment change requires faster management decisions, motivating further research towards improved decision-making approaches and decentralised decision-making structures.

Finally, fog computing will involve a complex interplay of economic actors. Although the fog ecosystem has not yet stabilised, one possible scheme comprises infrastructure providers, service providers, end users, and edge resource owners [ ]. In this scheme, infrastructure providers own and manage large-scale infrastructures of devices, networks, and clouds.

Service providers rent resources from infrastructure providers and create applications (e.g., video surveillance) used by end users. Edge resource owners (e.g., individuals, companies) own small-scale infrastructures of edge resources and rent these to infrastructure providers.

. Outlook

Coordinating the complex interactions between such actors and automatically managing the multiple associated contracts in line with actors' objectives is a challenging area that requires further research.

. . I D -M E ective decision-making is an essential ingredient of all management systems. In the work described in this document, decision-making typically relied on simple heuristics (e.g., Section . . , Section . ) or metaheuristics (e.g., Section . ) combined with performance, cost, and workload models (e.g., Section . . , Section . . ). e drawback of such techniques is the complexity of designing appropriate heuristics and of manually tuning them to handle changes in operating conditions (e.g., new application types, new workload types).

Machine learning techniques, such as deep reinforcement learning, have the potential to alleviate this drawback. Indeed, these techniques enable taking decisions based on dynamically collected data without requiring a priori modelling or expert knowledge, while automatically learning and adapting to changing environments. Machine learning techniques are increasingly being applied to cloud management problems with promising results [ , , , ]. Nevertheless, to enable widespread deployment of these techniques in real-world settings, many challenges remain to be addressed, including exploring the performance, stability, and generalisability of di erent learning algorithms [ ], improving their interpretability [ ], integrating them into practical cloud management systems, and evaluating their effectiveness.

. . D M

Current and emerging cloud and fog ecosystems encompass independent interacting systems operated by actors with distinct, potentially con icting interests (e.g., application owners, service providers). A challenge that arises in such decentralised systems is global manageability [ ], that is, coordinating the interactions of the independent systems in order to ensure global properties, such as maximising total value or fairness.

A well-known approach to addressing the global manageability challenge is applying eco-

  customers and the provider to jointly serve their interests. e assumed setting was that of a private PaaS system in which the objective of the provider is to e ciently distribute resources between customers, taking into account sel sh customers. Customers have various, customer-speci c objectives and run an open-ended set of application types. D O e following three chapters discuss each research strand in turn. ey present the context of each piece of research work, the main contributions, and how the contributions were evaluated. e nal chapter summarizes the key ndings and discusses areas for future study. of cloud systems should be automatically driven not only by QoS and technical goals but also by economic goals. is observation underlies the research described in this chapter. Speci cally, the chapter discusses automated resource management solutions for public cloud providers. Unlike most commercial cloud providers, the providers assumed in this work o er strong QoS support. In particular, they support establishing and maintain-

  depict the results of fault-tolerance (FT) scenarios. e gures

  Figure . : Heterogeneous customer pro le with three penalty policies

Figure

  Figure . : Meryn architecture

e

  private cloud has VMs and the public cloud has unlimited VMs. Simulations used a stripped-down prototype that creates no real VMs and applications and runs in a single machine with limited resources. Experiments used the Grid' [ ] testbed; the private cloud had physical nodes and the public cloud had physical nodes and was geographically distant from the private cloud. We used three workloads, each comprising a batch and a MapReduce workload submitted simultaneously. e batch workloads follow the Lublin workload model [ ]. e MapReduce workloads follow the Facebook workload reported in [ ]. We de ned three SLA classes (high, medium, and low) and randomly assigned classes to the applications in the workload.

  Figure . : Pro t for di erent workloads and policies

  Figure . b shows the generated pro t in experiments for the three workloads and the basic, advanced, Lopt , Bopt , and Sopt policies. e results are similar to the ones obtained with simulations. e main di erence is that with experiments all policies generate less pro t than with simulations.

Figure . :

 . Figure . : VM usage proportion for di erent workloads and policies (experiments)-VM usage is calculated as number of VMs multiplied by duration

Figure . :

 . Figure . : Workload completion time from the submission of the rst job to the completion of the last job (experiments)

  . . P S A is work focused on developing a practical solution for adapting multi-cloud applications and was performed within the PaaSage project ( -), funded by the European Union. Most adaptation solutions that emerged during the lifetime of PaaSage supported only horizontal scaling with a single cloud [ , , ]. Moreover, these solutions relied entirely on developers and operations engineers to design appropriate scaling rules to meet customer objectives. Adaptation in PaaSage, on the other hand, included support for cross-cloud application restructuring based on high-level objectives. is section rst outlines the overall PaaSage architecture and then provides further details on the adapter, the adaptation solution whose development I coordinated within PaaSage. P S A e PaaSage project developed a model-based, multi-cloud development and deployment platform that enables developers to access cloud services in a technology-neutral manner while guiding them to con gure their applications for best performance [ , ]. e project was motivated by the heterogeneity of the available clouds and the associated complexity of porting existing applications to a cloud or switching between clouds. e project slogan was "De ne your application once. Deploy to the full spectrum of Clouds".

Figure . :

 : Figure . : PaaSage architecture

  e platform includes solvers based on mixed integer linear programming, constraint logic programming, and learning automata [ ].e optimisation objective is expressed using a utility function representing the preferences of the application owner among

Figure

  Figure . : Adapter architecture

e

  adapter brings key features to the PaaSage platform. First, it enables automatic synchronisation of models with the running system, removing the need to manually write lowlevel recon guration scripts. Second, it supports a rich set of recon guration actions beyond horizontal scaling, such as large-scale application restructuring and migrating components across clouds. ird, it performs recon gurations in an e cient and reliable manner through parallel plan execution and handling failures. Finally, in conjunction with the reasoner and the executionware, it enables setting up control loops to monitor and adapt the application deployment. e bene ts of the PaaSage platform, including the adapter, were demonstrated using several use-case applications developed by PaaSage partners, including a ight-scheduling application developed by Lu hansa Systems, and a scienti c work ow engine developed by the AGH University of Science and Technology [ ]. e PaaSage project was successfully completed in November with an excellent rating from the reviewers. e PaaSage platform is currently being reused and extended within the Melodic European project ( -)[ ] that speci cally targets data-intensive applications and supports data-aware application deployment. We used the PaaSage technology to implement the DiFFuSE framework described in Section . . In [ ], we describe early work on applying PaaSage to IoT applications in order to support on-demand provision of cloud resources when edge resources are directed at continuously optimising both the performance and the monetary cost of a multi-cloud application through a speci c con guration of PaaSage mechanisms.

Figure . :

 . Figure . : Architecture for continuous deployment optimisation

Figure . :

 . Figure . : Response time, throughput, and pro t with the validator switched o (Scenario ) and on (Scenario )

Figure

  Figure . . e main components are the user interface, the SPL manager, the model manager, the adaptation manager, and the communication component. e user interface handles interactions with two types of users, platform administrators and application administrators. Platform administrators de ne and specialise SPL-based templates that describe features supported by particular IaaS clouds (e.g., based on OpenStack).ese features include VM hardware types, operating systems, and supported so ware stacks. Application adminis-

Figure . illustrates the

  Figure . illustrates the design of the XIPE proactive adaptation solution. e solution consists of three main modules: monitoring, prediction and execution. e monitoring module collects information about the memory utilisation of the VM and the response time (RT) of the application component deployed in the VM. e prediction module consists of predictors for RT and memory utilisation that receive as input information from the monitoring module. e predictors rely on Recursive-Least-Square (RLS) lters [ ]. e execution

e

  work validated the e ectiveness of the model-driven approach followed by PaaSage and XIPE in facilitating application deployment and recon guration. Moreover, the work validated the e ectiveness of the adaptive deployment approach in optimising the application owner pro t under various circumstances. A limitation of the work is that it assumes the capability to predict application performance for di erent deployments and workloads. A large amount of research has recently focused on such predictions using various methods including analytic models and machine learning [ ]. Exploiting such research is a natural direction for future work. this work was enabling legacy epidemic simulation applications to be deployed in the cloud. Indeed, the cloud is an attractive platform for resource-intensive applications because it enables rapidly obtaining practically unlimited amounts of resources with pay-per-use pricing [ ]. Yet, legacy simulators, such as those developed by the BIOEPAR [ ] research unit at INRAE (National Institute of Agricultural Research), are typically structured as monolithic applications executed on dedicated, high-end multicore systems and require signi cant restructuring in order to exploit the cloud.

  Figure . : DiFFuSE architecture

Figure . :

 . Figure . : Data transfer time for di erent numbers of workers

  are received the VM Scheduler places them initially on the nodes with the lowest resource utilisation. To minimize the VM allocation error, the VM Scheduler might migrate VMs between nodes. e process of migrating VMs among nodes is called load balancing. As having a high number of migrations leads to a performance degradation

Figure

  Figure . : Merkat prototype

  Figure . (a) describes the results of this comparison. It considers full-deadline customers and the associated Merkat policy. Figure . (b) describes the number of applications that successfully nished their execution before their deadline.

  Figure . : Performance in terms of (a) total customer satisfaction, and (b) percentage of successfully nished applications for Merkat, FCFS, and EDF-contention increases from le to right.

  nomic and pricing mechanisms [ ]. ese mechanisms specify the rules and incentives that govern interactions among self-interested agents, which trade services with each other. We have already applied auction mechanisms to coordinate the interactions among customers and the provider in private clouds (Chapter ) and among application type-speci c resource groups in a PaaS implementation (Section . ). Although there is an extensive literature on applying economic and market-based mechanisms in cloud resource management [ , ], Conclusion the mechanisms have seen limited adoption in real, commercial clouds. e main reason is arguably the complexity of such mechanisms for human users [ ]. is may be of less concern when automated agents are involved, potentially employing sophisticated decisionmaking techniques (Section . . ). Further research is clearly needed in using economic mechanisms for decentralised management in cloud and fog deployments. An important requirement is selecting appropriate mechanisms for a given setting. Such settings may include, for instance, selling negrained IaaS resources for short time intervals [ ], leasing edge resources to fog infrastructure providers [ ], allocating micro data centre resources to low-latency applications [ ], or placing functions on fog-based FaaS platforms [ ]. Selecting mechanisms will require considering not only their economic properties (e.g., economic e ciency, truthfulness), but also their technical properties (e.g., scalability

  plication, comprising di erent services (e.g., computation, storage) with di erent characteristics (e.g., VM types) from di erent cloud providers. e availability, cost, and performance of any selected cloud infrastructure are also subject to change (e.g., network failures, price changes, performance variations due to resource sharing). Ensuring that customer objectives are satis ed despite this complexity and dynamism is thus the central challenge in designing application management systems.Several concerns arise in overcoming this challenge. First, the management system should be exible to support di erent customer objectives. Customer objectives typically include achieving the SLOs of application users as well as reducing the charges paid to IaaS or PaaS providers for application execution. Second, the management system should take into account the potential costs of recon guration actions. Indeed, adapting a deployment requires Resource management for providers.e rst strand focused on provider-side management. We proposed two automated solutions, one for SaaS providers and one for PaaS providers. In both cases, the provider objective is to increase pro t. e SaaS solution uses a single IaaS cloud as the infrastructure. e PaaS solution is extensible with respect to application types and uses as an infrastructure a private IaaS cloud that automatically bursts into public IaaS clouds.

	. . Application management for customers. e second strand focused on customer-side
	management. First, we proposed multi-cloud application management solutions for
	IaaS customers. e considered customer objectives are increasing performance and
	reducing cost. e applications were assumed to be designed to run in the cloud. Fol-
	lowing that, we focused on a particular type of legacy applications, that is, epidemic
	simulation applications, and proposed a solution for enabling them to fully exploit
	cloud platforms.
	. R	A
	My activities can be structured into three research strands:

executing recon guration actions (e.g., migrating components to di erent clouds), potentially expensive in terms of time and monetary costs.

ese costs should be carefully balanced against predicted bene ts before adapting the deployment. ird, the management solution should allow customers to deploy an application across multiple clouds. Potential bene ts of multi-cloud deployment include avoiding dependence on a single cloud provider, taking advantage of lower resource prices or resource proximity, and enhancing application availability. E e simplest way to migrate a legacy application to the cloud is encapsulating the application in one or more VMs that run in an IaaS cloud. However, this type of migration does not allow realising the full benets of the cloud in terms of elasticity, performance, and cost savings. Realising those bene ts invariably requires restructuring the legacy application to fully exploit cloud services. Application management could then extend to supporting customers in restructuring legacy applications and running them in the cloud. To facilitate this restructuring, customers should be provided with tools, guidelines, and reusable code. e application deployment should then be continually managed to optimise application performance and cost, as described previously. Since failures are common in cloud environments, automated support for dealing with failures should also be included. . Application and resource management in private clouds. e previous two research strands studied systems that independently serve the interests of either providers or customers. is strand focused on designing a collaboration structure that allows the

  IaaS systems as a target infrastructure for deploying services, which then led us to consider the economic goals of the service provider. We thus extended the solution to support SLAs specifying both QoS and prices and to actively seek to maximize the provider's pro t[ , , ]. e main novelty of this work was the provision of a complete SLA management solution oriented towards increasing the provider pro t. Indeed, although a signi cant amount of research had investigated QoS management for large-scale distributed applications[ , , ], there was little research in explicitly addressing the provider's business objectives. Moreover, although a few systems employed similar QoS and resource management mechanisms[ , ] or had exible architectures, potentially enabling them to integrate such mechanisms [ , , ], our work was the rst to describe the design of these mechanisms,

	R	E	M	S S P
	e initial motivation for this work was developing a QoS assurance solution for services
	deployed on grid infrastructures [ , , ]. e increasing popularity of clouds prompted
	us to add

  levels (i.e., high, medium, low) of response time and reliability QoS. e QoS levels are translated to mechanism-level con gurations that enable the system to deliver the targeted QoS level. Speci cally, response time levels are translated to the number of workers used for service execution. is translation is done by pro ling the service with di erent numbers of workers and classifying response time values as high, medium, and low. Reliabil-

	. . S	D	M
	e approach provides mechanisms for creating SLAs, managing resources, and managing
	the execution of services.
	SLA C
	e proposed approach de nes di erent SLA types (i.e., fast, safe, classic, standard) by com-
	bining di erent
	time used. Resources can be shared among workers from di erent contracts. e provider
	objective is pro t optimisation. e pro t takes into account the cost of underlying resources
	over time as well as the penalties.

To execute a customer request, the SaaS system uses resources provided by a private or public IaaS cloud. Resources are assumed to be homogeneous (i.e., VMs with the same number of cores and memory sizes) and charged per ity levels are translated to the number of times that faulty workers are replaced. For example, if reliability is high, the system replaces both delayed and crashed workers; if reliability is medium, it only replaces crashed workers.

Table . :

 . Percentage of delayed applications (A) and average delay of delayed applications (B) with the optimisation policy (experiments)In seeking to address every customer need, commercial IaaS providers are delivering an everwidening range of features, making it increasingly complex to use IaaS clouds. Shielding customers from this complexity is the central theme of this chapter. Speci cally, this chapter describes automated application management solutions for customers who want to deploy their applications across one or more IaaS clouds.

	. D		M	M	-	A
		workload	workload MODELLING	DECIDING	workload EXECUTING
		A	B		A	B		A	B
	Lopt	%	. %	%	. %	. %	. %
	Bopt	. %	%	. %	. % MONITORING	%	. %
	Sopt	%	. %	. %	%	. %	. %
								Cloud deployment
	to improve their QoS. Further research in dynamic application adaptation will be discussed
	in Chapter and Chapter . Natural extensions of this work include taking into account
	data transfer costs, and adding support for further application types-such as work ow or
	data streaming applications-allowing a more complete assessment of the usefulness of the
	approach.
	MIHMES project (	-	) [ ], funded by ANR under the Investments for the Fu-
	ture Program, and the DiFFuSE ADT project (		-	) funded by Inria. I performed
	this work in collaboration with Linh Manh Pham (postdoctoral engineer), Yvon Jé-
	gou, and Christine Morin from Inria, and Sandie Arnoux, Gaël Beauneée, Luyuan Qi,
	Philippe Gontier, and Pauline Ezanno from INRAE Oniris.
	We discuss each part in turn in the following.

S e chapter presented two automated resource management solutions for cloud providers. e rst solution was designed for SaaS providers that manage resources from an IaaS cloud and deliver master/worker-based services. e second solution was designed for PaaS providers that manage resources from a hybrid IaaS cloud and host extensible application types. e next chapter turns to solutions for application management on behalf of cloud customers. A M C e chapter is divided into two parts. • e rst part discusses managing modular applications, composed of independently deployable units communicating over the network. ese may be applications designed to run in the cloud or applications that can be easily migrated to the cloud with minimal or no changes. is work was performed in the context of the PaaSage European project ( -) [ ] in collaboration with Linh Manh Pham (postdoctoral engineer), Arnab Sinha (postdoctoral engineer), and Christine Morin from Inria, and the PhD thesis of Carlos Ruiz Diaz [ ] that I co-supervised with Hector-Duran Limon (University of Guadalajara). • e second part discusses managing a speci c type of applications, namely epidemic simulation applications, with a monolithic structure. Such applications require signicant restructuring in order to run in the cloud and thus gain access to large amounts of resources rapidly and cost-e ectively. is work was performed in the context of the Application Figure . : Application management architecture e motivation for this research was reducing the complexity of deploying and managing applications over IaaS clouds. e research assumes that the application is structured as a set of interconnected, deployable components (e.g., web servers, databases) that need to be deployed over one or multiple IaaS clouds. is deployment involves selecting appropriate VM types to host each component together with associated cloud services (e.g., load balancers, storage). Making this selection involves considering an extremely wide range of cloud o erings with di erent characteristics (e.g., di erent CPU and memory capacities for VMs) and prices, provided by di erent cloud providers, accessible through di erent interfaces. ese

  Deploying and Managing Multi-cloud Applications

	PaaSage
	Adapter
		Validator
	Reasoner
	Stability interval	Reconf. duration
	predictor	predictor
	Performance/Cost	Reconfiguration
	metrics	actions
	Cloud Application
	IaaS Clouds

•

  Experiment Services: control the life-cycle of experiments. ey track the progress of the computations, they handle service failures, and they provide con guration data about the experiments to other services. R C is layer enables services to exchange data in a reliable and scalable way. In DiFFuSE, services run as separate processes communicating over the network. Services use, provide, and track the state of resources, de ned as any named data (e.g., experiment con gurations, experiment data). A resource can have multiple providers, each o ering the same data. Such replicated resources can be used to support fault-tolerance and improve performance. A service can obtain the data from an available provider and become itself a provider. -based applications based on DiFFuSE. In total, . % of the original BVDV code and . % of the original MAP code were modi ed in order to convert them to cloud-based applications. In the nal DiFFuSE-based BVDV application, the proportion of the code that is speci c to the BVDV spread simulation (rather than reusable code) is around . %. In the nal MAP application, this proportion is around . %. ese measures provided evidence of the limited e ort required for applying DiFFuSE and of the added value of the DiFFuSE reusable code. Using the BVDV and MAP simulators as reference applications, we conducted a set of experiments to evaluate DiFFuSE in terms of performance, cost e ectiveness, failure handling, and elasticity in single and multi-cloud settings. e experiments validated that using DiFFuSE reduces execution time and cost compared to the legacy simulators, thus enabling scientists to perform simulations previously considered as impractical or prohibitively expensive. Moreover, the experiments demonstrated the supported exibility in deploying DiFFuSE-based applications, which allows customers to make di erent cost-performance trade-o s by controlling the number and types of used cloud resources. Finally, the experiments validated the elasticity support provided by DiFFuSE as well as its ability to simulta-

	transformed to cloud
	. . E
	DiFFuSE was used to restructure two legacy simulators: a simulator of the spread of bovine
	viral diarrhea virus (BVDV), and a simulator of the spread of Mycobacterium avium sub-
	species paratuberculosis (MAP). e legacy simulators were developed by INRAE and were

e layer includes mechanisms to handle di erent types of failures, including deployment-order failures, service failures and, in particular, worker failures. It handles deployment-order failures through using the nanomsg message library [ ], which transparently applies retry and timeout mechanisms, allowing services to be launched in any order. It handles service failures through maintaining links between services and having services check that they periodically receive messages from linked services and take appropriate actions when failures are detected. It handles worker failures through reallocating lost simulation runs to other workers. C D E is layer enables hosting the services composing the epidemic simulator on a multi-cloud infrastructure, relying on the PaaSage platform (see Section . . ). PaaSage receives as input CAMEL descriptions of the simulation services and of available cloud providers and generates a deployment on available cloud resources. Moreover, PaaSage dynamically adapts the deployment to react to environment changes, e.g., violations of performance objectives, or changes in cloud prices. It notably triggers elasticity actions (e.g., scaling out a component) based on ECA (Event Condition Action) rules de ned in CAMEL. neously exploit diverse VMs from multiple clouds.

Full evaluation results can be found in [ ].

  [ , , ]. Although these systems have exible architectures, which can support various application types, they provide no support for individual, customer-speci c SLOs. Moreover, when handling resource contention, these systems rely on priorities or quotas assigned by administrators. As a result, these systems produce e cient allocations only if administrators know the value of resources for all customers at all times or if customers truthfully declare such values. Finally, much research has proposed using markets for sharing the resources of clusters, grids, and clouds[ , , , , , ]. However, these systems did not target a private cloud and provided no adaptation support for meeting customer SLOs.is work was performed in the context of Stefania Costache's PhD thesis [ ] that I co-

supervised with Christine Morin (Inria) and Samuel Kortas (EDF R&D). e work made two main contributions

[ , , ,

  VM load balancing algorithm ComputePlacement (nodes, vms, newvms, N i te r , E max , M max ) for vm ∈ newvms do node ← least loaded node from nodes node.vms ← node.vms ∪ {vm} solution ol d ← nodes solution best ← nodes nIterations ← tabu_l ist ← ∅ // list of forbidden moves e wors e ← inf e = ComputeErrors(nodes) // based on Equation . for each VM and each resource (i.e., cpu and memory) while nIterations < N i te r and e wors e > E max do (vm, e max ) ← nd vm with e max = max

≤i≤n max{e i ,c pu , e i ,me m } source ← vm.node destination ← node which minimizes e max , (vm, node) ∈ tabu_l ist vm.node ← destination source.vms ← source.vms -{vm} destination.vms ← destination.vms ∪ {vm} tabu_l ist ← tabu_l ist ∪ {(vm, source)} e = ComputeErrors(nodes) e ′ max ← max ≤i≤n max{e i ,c pu , e i ,me m } nMigrations = calculate number of migrations required to reach the current solution from solution ol d if e ′ max < e wors e and nMigrations < M max then solution best ← nodes e wors e ← e ′ max nIterations ← else nIterations ← nIterations + for node ∈ solution ol d do for vm ∈ node.vms do if vm.node ≠ node and vm ∉ newvms then migrationPl an ← migrationPl an ∪ {(vm, vm.node)} for vm ∈ newvms do de ploymentPl an ← de ploymentPl an ∪ {(vm, vm.node)} return (migrationPl an, de ploymentPl an)

  information, and adaptation policy parameters, such as desired SLO (e.g., execution time limits). Based on this information, the Virtual Platform Manager creates the virtual platform and deploys the application controller, responsible for further application management based on the adaptation policies. e Virtual Platform Manager registers the virtual platform with the Virtual Currency Manager to allow charging the application for its resource usage. VMs are allocated using an IaaS Cloud Manager that provides interfaces to start, delete and migrate VMs, manage their storage and network, and keep information about infrastructure users. Merkat is implemented in Python and relies on the Twisted [ ] and ZeroMq [ ] libraries. It uses OpenNebula as the IaaS Cloud Manager [ ]; the default scheduler of Open-Nebula is replaced by Merkat's VM Scheduler.

Table . :

 . Comparison of Merkat, Maui, and Merkat's simulation in CloudSim.

  Dis chapter described a solution for application and resource management in private PaaS clouds. In this solution, the provider operates an auction that allocates ne-grained virtual resources to customers. Customers operate applications that autonomously adapt their resource demand in a decentralised manner, seeking to satisfy their QoS objectives. A straightforward way to continue this work is to support additional resource types (e.g., network and storage resources), application types, SLOs, and policies for meeting these SLOs, including policies that consider the costs of recon guration. A limitation of the work is that developing adaptation policies remains a complex and ad hoc process, mainly due to the price dynamicity and uncertainty. It would thus be interesting to provide improved support for developing such policies (e.g., based on the bidding strategies for EC spot pricing [ ]) or even to investigate alternative auction mechanisms that simplify the policy design space [ , Building so ware can be tedious and time-consuming, particularly when multiple development teams are collaborating, as was the case, for instance, with the PaaSage platform. Nevertheless, producing usable prototypes is invaluable. It enables broadening the impact of the work as well as obtaining helpful feedback from real users, as was the case, for instance, with the DiFFuSE framework. in the number of Internet-connected devices has triggered the emergence of diverse IoT (Internet of ings) applications and use cases, such as autonomous cars, smart cities, and video surveillance. ese applications impose stringent requirements on the underlying infrastructure in terms of available bandwidth and lowlatency computation. Traditional cloud infrastructures cannot meet these requirements as they rely on distant data centres accessible over large-scale networks, placing economic and physical limits on bandwidth and latency [ ]. is has led to growing industrial and research interest in fog computing, sometimes referred to as edge computing. Fog computing is an extension of the traditional cloud computing model in which compute, storage, and network capabilities are distributed closer to users along a cloud-to-thing continuum [ ].

	Conclusion
	decomposition then makes it easy to exploit cloud platform features, such as independent that posed by cloud systems; namely, allocating resources to dynamic workloads in line with F C e rapid growth
	service scaling. application QoS requirements and provider objectives.
	e nal strand focused on designing a framework for resource and application manage-We are working on resource management for high-performance embedded systems in the
	ment in private clouds. is research took the idea of applying economic reasoning a step further: it transformed the private cloud into an economy where applications sel shly at-context of Baptiste Goupille-Lescar's PhD thesis that I co-supervise with Eric Lenormand ( ales Research & Technology) and Christine Morin (Inria). e thesis is funded by a C tempt to achieve their own goals through participating in an auction. e main insight of CIFRE contract with ales Research & Technology. In this work, applications require real-
	this work was that the market-based approach is a natural and e ective way to coordinate time guarantees, and the provider objective is to increase the utilisation of the computing
	interactions among applications and the platform, promoting extensibility with respect to platform. Research challenges include the heterogeneity of the resources, and the dynamism
	application types and increasing total value. and variability of the workload combined with the real-time application requirements. To
	An important part of my research activities was developing working so ware systems. address those challenges, we are developing a QoS-aware resource management approach is nal chapter revisits some key ndings of my research and identi es directions for So ware development was useful not only for experimentally validating the results (typically that maintains a continually updated runtime model of the platform architecture and appli-future work. using Grid' deployments), but also for generating, testing, and improving solution ideas cations, allowing accurate predictions of future resource availabilities. Preliminary results
	. S e document has separated my research activities into three strands. show that the approach leads to reduced application latencies compared to basic solutions e rst strand fo-while allowing gradual performance degradation in overload scenarios [ , ]. in early development phases. . O F ---S (F S)
	Cloud computing is continually evolving to accommodate a widening range of application
	types, programming models, underlying infrastructures, economic actors, and pricing mod-
	]. els, posing new challenges and opening up exciting research opportunities [ ]. I plan to con-
	e Merkat prototype was deployed in an EDF R&D testbed, but we unfortunately ob-tinue investigating automated management solutions for domains beyond traditional clouds
	tained few data on its usability and acceptance by users. systems for private clouds, developed in the same period as Merkat or later, use traditional e most widely deployed PaaS methods to handle contention, such as priorities or quotas [ , , ]. Experience with ear-lier market-based resource management systems showed that users, when given the choice, through applying new or improved management approaches. In the following, I present e second strand focused on designing customer-side management systems, producing the research directions that I intend to pursue together with some ongoing work. ese di-methods and tools for adapting multi-cloud applications. ese tools were notably integrated rections fall into three categories: management for domains beyond traditional clouds, im-within the PaaSage open-source platform and exploited by several academic and industrial proved decision-making, and support for decentralised management. partners. e main insight of this work was that bene cial application management requires may prefer traditional methods to markets for simplicity reasons, even if this choice results in lower utility [ ]. It thus remains unclear whether markets are appropriate in private cloud considering recon guration costs as well as bene ts. To reason about recon gurations, it is useful to maintain abstract representations of the running system and generate explicit re-. . B T C velop an automated resource management solution for FaaS [ ]. is work is performed settings, where customers belong to the same organisation and simpler methods to solve con guration plans. e e ectiveness of applying economic analysis to such recon guration H -, -, Modern real-time embedded sys-in the context of Yasmina Bouizem's PhD thesis that I am co-supervising with Djawida Dib contention are feasible. Market-based approaches are certainly useful in completely decen-plans was also con rmed by the work. Another focus in this strand was facilitating the de-tems such as those monitoring and reacting to evolving physical environments (e.g., ground (University of Tlemcen) and Christine Morin (Inria). e provider objectives chosen for this tralised systems with no single locus of control, a point revisited in the next chapter. ployment of epidemic simulation applications, producing the DiFFuSE framework built on radar systems, surveillance drones) are increasingly expected to cope with dynamic work-solution are satisfying the performance and availability requirements of customers while re-
	PaaSage. e DiFFuSE framework was successfully used by epidemiology scientists at IN-loads, di cult to predict in advance. ese systems must execute dynamic combinations of ducing energy consumption. Challenges in developing this solution include implementing
	RAE to migrate legacy simulators to the cloud. An insight gained from this work was that applications, each having varying resource needs, while satisfying stringent timing require-fault-tolerance mechanisms, such as active replication, and taking into account the perfor-
	legacy simulators can be decomposed into separate services with limited code changes; this ments. e resource management problem posed by these systems becomes thus similar to mance, availability, and energy consumption objectives in a coordinated manner.

cused on designing provider-side management systems, producing solutions for SaaS and PaaS providers. e main insight of this work was that SLAs simplify decision-making because they link QoS levels (e.g., response time values) to economic consequences (prices, penalties). As a result, decision-making can be decomposed into predicting the QoS impact of di erent management actions, translating this impact into monetary terms, and performing an economic cost-bene t analysis. Another insight was that building an automated SLA-driven management system requires a exible architecture that supports a wide range of management actions (e.g., adding/removing resources, cancelling contracts) and enables timely response to environmental changes (e.g., uctuating workloads, failures). e FaaS model, the core element of serverless computing, is a recent addition to cloud technologies that enables customers to deploy functions on the provider's infrastructure without being concerned with provisioning or operating servers. e functions are executed on demand and customers are charged only for the amount of time that their functions are running. e bene ts of FaaS include fully automatic scaling, a general programming model, and reduced operational costs, leading to a rising popularity in industry. e model is supported by all major cloud providers as well as several open-source so ware platforms [ , , ]. e serverless computing model and FaaS are sparking new research into extending FaaS platforms to support additional application types (e.g., data analytics) and underlying infrastructures (e.g., edge devices) [ , ].

One limitation of current FaaS platforms is that they lack support for managing the QoS experienced by FaaS customers. To overcome this limitation, we have initiated work to de-

  ). For instance, typical auction mechanisms employ a central, trusted auctioneer that collects and processes all bids, creating scalability, reliability, and trust challenges. Addressing such challenges may require turning towards distributed auction mechanisms[ , ]. Finally, much work remains to be done in building practical implementations of such mechanisms and participating agents, and demonstrating their reliable and robust operation in real cloud/fog deployments. B . D. Abramson, R. Buyya, and J. Giddy. "A Computational Economy for Grid Computing and Its Implementation in the Nimrod-g Resource Broker". Future Gener. Com-A. P. Achilleos, K. Kritikos, A. Rossini, G. M. Kapitsaki, J. Domaschka, M. Orzechowski, D. Seybold, F. Griesinger, N. Nikolov, D. Romero, and G. A. Papadopoulos. " e Cloud Application Modelling and Execution Language". Journal of Cloud Computing : , , D. Agarwal and S. K. Prasad. "AzureBOT: A Framework for Bag-of-Tasks Applications on the Azure Cloud Platform". In: IEEE International Symposium on Parallel and Distributed Processing, Workshops and Phd Forum. , pp. -. . A. Almeida, F. Dantas, E. Cavalcante, and T. Batista. "A Branch-and-Bound Algorithm for Autonomic Adaptation of Multi-Cloud Applications". In: th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing. Amiri and L. Mohammad-Khanli. "Survey on Prediction Models of Applications for Resources Provisioning in Cloud". en. Journal of Network and Computer Applica-Apache Hadoop. : https://hadoop.apache.org/ (visited on March , ). . H. Arabnejad, C. Pahl, P. Jamshidi, and G. Estrada. "A Comparison of Reinforcement Learning Techniques for Fuzzy Cloud Auto-Scaling". In: th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID). H. Eriksson, M. Raciti, M. Basile, A. Cunsolo, A. Froberg, O. Lei er, J. Ekberg, and T. Timpka. "A Cloud-Based Simulation Architecture for Pandemic In uenza Simulation". In: AMIA Annual Symposium Proceedings. , pp. -. . E. Feller, L. Rilling, and C. Morin. "Snooze: A Scalable and Autonomic Virtual Machine Management Framework for Private Clouds". In: Proceedings of the th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (Ccgrid ). CCGRID ' . IEEE Computer Society, Washington, DC, USA, D. F. Ferguson, C. Nikolaou, J. Sairamesh, and Y. Yemini. "Economic Models for Allocating Resources in Computer Systems". In: Market Based Control of Distributed Systems. World Scienti c. Press, , pp. -. . N. Ferry, F. Chauvel, H. Song, A. Rossini, M. Lushpenko, and A. Solberg. "CloudMF: Model-Driven Management of Multi-Cloud Applications". ACM Trans. Internet Tech-A. L. Freitas, N. Parlavantzas, and J.-L. Pazat. "An Integrated Approach for Specifying and Enforcing SLAs for Cloud Services". In: Proceedings of the th International Confer-//dx.doi.org/10.1016/j.future.2013.10.005. . Gartner Forecasts Worldwide Public Cloud Revenue to Grow . % in . : https: //www.gartner.com/en/newsroom/press-releases/2019-04-02-gartnerforecasts -worldwide -public -cloud -revenue -to -g (visited on March , ). . Gartner Says Percent of Spending in Key IT Segments Will Shi to the Cloud by . : https://www.gartner.com/en/newsroom/press-releases/2018-09-18-gartner-says-28-percent-of-spending-in-key-IT-segments-willshift-to-the-cloud-by-2022 (visited on March , ). . F. Glover, M. Laguna, et al. Tabu Search. Vol. . Springer New York, . . Oracle Grid Engine. : https://www.oracle.com/technetwork/oem/gridengine-166852.html (visited on March , ). . PaaSage. : https://paasage.ercim.eu/ (visited on March , ). . B. Palanisamy, A. Singh, and L. Liu. "Cost-E ective Resource Provisioning for MapReduce in a Cloud". IEEE Transactions on Parallel and Distributed Systems : , , N. Parlavantzas, L. M. Pham, C. Morin, S. Arnoux, G. Beaunée, L. Qi, P. Gontier, and P. Ezanno. "A service-based framework for building and executing epidemic simulation applications in the cloud". Concurrency and Computation: Practice and Experience : , . : https://doi.org/10.1002/cpe.5554. . N. Parlavantzas, L. M. Pham, A. Sinha, and C. Morin. "Cost-E ective Recon guration for Multi-Cloud Applications". In: th Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP). IEEE, Cambridge, , L. M. Pham, N. Parlavantzas, C. Morin, S. Arnoux, L. Qi, P. Gontier, and P. Ezanno. "DiFFuSE, a Distributed Framework for Cloud-Based Epidemic Simulations: A Case Study in Modelling the Spread of Bovine Viral Diarrhea Virus". In: IEEE International Conference on Cloud Computing Technology and Science (CloudCom). IEEE, P. Rohou. D . . -Final Report. Research Report. PaaSage: Model Based Cloud Plat-

	put. Syst. : , 00085-7. p. . : : 10.1109/CCGrid.2014.25. , pp. -. -X. : 10.1186/s13677-019-0138-7. : -X. : 10.1016/S0167-739X(02) , pp. -. . Amazon EC Spot Instances. : https : / / aws . amazon . com / fr / ec2 / spot/ (visited on March , ). , pp. -. : -. : 10.1016/j.jnca.2017.01.016. . M. tions , , pp. . : ----. : 10.1109/CCGrid.2012.71. nol. : , , : -: . : -. : 10.1145/3125621. . Fission -Serverless Functions for Kubernetes. : https://fission.io/ (visited on -March , ). ence on Cloud Computing (CLOUD). IEEE, , pp. -. : ----. : 10.1109/CLOUD.2012.135. . A. G. Garćia, I. B. Espert, and V. H. Garćia. "SLA-Driven Dynamic Cloud Resource Management". Future Generation Computer Systems : , , pp. -. : -X. -. : -. : 10.1109/TPDS.2014.2320498. . Parallel Workloads Archive. : https://www.cse.huji.ac.il/labs/parallel/ workload/. pp. -. : ----. : 10.1109/PDP2018.2018.00088. Hong Kong, , pp. -. : ----. : 10.1109/CloudCom. : http:pp. 2017.41.
				: ,	, pp. -.	:
		-	.	: 10.1002/spe.2311.
		form Upperware,	.
		. RUBiS.		: https : / / projects . ow2 . org / view / rubis/ (visited on March ,
		).		, pp. -
	.	: 10.1109/CCGRID.2017.15.

. . . Apache Brooklyn. : https : / / brooklyn . apache . org/ (visited on March , ). . . . . FogCity -QoS-Aware Resource Management for Smart Cities. : https://team. inria.fr/myriads/projects/fogcity/ (visited on March , ). . . . . R. C. Price, W. Pettey, T. Freeman, K. Keahey, M. Leecaster, M. Samore, J. Tobias, and J. C. Facelli. "SaTScan on a Cloud: On-Demand Large Scale Spatial Analysis of Epidemics". In: Online Journal of Public Health Informatics; ( ): Ojphi.V i . . . . C. Quinton, D. Romero, and L. Duchien. "SALOON: A Platform for Selecting and Con guring Cloud Environments". So w. Pract. Exper.

.

Algorithm Optimisation Policy

Require: Request req to host a new application Ensure: Resources for req are allocated R re q ← resources required by req R ← available resources in current group if (R re q ≤ R) then Get R re q from current group Exit Get R from current group R mi ss in g ← R re q -R R av ai l ab l e ← available resources in other groups if (R mi ss in g ≤ R av ai l ab l e ) then Get R mi ss in g from corresponding groups Exit Get R av ai l ab l e from corresponding groups R mi ss in g ← R mi ss in g -R av ai l ab l e Collect donating bids from all groups and select group G i with lowest bid, called donating_bid, that satis es QoS violation thresholds Collect waiting bids from all groups and select group G j with lowest bid, called waiting_bid, and wait time wait that satis es QoS violation thresholds if (donating_bid < waiting_bid) then req pr iv ate_cost ← cost(hosting req on R re q private resources) + donating_bid private_option ← donating else req pr iv ate_cost ← cost(hosting req on R re q private resources) + waiting_bid private_option ← waiting req pub l i c_cost ← cost(hosting req on (R + R av ai l ab l e )private and R mi ss in g public resources) if (req pr iv ate_cost > req pub l i c_cost ) then

Rent R mi ss in g from corresponding public cloud Exit if (private_option = waiting) then if (req supports running with partial amount of resources) then Start running the new application with available resources Wait for wait time Get R mi ss in g from G j else Get R mi ss in g from G i System Virtualization Hypervisor

• One to multiple OSes on a physical node thanks to a hypervisor (an operating system of OSes)

Virtual Machines (VMs)

Physical Machine (PM)

Virtual Machine Monitor

"A virtual machine (VM) provides a faithful implementation of a physical processor!s hardware running in a protected and isolated environment.

Virtual machines are created by a software layer called the virtual machine monitor (VMM) that runs as a privileged task on a physical processor."

3 Thursday, June 30, 2011

System Virtualization Hypervisor

• One to multiple OSes on a physical node thanks to a hypervisor (an operating system of OSes)

Virtual Machines (VMs)

Physical Machine (PM)

Virtual Machine Monitor

"A virtual machine (VM) provides a faithful implementation of a physical processor!s hardware running in a protected and isolated environment.

Virtual machines are created by a software layer called the virtual machine monitor (VMM) that runs as a privileged task on a physical processor." relies on a market-based mechanism, namely an auction. We selected the proportional-share auction [ ] in which a bidder submits a bid for a resource, receives an amount of resource proportional to this bid, and is charged exactly this bid. e main advantage of such auctions is their simplicity and scalability (the computational complexity of calculating the allocation grows linearly with the number of bids). Moreover, under some theoretical assumptions, proportional-share auctions have been shown to be optimal in terms of economic e ciency among all simple and scalable market mechanisms [ ].

Speci cally, in our approach, the application controller submits bids on CPU and memory resources for the application VMs. Based on these bids, the PaaS system allocates an amount of CPU and memory to each running VM and charges the customer according to the auction rules. Resource allocations are ne-grained (i.e., % of one core and MB of RAM)

and vary dynamically according to the total resource demand. Payments rely on a virtual currency. When a customer registers with the system, the administrator assigns a number of credits to the customer's account. When the customer submits an application, the customer allocates an application budget to the application. e customer can set the application budget to be renewed automatically to avoid cases in which the application budget is depleted in the middle of the application's execution. Customer payments are redistributed to customers' accounts over a typically long time period.

.