Marc-Elian Bégin

Christophe Blanchet

Kathryn Cassidy

Evangelos Floros

Javier Fontan

Eduardo Huedo

Stuart Kenny

Ignacio M Llorente

Charles Loomis

Louise Merifield

Sixsq Wp3

Tcd Wp3

Vangelis Floros

Grnet Wp3

Ucm Wp4

Ucm Wp6

Tcd Wp5

Ucm Wp3

Sixsq Wp4

Enhancing Grid Infrastructures with Virtualization and Cloud Technologies

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

1 Publishable Summary

Project Context and Objectives

The StratusLab project is aimed at service provisioning, networking, and research of technologies that will bridge cloud and grid infrastructures to simplify and optimize the use and operation of existing distributed computing infrastructures (e.g. European Grid Infrastructure) and to provide a more flexible, dynamic computing environment for scientists.

The European production grid infrastructure has had many notable successes. It has allowed scientists from all over Europe and indeed from all over the world to federate their computing resources to advance their scientific aims. More importantly, the infrastructure allows them to federate their data and expertise to accomplish more than they would be able to do singlehandedly. Common APIs and service interfaces make it possible to take advantage of these distributed resources without having to modify applications for each site.

Despite its success, the grid also has its limitations. The uniformity of service interfaces unfortunately does not extend to the underlying computing resources, where users are exposed to significant heterogeneities in the computing environment, complicating applications and increasing failure rates. Passive calculations are handled well by the grid, but many applications require active services to coordinate the distributed analyses. Either scientists must provide their own resources for such services or negotiate with a particular site to provide them. This reduces the speed at which new calculations can be done.

Virtualization technologies provide a mechanism for offering customized, uniform environments for users with negligible performance degradation. Using grid technologies combined with virtualization allows the grid to provide users with a homogeneous computing environment, simplifying applications and reducing failures. Emerging cloud technologies allow users to dynamically allocate computing resources (often in less than a minute) and to specify the characteristics for the allocated resources. The fusion of cloud and grid technologies provides a more dynamic and flexible computing environment for grid application developers.

Cloud and virtualization technologies also offer other benefits to administrators of resource centers, such as the migration of live services for load balancing or the deployment of redundant servers. Reduced costs for managing resources immediately benefit users by freeing money for additional computing resources or by having better user support from administrators.

A combined computing infrastructure that uses grid technology's strengths for federating resources, virtualization's strengths in providing custom, uniform environments, and the cloud's strengths in dynamic resource allocation, maximizes the utility of European distributed computing resources to scientists.

The StratusLab project creates an complete, coherent, open-source private cloud distribution to allow administrators of grid resources centers to take advantage of virtualization and cloud technologies. It provides new ways of using existing distributed computing resources to make the infrastructure more adaptable and more useful for scientists.

Summary of Work Performed and Achievements

In the fourth quarter, two beta releases of the StratusLab cloud distribution were made and deployed on the project's reference cloud infrastructure. The v1.0 production release is expected just after the close of Q4 due to the scheduling of the project's development sprints. The project continues to operate a production grid service over the StratusLab cloud distribution, demonstrating its functionality and stability. The project has also demonstrated the deployment of a grid site with the Claudia service manager with some autoscaling features. Detailed achievements are given below.

EGI User Forum

The project booked an exhibition booth at EGI User Forum 2011, which was held in Vilnius, Lithuania from 2011-04-11 to 2011-04-14. Seven posters were displayed covering the topics Agile Development, Bioinformatics, Grid-Cloud Integration, Marketplace, SlipStream Integration, Cloud Storage and Reference Infrastructure. Flyers were also printed and distributed as well as tshirts. The StratusLab tutorial video and other demonstration videos were played on a screen in the booth during the event. The StratusLab booth received the "Best Exhibition" award. A StratusLab Tutorial was also scheduled at the EGI User Forum. It was extremely well attended, with 54 participants in total, 24 of whom tried and succeeded in installing and running the StratusLab client.

Increased Contacts

The project has continued to increase the number of contacts with individuals, groups, and projects through presentations of StratusLab and via direct interaction. Notably in this quarter there have been discussions with the HPC Cloud (Amsterdam), NIIF (Budapest), and the Mantychore project concerning possible collaborations on development and deployment. Additionally, a collaboration with HEPiX Virtualization Working Group concerning the design of the Marketplace has taken place in this period.

MoUs More formally, Memoranda of Understanding have been signed with EGI and with the EDGI project, defining the goals, tasks, and timelines for collaboration between the projects. The negotiations of an MoU with VENUS-C have concluded, but the MoU is not yet signed. MoUs with EMI and IGE are still being negotiated.

Base and Customized Appliances

The project has increased the number of base (operating system) images that are made available in the Appliance Repository and Marketplace. All of these images have been updated to conform to the new contextualization recipe required by the latest StratusLab releases. In addition, two customized bioinformatics images have been created to validate the image creation procedure and to provide useful test VMs for this community.

Improved Build Processes The Scrum process keeps on improving, with more effective planning meetings and demos. In addition, a number of jobs have been added to the continuous integration server to test a wider range of services and a large number of defined use cases. The state of jobs in the Hudson integration server is now a discussion item in the daily stand-up. This follows a decision by WP4 to introduce a 'stop-the-line' culture, where all development and integration activities are stop as soon as a failed job occurs in Hudson. This has lead to improved quality of the StratusLab releases.

StratusLab Releases This quarter has seen two beta releases (v0.3 and v0.4) of the StratusLab distribution, leading up to the v1.0 release expected in mid-June. The release v0.4 introduced a change in the database used by OpenNebula, from SQLite to MySQL, which increases the performance and scalability of the system. Migration scripts were created that allowed existing StratusLab deployments to be migrated without loss of information. The underlying operating system was also changed from CentOS 5.5 to Fedora 14 to take advantage of newer features in the KVM hypervisor and to avoid needing to work around limitations stemming from the older kernel in CentOS 5.5.

Evaluation of Current Release

The project has evaluated the current release in terms of the initial requirements identified by the project and also with respect to the requirements identified in the EGI VIrtualization Workshop. This has identified gaps in functionality and important points to be included in the Y2 roadmap.

Marketplace This quarter has seen the continued design and implementation of the Marketplace, a registry for shared machine and disk images. A reference implementation of the Marketplace has been made available for use in testing and for development of the client tools. The Marketplace implementation has been extended to include new features such as SPARQL querying of the metadata, new search functionality, and browser-based upload of metadata files. A test deployment of the Marketplace has also been deployed to support tests by the HEPiX Virtualization Working Group.

Test of Parallel File Systems Operational experience has shown that NFS lacks the performance attributes required for the delivery of efficient cloud services. Ceph and GlusterFS have been investigated to see if they offer better scalability and performance. Unfortunately, neither provides an adequate replacement and NFS has been retained while further investigations continue.

Persistent Disk Service An important requirement coming from the users is the need for persistent storage. A prototype persistent disk service and been developed and integrated to satisfy this requirement. This prototype will need to evolve to become an efficient and scalable solution following the feedback from this prototype.

Grid and Cloud Services

The project has continued to provide a reference cloud infrastructure over which a production grid site within EGI is being run. Experience garnered from this has lead to the production of a technical note available from the project's web site providing feedback to StratusLab developers and recommendations to those running grid services over a cloud.

Grid Site Deployment via Claudia A grid site has been deployed in the test infrastructure by using Claudia, which has been fully integrated into the standard StratusLab distribution. Component scalability has been carried out considering virtual hardware resources (e.g. VM CPU) and some work in being done towards the scalability driven by Key Performance Indicators (KPI) such as the number of queued jobs.

OpenNebula Adaptations Work has been done in order to adapt OpenNebula to the typical operations of a grid site, like virtual resource placement heuristics, cloud-aware image management techniques, and management of cloud-aware networks. Other enhancements like improvements to the OpenNebula OCCI implementation and group/role support for authentication will be taken advantage of in future releases.

Exploitation and Sustainability Plan

The first Exploitation and Sustainability Plan was written and delivered in this quarter. This document outlines the plans for exploitation of the project's results by each of the StratusLab partners and provides a first plan on sustainability of the StratusLab distribution after the project ends in May 2012. This document will be updated and refined in the second year of the project.

The official release of v1.0 will occur early in Q5, followed by an updated architecture which will be used to define the roadmap for Y2 and release 2.0 of the StratusLab cloud distribution.

Final Results and Potential Impact and Use

Most scientific and engineering research requires significant computing resources. Distributed computing infrastructures have brought unprecedented computational power to a wide range of scientific domains. Although, these architectures and the related software tools have been considerably improved over the years, they exhibit several difficulties, mainly due to limitations of physical platforms, which discourage adoption of grid technologies. StratusLab has the potential to profoundly change existing grid infrastructures.

Improved Interdisciplinary Scientific Collaboration

Cloud technologies are expected to have significant impact, both immediate and long-term, in the way scientific research is carried out. Grid infrastructures have provided a remarkable advantage over the past years offering access to vast amount of computing power and storage space, and most importantly by offering a sustainable platform for scientific collaboration enabling the sharing of computing resources and scientific data. Cloud computing is expected to take this one step further by facilitating the easy deployment of customized grid infrastructures. These infrastructures are expected to have further positive impact on the way interdisciplinary scientific research is taking place.

StratusLab focuses on the provision of scientific infrastructures over cloud computing, investigating in particular the provision of customized Virtual Machine images. This customization will be done on the user side, which means that the user can have more immediate influence on the infrastructure itself. In this way the infrastructure will adapt to the user requirements and not vice-versa. By easing the management of grid sites and the configuration of hosting services we expect to attract a broader number of scientific communities and further facilitate their collaboration.

Impact on DCI Evolution

Currently, there is a big shift in all e-Infrastructure projects, and related efforts in Europe, to expand their activities in order to include cloud computing technologies. StratusLab will play a key role in this landscape by providing a focused environment for development, deployment and experimentation of cloud computing services.

The projects proposal reflects an evolutionary path from the existing largescale monolithic grid e-Infrastructures to novel, beyond the state-of-the-art, cloudbased, grid-enabled ones. Through its expected collaborations with other projects, StratusLab will disseminate its findings and drive direct impact on the way e-Infrastructure provision is currently done.

Improved Usability of DCI Platforms

Virtualization is the cornerstone of cloud computing and a key for achieving optimal usability of DCI platforms. Moreover, virtualized environments have the ability to adapt to different hardware platforms enabling a quick transition from one environment to another.

StratusLab operates such a virtualized platform on a variety of hardware environments. By offering customized machine images, users will be able to set-up an environment that better suits their application requirements. This will dramatically improve the current situation where current infrastructures are forced to offer a common configuration-a common denominator-that tries to do its best to satisfy many users with different runtime requirements. Another aspect where StratusLab will contribute is on power consumption efficiency (Green Computing) and the increase reliability by incorporating failover mechanisms using virtual machine snapshots and migration. In this first quarter, the primary objective was to prepare the foundations for a successful project. In more detail this involved:

• Deployment of collaborative software development tools,

• Starting dialog between StratusLab and targeted communities,

• Make the project visible to targeted communities and general public,

• Put in place the software development processes and policies,

• Define the initial architecture for the StratusLab software, and

• Deploy the initial project infrastructure.

Within this quarter all of these have been obtained providing a solid basis for the first public release of the StratusLab software in Q2 with additional features appearing rapidly afterwards.

Quarter 2

In the second quarter, the emphasis was on making the first public release of the StratusLab cloud distribution. Detailed objectives were:

• Increase project visibility particularly at the EGI Technical Forum,

• Initial public release of StratusLab cloud distribution,

• Reference infrastructure available to outside users,

• Support provided for release, and

• Initial design of advanced management services.

All of these objectives have been met, allowing the project to build a featurecomplete release during the next quarter.

Quarter 3

In the third quarter, the primary objective was to provide a feature-complete release and demonstrate its utility for running grid services. The detailed objectives were:

• Continued dialog with and support of targeted communities,

• Increasing visibility of project by targeted communities,

• Regular public releases concluding with functionally complete beta,

• Production grid site running over a stable StratusLab cloud, and

• Integration of the service manager into the distribution.

Nearly all of these objectives have been achieved with the StratusLab v0.3 release at the end of the quarter. One highlight is the certification of a production grid site running over the StratusLab distribution. The v0.3 release is nearly functionallycomplete, missing only a solution for storage. This will be developed early in Q4.

Quarter 4

In the fourth quarter, the preparation and release of the StratusLab 1.0 Cloud Distribution was the primary focus of the project. The detailed objectives were oriented around this goal:

• In depth evaluation of the StratusLab v1.0 distribution,

• Increasing visibility of project by targeted communities and evaluation of our dissemination strategy,

• Continued public, preview releases culminating with the StratusLab v1.0 distribution,

• Continued operation of reference infrastructure and production grid site, and

• Complete integration and use of the service manager in the v1.0 release.

The preparations for the StratusLab 1.0 are very advanced and it is expected that the 1.0 release will be made at the end of Sprint 14 that will conclude just after the close of Quarter 4. With this release, the focus will shift to improving the existing services and providing more advanced functionality like hybrid clouds and deployments of predefined systems (e.g. hadoop).

Review Recommendations

Not yet applicable.

Progress and Achievements

In

Base and Customized Appliances

The project has increased the number of base (operating system) images that are made available in the Appliance Repository and Marketplace. All of these images have been updated to conform to the new contextualization recipe required by the latest StratusLab releases. In addition, two customized bioinformatics images have been created to validate the image creation procedure and to provide useful test VMs for this community.

Improved Build Processes The Scrum process keeps on improving, with more effective planning meetings and demos. In addition, a number of jobs have been added to the continuous integration server to test a wider range of services and a large number of defined use cases. The state of jobs in the Hudson integration server is now a discussion item in the daily stand-up. This follows a decision by WP4 to introduce a 'stop-the-line' culture, where all development and integration activities are stop as soon as a failed job occurs in Hudson. This has lead to improved quality of the StratusLab releases.

StratusLab Releases This quarter has seen two beta releases (v0.3 and v0.4) of the StratusLab distribution, leading up to the v1.0 release expected in mid-June. The release v0.4 introduced a change in the database used by OpenNebula, from SQLite to MySQL, which increases the performance and scalability of the system. Migration scripts were created that allowed existing StratusLab deployments to be migrated without loss of information. The underlying operating system was also changed from CentOS 5.5 to Fedora 14 to take advantage of newer features in the KVM hypervisor and to avoid needing to work around limitations stemming from the older kernel in CentOS 5.5.

Evaluation of Current Release

The project has evaluated the current release in terms of the initial requirements identified by the project and also with respect to the requirements identified in the EGI VIrtualization Workshop. This has identified gaps in functionality and important points to be included in the Y2 roadmap.

Marketplace This quarter has seen the continued design and implementation of the Marketplace, a registry for shared machine and disk images. A reference implementation of the Marketplace has been made available for use in testing and for development of the client tools. The Marketplace implementation has been extended to include new features such as SPARQL querying of the metadata, new search functionality, and browser-based upload of metadata files. A test deployment of the Marketplace has also been deployed to support tests by the HEPiX Virtualization Working Group.

Test of Parallel File Systems Operational experience has shown that NFS lacks the performance attributes required for the delivery of efficient cloud services. Ceph and GlusterFS have been investigated to see if they offer better scalability and performance. Unfortunately, neither provides an adequate replacement and NFS has been retained while further investigations continue.

Persistent Disk Service An important requirement coming from the users is the need for persistent storage. A prototype persistent disk service and been developed and integrated to satisfy this requirement. This prototype will need to evolve to become an efficient and scalable solution following the feedback from this prototype.

Grid and Cloud Services

The project has continued to provide a reference cloud infrastructure over which a production grid site within EGI is being run. Experience garnered from this has lead to the production of a technical note available from the project's web site providing feedback to StratusLab developers and recommendations to those running grid services over a cloud.

Grid Site Deployment via Claudia A grid site has been deployed in the test infrastructure by using Claudia, which has been fully integrated into the standard StratusLab distribution. Component scalability has been carried out considering virtual hardware resources (e.g. VM CPU) and some work in being done towards the scalability driven by Key Performance Indicators (KPI) such as the number of queued jobs.

OpenNebula Adaptations Work has been done in order to adapt OpenNebula to the typical operations of a grid site, like virtual resource placement heuristics, cloud-aware image management techniques, and management of cloud-aware networks. Other enhancements like improvements to the OpenNebula OCCI implementation and group/role support for authentication will be taken advantage of in future releases.

Exploitation and Sustainability Plan

The first Exploitation and Sustainability Plan was written and delivered in this quarter. This document outlines the plans for exploitation of the project's results by each of the StratusLab partners and provides a first plan on sustainability of the StratusLab distribution after the project ends in May 2012. This document will be updated and refined in the second year of the project.

The official release of v1.0 will occur early in Q5, followed by an updated architecture which will be used to define the roadmap for Y2 and release 2.0 of the StratusLab cloud distribution.

WP2: Interaction with Targeted Communities

The WP2 activity has continued to expand the number of people, groups, and projects they have contacted about the StratusLab distribution. There is significant interest in evaluating and using project's software. However after the release of the StratusLab 1.0 distribution, the activity will have to make a more concerted and systematic effort in getting feedback from the contacted people, groups, and projects to ensure that the project continues to meet the requirements of our users. Similarly, the activity will need to more systematically review and improve the documentation with each release.

The activity has begun developing customized appliances, specifically two appliances for the bioinformatics community. The activity has also developed a prototype persistent disk service to satisfy a need expressed by many user communities. The appliances and the prototype service will need to evolve according to users' needs in upcoming releases of the StratusLab distribution.

Progress Towards Objectives by Task

Task 2.1: Interactions with Resource Providers and End-users

TID internal Cloud testbed A new testbed, using with open source technologies, is being created inside the TID educational infrastructure for cloud tools. The StratusLab distribution is going to be used to install and configure Claudia and OpenNebula software on this testbed. TID personnel are aware of the value of StratusLab due to some internal demonstrations and presentations.

Spanish Supercomputing Centers Claudia and OpenNebula have been installed inside the CESGA1 supercomputing center in Santiago (Spain) by using StratusLab distribution. In the coming weeks, it is planned to install them also in Calendula data center, the computation center in Castilla Leon region2 . Although this work is part of the NUBA project, this activity is providing visibility to StratusLab. In the NUBA consortium there are data center partners (CESGA and Calendula), SMEs (XERIDIA and CATON), and large enterprises (ATOS and TID). It seems to be an important forum to disseminate StratusLab results.

Virtual Spain Claudia from the StratusLab distribution in being used in the nationally-funded Virtual Spain project (CENIT research programme), in which TID is collaborating.

Fiware TID was presenting Claudia and TCloud in the Fiware kickoff-meeting. TID pointed to StratusLab tools as the way to install and configure those assets.

Contacts with Other Projects

The project has made contact with other projects such as HPC Cloud (Amsterdam), NIIF (Budapest), and the Mantychore project. Initial discussions to collaborate on development, deployment, and standards took place which will need to be followed up in the following quarter. Some topics for collaboration are a user-level graphical user interface, storage implementations, efficient caching of VM images, and API standards such as OCCI, TCloud, and CDMI.

Appliances for the Bioinformatics Community CNRS IBCP has created two customized machine images for the bioinformatics community: "biological databases repository" and "bioinformatics compute node". The "biodata repo" VM aims to provide users with access from any cloud node to international reference databases recording biological resources such as protein or gene sequences and associated data, protein structures, or complete genomes. This appliance acts as an proxy between the internet where all the reference databases are published and the cloud internal virtual nodes that will compute the bioinformatics analyses. The "biocompute node" VM has pre-installed bioinformatics software such as ClustalW, BLAST, FastA and SSearch. Because these methods require access to reference data for processing, this appliance is linked via an NFS mount to the "biodata repo" appliance. This work has been reported in detail in the Milestone MS3 "Creation of Virtual Appliances for Bioinformatics Community". A poster has been presented at the EGI User Forum in Vilnius. Another poster has been submitted and accepted to the French annual Bioinformatics conference JOBIM 2011.

Persistent Disk Service An important requirement coming from the users is the need for persistent storage. The activity has created and integrated a prototype persistent disk service to satisfy this requirement. This prototype will need to evolve to become an efficient and scalable solution following the feedback from this prototype. EGI User Forum LAL played a leading role in the development of the user tutorial presented at the EGI User Forum in Vilnius, Lithuania. It was extremely well attended with 54 participants in total with 24 who tried to install the StratusLab client. All 24 succeeded in using the Reference Infrastructure provided by WP5. It was also instrumental in preparing the posters displayed in the project's booth.

Task 2.2: Intensive Evaluation of StratusLab Products

Claudia Documentation A wiki page has been created to document Claudia3 , including installation instructions, packages, and its usage.

Use Cases in Hudson An important contribution to the testing and stability of the StratusLab software has been the inclusion of jobs within the hudson continuous integration server that test specific use cases and benchmarks. These are executed automatically when changes to the code are made. Failures are followed up immediately by the integration activity to ensure that the project's software distribution continues to satisfy the primary use cases. The jobs test the complete VM lifecycle, the Marketplace, the Registration Service, and the benchmarks.

Manual Installation in Bioinformatics Laboratory CNRS IBCP has evaluated the manual installation of the StratusLab framework to deploy a cloud site devoted to Bioinformatics. Several issues have been identified and reported to the project's developers, who have corrected them. Manual installation is potentially the most realistic way of deploying the StratusLab system in bioinformatics laboratories that do not have system administrators who are experts with the Quattor system. The main goal of this evaluation was to fix bugs and to confirm the reliability of this procedure.

Evaluation of Reference Cloud with Bioinformatics Appliances CNRS IBCP has used their two virtual appliances, "Biological databases repository" and "Bioinformatics compute node" to evaluate the reference cloud deployed in GRNET. This work has been reported in details in the Milestone MS11 "Operation of Site Running StratusLab toolkit".

Evaluation of Current Release

The activity has evaluated the current release in terms of the initial requirements identified by the project (D2.1) and also with respect to the requirements identified in the EGI VIrtualization Workshop. This evaluation is contained in the deliverable D2.2.

Issues and Corrective Actions

More Followup The activity has made contact with a large number of individuals, groups, and projects over the course of the first year. However, the level of feedback and interactions has been moderate. With the release of the StratusLab 1.0 release, a more concerted effort will need to be made to follow up on these contacts and to solicit actively relevant feedback. The survey planned for Q5 will be a start, but more direct personal contact will be needed as well.

Improved Documentation

The activity has been rather lax in keeping the documentation on the web site consistent with the current release. Embedding jobs in hudson for the primary use cases will help flag changes needed in the documentation and tutorials. Nonetheless, more systematic review and update of the documentation needs to be done by the activity.

Use of Resources

The effort provided by the three institutes involved in the activity (CNRS/LAL, CNRS/IBCP, and TID) is consistent with expectations in Q4. There is still an overall deficit in effort over the first year of the project of 12% that will likely be compensated by additional effort and spending in the second year of the project. The effort has been sufficient to meet the objectives of the work plan.

WP3: Dissemination

The project continued to increase its visibility among European projects. The dissemination effort focused largely on the participation in the EGI User Forum 2011. Project partners continue to give talks to promote and explain StratusLab and participated in several workshops where the StratusLab project was presented during discussion sessions.

Collaboration activity has increased with Memoranda of Understanding (MoU) now signed with the EGI and EDGI projects. Other MoUs are being explored (e.g. EMI and IGE); an MoU will not be pursued with ERINA+ because of a large amount of additional effort required in StratusLab and an incompatibility with the timelines. Involvement with SIENA has also continued with StratusLab participating in the Cloudscape III meeting.

The Exploitation and Sustainability First Plan (D3.3) was written and delivered in this quarter.

Progress Towards Objectives by Task

Task 3.1: Dissemination

Release Dissemination A release dissemination plan, based on the plans for previous releases, was devised for version 0.4. A larger dissemination effort is planned for the upcoming release 1.0, and the release dissemination plan is being updated for this purpose.

OpenNebula StratusLab partner UCM were involved in the release and dissemination of OpenNebula 2.2, which is a major component of StratusLab from version 0.3.

Media & Publications

The StratusLab version 0.3 release was announced with a release annoucement in iSGTW 4 which ran on 15th March.

CNRS IBCP has submitted a poster to the French annual Bioinformatics conference JOBIM 2011 to be held in Institut Pasteur in Paris in June 2011. The subject is "Virtualisation of Bioinformatics Applications on Cloud Infrastructure". The submission introduces the virtual bioinformatics appliances that have been built by the partner CNRS and the benefit to bioinformatics scientists and engineers of using the cloud service from StratusLab. The poster has been accepted for presentation.

StratusLab also provided input to the SIENA white paper "SIENA European Roadmap on Grid and Cloud Standards for e-Science and Beyond Cloudscape III Use Cases and Position Papers".

Website A significant redesign of the project website [START_REF]2010 Phone Audioconference about monitoring and accounting in StratusLab[END_REF] was undertaken with a new version expected to go live shortly after the end of Q4, and before the release of version 1.0. The new version has a more appealing design and is easier to navigate. It also gives more visibility on the front page to the news items and the Twitter feed, in order to ensure that the main content is regularly updated and to increase repeat visits.

A section devoted to Bioinformatics has been created on the project web site. These pages introduce the current context in Bioinformatics and what advantages the scientific community could take from the StratusLab realizations in link with their science. The two bioinformatics appliances developed by CNRS IBCP are also described in terms of what they provide and how to use them on a StratusLab cloud such as the reference infrastructure in GRNET.

Figure 3.1 shows the number of visits to the website. The number for Q4 (4,579 visits) is similar to that for Q3 (4,623 visits).

Online, the project continued to use the Twitter account to announce development progress, releases and participation at events. The StratusLab Twitter feed now has 53 followers.

EGI User Forum 2011

The project booked an exhibition booth at EGI User Forum 2011, which was held in Vilnius, Lithuania from 2011-04-11 to 2011-04-14. Seven posters were displayed covering the topics Agile Development, Bioinformatics, Grid-Cloud Integration, Marketplace, SlipStream Integration, Cloud Storage and Reference Infrastructure. Flyers were also printed and distributed as well as t-shirts. The StratusLab tutorial video and other demonstration videos were played on a screen in the booth during the event. The StratusLab booth received the "Best Exhibition" award.

Talks were presented by Charles Loomis (LAL) and Vangelis Floros (GRNET) on "Operating Grid Services on the StratusLab Cloud" and "StratusLab Collaborations".

A StratusLab Tutorial was also scheduled at the EGI User Forum. It was extremely well attended, with 54 participants in total, 24 of whom tried and succeeded in installing and running the StratusLab client.

Workshops CNRS representatives (C. Blanchet and C. Loomis) have participated to the national workshop "RENABI GRISBI, Science and Technology Days" in Lyon (26 May 2011, 49 participants). CNRS IBCP has introduced the Stratus-Lab developments as a reliable perspective for the French Bioinformatics infrastructure RENABI GRISBI. CNRS LAL has presented the project, its goals and the current developments.

GRNET representative Vangelis Floros participated in the e-Infranet Cloud Computing Workshop on 29-30 March in Leuven, Belgium, presenting the Stra-tusLab's point of view in the workshop discussions.

The StratusLab Marketplace was presented at the HEPiX Workshop in Darmstadt, Germany which took place over the period 2-6 May.

Talks A brief list of talks describing the project delivered during Q4 is given in Table 3.1. Details and links, where available, are given on the project website6 .

Task 3.2: Collaboration with Standards Bodies and Related Projects

Memoranda of Understanding StratusLab has signed two Memoranda of Understanding to date, with the EGI and EDGI projects at the EGI User Forum in Vilnius, Lithuania in March. The MoU with EGI outlines a number of specific areas for collaboration including the development of cloud middleware components based on requirements gathered through the various scientific communities within EGI along with complementary training strategies, cooperation on workshops and technical meetings and development of standards.

The MoU with the EDGI project allows EDGI to take advantage of a cloud infrastructure with StratusLab providing expertise in building virtual appliances as well as supplying base images for EDGI to build their appliances, and grid appliances for their interoperability testing. StratusLab also provides a test infrastructure to EDGI and use of the Marketplace. The EDGI project implements an instantiation of some StratusLab use cases and thus will provide requirements, as well as feedback on their use of the StratusLab tools and infrastructure.

A draft MoU with VENUS-C has been defined covering collaboration in the areas of user requirements and experiences and on sharing expertise on standards, accounting, and OpenNebula (a core component of StratusLab). Furthermore, Stratuslab will investigate the use of the shared storage solution of VENUS-C while VENUS-C will investigate developing high-level APIs for some of the StratusLab benchmarks.

The project is continuing to explore the possibility of MoUs with other projects such as EMI, IGE and Cyfronet.

SIENA Vangelis Floros (GRNET) represented StratusLab at the Cloudscape III meeting organized by SIENA, in Brussels, Belgium on 15-16 March, and presented an overview of the project and its progress and success to date. Cloud service. Mantychore representatives were also present at this meeting and described their project and technology, which appears to have applications for Stra-tusLab.

EGI User Virtualization Workshop

EMI Informal discussions have continued with the EMI project, with a number of requests and queries coming from EMI. Two members of the EMI Virtualization group have now been given access to the StratusLab reference cloud service.

Task 3.3: Development of Exploitation and sustainability Plan

This task began in PM10. The initial work exploring possible avenues of exploitation and sustainability has been done and the deliverable D3.3 is due concomitantly with this report.

Issues and Corrective Actions

It was expected that a discussion forum would be in place by end of Year 1, and although the project has put in place a discussion mechanism on the website, this has not yet been widely disseminated and usage is low and difficult to measure. The project is revisiting the issue of whether this is the appropriate forum for such discussions. More detail is provided in StratusLab Deliverable 3.2 Report on Dissemination, Collaboration and Standardization Activities.

More comprehensive discussion functionality is expected to be launched along side the release of version 1.0.

Use of Resources

WP3 used 95% of expected effort in Q4 and so can be considered on-target in terms of effort, with all partners contributing. This is an increase over previous quarters and reflects the additional effort associated with Task 3.3, and with increased dissemination and collaboration in general.

WP4: Software Integration and Distribution

During Q4, WP4 continued integrating features resulting in the production of Stra-tusLab distributions: v0.3, v0.4 and the preparation for v1.0 (the actual release of v1.0 officially took place mid-June). We further improved the build, test and release procedures. We now have automated Hudson jobs for all major services and components of StratusLab.

The release v0.4 introduced a change in the database used by OpenNebula, from SQLite to MySQL. This required migrating existing SQLite databases to MySQL. Specific upgrade scripts were written, in tight collaboration between WP4 and WP5, such that existing StratusLab deployments can be migrated without loss of information.

To prepare for v0.4, as a candidate release for v1.0, SixSq hosted an integration face-to-face meeting in Geneva. This day and a half meeting was very productive, with all partners and work packages well represented. Several small working groups were spawned running in parallel, presenting and assessing every half day the progress done.

The Scrum process keeps on improving, with more effective planning meetings and demos. A notable improvement during Q4 is the addition of a definition of 'done' to each user story accepted for a sprint, and recorded in JIRA. This provides much clearer context for what is expected from each user story such that it can be considered completed. This 'done' definition also forms the basis of what is expected to be demoed during the demo meeting for each story.

The state of jobs in the Hudson integration server is now a discussion item in the daily stand-up. This follows a decision by WP4 to introduce a 'stop-theline' culture, where all development and integration activities are stop as soon as a failed job occurs in Hudson. The result was positive with much improved stability of the Hudson jobs in the second half of Q4 and higher confidence in the quality of StratusLab.

A major integration effort completed during Q4 was the transition from CentOS to Fedora 14 as the official operating system for the StratusLab distribution. This means that StratusLab now standardizes on an up-to-date operating system, taking advantages of recent developments, libraries and services. The key decision factor for this transition was the poor support for virtualization technologies from the old Linux kernels such as the one CentOS runs, compared to more recent distributions, such as Fedora 14.

As defined during the Scrum planning meetings, WP4 focused its integration effort on the following services:

• Marketplace • Registration service • Claudia • Automatic image creation
• Policy enforcement (e.g. quota, Marketplace images)

• Quarantine • Persistent Storage • On-demand Cluster creation • SlipStream / StratusLab integration
WP4 also worked on support for SuSE Linux virtual machines, the standard operating system used by the European Space Agency, which has expressed interest in evaluating StratusLab and SlipStream in the context of its operational software procurement.

Progress Towards Objectives by Task

Task 4.1: Definition of Reference Architecture

This task was not active during Q3. However, the architecture was better documented in the form of an 'architectural vision' with a new page on our online documentation server, including all current services. Work has also started in upgrading the current vision for the second year of the project to better guide our work towards StratusLab v2.0.

As part of this ongoing work, the StratusLab roadmap is available on our website to collect comments from our community.

Task 4.2: Integration of Open-source Distribution

SixSq, GRNET and LAL have continued to improve the Hudson continuous integration server. The Hudson infrastructure is now composed of over six machines (from GRNET and LAL). The LAL machines are managed by Quattor, which allows us to re-image the machines every day (at 02:00), from which an automatic installation, configuration and end-to-end test is conducted. While this was introduced during Q3, all these machines were upgraded to Fedora 14. We are now also taking advantage of StratusLab by integrating virtual machines managed by GRNET's reference infrastructure, as Hudson slaves. This is giving us increased flexibility since we can more easily duplicate and extend our build and test infrastructure.

Most services, such as the command-line client, the OpenNebula proxy, the Marketplace, the web monitor, registration application, persistence storage, include unit and system tests, which are automatically executed by Hudson after each code commit.

In order to get better support for virtualization from the Linux kernel, without having to upgrade default kernel (which is possible but a very intrusive operation which would invariably cause resistance from the system administers community, for good reasons), we decided to depart from our earlier CentOS operating system baseline. CentOS is an 'old' operating system. Since then, KVM (our default virtualization solution) has been integrated with the Linux kernel. To take advantage of this integration, a recent kernel is required. We therefore, decided to change our operating system baseline to Fedora 14, a more recent operating system that is also RPM-based. This switch also means that we do not need any longer to maintain our own packages for a number of dependencies that StratusLab requires, therefore streamlining its installation and maintenance.

From v0.4, StratusLab includes a new registration web application, which allows users to register with a StratusLab installation, using a clear workflow, confirming their identity via email and their acceptance of StratusLab usage policy.

The quarantine feature was also integrated with the distribution, including automated testing.

A long awaited OpenNebula feature was added at StratusLab's request to add error reporting to the client when failures occur during operations. This means that now users can be informed of the likely source of an error, such that they can take corrective actions, if errors are caused by the user. The command-line tools and web monitor were upgraded to take advantage of this feature.

StratusLab end-users can now attach three types of extra disks: volatile (created on the fly during instantiation), persistent (taking advantage of the persistent service) and read-only (from disk images registered with the Marketplace and Applicances Repostory). The command-line client options to handle extra disk was harmonised to reflect this.

In order to improve performance, StratusLab now ships with MySQL as the default database, replacing SQLite, OpenNebula's default database. This improves significantly the performance of OpenNebula, therefore StratusLab, when scaling deployments.

A computing cluster can now be instantiated using a single command (stratusrun-cluster), taking advantage of StratusLab's ability to deploy several virtual machines. This command was developed leveraging StratusLab's core Python client modules and API.

During this quarter, Claudia (the Service Manager component developed by TID) was further integrated in the project's continuous integration system, with the help of CNRS/LAL and SixSq. This included integrating authentication and connection to a proxy service, such that Claudia follows the same authentication strategy as all StratusLab services. In addition, more packages and configuration parameters have been included in the StratusLab distribution for feeding Claudia with monitoring information coming from OpenNebula and Ganglia. Finally, a significant effort was also put in developing Quattor profiles to install and configure Claudia automatically with Quattor.

Task 4.3: Contextualization of Grid Services

The generic contextualization mechanism devised during earlier quarters was further enhanced during this quarter. Integration work with SlipStream provided valuable feedback especially in order to be able to parameterize virtual machine instances. For example, it is now possible, via the standard StratusLab tools, to provide new VMs with scripts to be executed during boot and parameters to change new VM behavior.

The Appliances Repository and Marketplace now contain a richer set of validated virtual machine images.

Task 4.4: Technical Support

WP4 has provided support for the StratusLab distribution. This support continued to be provided via the daily standup meetings, phone calls, Skype and email. Further, as planned in the program of work, WP5 is taking over more of the first line support, where WP4 intervenes as second line support when and were required, coordinated with WP5.

WP4 has continued to support and manage the software development procedure based on Scrum. All main Scrum events are now routinely taking place (daily stand-up, demo and planning meetings), to which the large majority of partners are taking part regularly.

The user stories and tasks selected during the planning meetings include a balance of bug fixes and new features, such that current versions are maintained, while new versions are being prepared. An important improvement introduced during Sprint 14's planning meeting was to add to each user story an agreement of the definition of 'done'. This is important to ensure that functionality delivered by WP4 to WP5 is meeting expected quality criteria. This definition of done often includes the minimum requirements of having unit and/or system tests written, as well as being integrated with Hudson's continuous integration and test strategy.

Issues and Corrective Actions

Upgrade of the production system (reference architecture) is still laborious and taking longer that expected. This is in part due to the slow frequency at which StratusLab releases. Further, StratusLab's aggressive development, integration and test schedule means that a significant number of services and components have to be upgraded/installed by WP5 at every release. To mitigate this, the definition of done was improved and made more systematic during the planning meetings, such that WP5 is better able to estimate the maturity of the features integrated in each release. The next item of improvement would be to release more often, putting more effort in automating the upgrade of our production systems, thus reducing the size of the increments and time during which the service is off-line, and minimizing the risk of unexpected side-effects for our users.

Use of Resources

The effort involved in the activity is nominal for this quarter and the work is progressing according to the plan.

WP5: Infrastructure Operation

Work in WP5 progressed in Q4 with no diversions from the original planned work.

The infrastructure operations and support activity continued to serve an extended base of users and to provide a stable set of cloud and grid services. WP5 contributed significantly towards the release of stable version 1.0 expected at the beginning of Y2 of the project by providing input and requirements to the two main R&D activities of the project, namely WP4 and WP6. Highlights for Q4 include the upgrade of the cloud reference service to Stratus-Lab distribution v0.3, the operation of a stable pre-production Marketplace service, development of a Registration Service, and the compilation of the "Installing and operating a production grid site in the StratusLab cloud" Technical Report which provides recommendations for optimal interoperation of grid and cloud services. The activity also investigated parallel file systems such as Ceph and GlusterFS in an attempt to improve the scalability and performance of the Reference Infrastructure; neither solution was a marked improvement over the current one. Table 3.2 summarizes the services that currently offer web-based access in the context of WP5.

Progress Towards Objectives by Task

Task 5.1: Deployment and Operation of Virtualized Grid Sites

Production Cloud Service The reference cloud service in GRNET continued to operate on a production basis offering a stable testbed for both internal and external users. Internally the service is used for the provision of the project's production grid site and for various test and development activities. For external users, the reference cloud service offers a showcase of StratusLab distribution capabilities in a real environment.

The number of registered external users continues to grow as more people are Total number of VMs instantiated by the end of Q4 2432 becoming aware of the project activities and are interested to try out the developed cloud solutions. The reference cloud service also is a useful tool for collaboration with the other DCI projects and an integral instrument for the MoU's signed with them. The service is also used regularly for demonstrations and tutorials given by the project. For this purpose, a number of temporary demo accounts have been created which are active only for a limited period of time. During Q4 the service has been upgraded to v0.3 of StratusLab distribution. The base OS still remains CentOS 5.5 although this is planned for change in Q5 with the installation of v0.4.

As of Q4 the service offers three methods of user authorization:

1. Username/password generated internally by WP5 2. Digital certificate

VOMS proxy certificate

Username/password still remains the most popular choice due to its simplicity and fast learning curve, despite being less secure. The certificate-based solutions although more secure are still underutilized due to their complexity of usage but also because not everyone is accustomed to PKI technologies. Table 3.3 provides an overview of the service usage.

The number of physical resources has remained unchanged. Nevertheless due to the increasing demand from external users we are currently looking for ways to incorporate more physical nodes and thus expand the capacity of the hosting infrastructure.

Operation of a production Grid Service The operation of the production grid site HG-07-StratusLab continued in Q4 without any problems. The provision of grid service over virtualized resources does not seem to introduce any particular changes in the way they are administered. The underlying middleware has followed the evolution of gLite and the nodes have been kept up to to date with any major releases of the software or special requirements communicated by the EGI operations groups. The respective VM appliances that were used to setup the site are also kept up-to-date with the grid middleware releases and are regularly updated in the Appliance Repository.

The site still offers a minimal set of physical resources and therefore has not been heavily used by real grid applications so far. Nevertheless, the experience we have gained from its deployment and operation has been valuable and has led to compilation of the Installing and operating a production grid site in the StratusLab cloud Technical Report7 . This document summarizes our experience from the installation of the site over the reference cloud service, identifies a number of issues that prohibit optimal exploitation of the underlying cloud resources and offers a set of recommendation to alleviate them. The document has proved also extremely useful for the design of grid elasticity functionality that is currently being developed in the context of WP6.

The GStat page with all the details of the site as they are reported from the Site-BDII is available on the web8 .

Pre-production Services LAL has deployed an initial pre-production cloud service which is available to the laboratory's users and system administrators. This service will be opened to the wider community once registration procedures are in place and the StratusLab release provides a mechanism for enforcing site machine image policies.

Support Infrastructure The Hudson continuous integration service had to be relocated to a new node, within the data center, in order to perform required maintenance tasks to the previous hosting node (see "Issues and Corrective Actions" section for more details). Three servers have been allocated to WP4 for development testing. Additionally, two servers have been allocated to WP6 for the Claudia integration tasks. Two Quattor-controlled machines at LAL have been allocated for cloud deployment and functionality tests through the hudson continuous integration server.

Registration Service A condition from the CNRS-IN2P3 security officers for opening the LAL pre-production service to the public was the registration of users of the service. The registration must collect simple identification information (name, email address, etc.) but must also require that the user agree to the existing EGI Acceptable Use and Security Policies. To satisfy this requirement, a registration service was developed to handle this registration. The information is kept in an LDAP server which is compatible with the StratusLab authentication services, allowing access via username/password pairs and via grid certificates. This has been integrated as part of the standard StratusLab release.

Quattor Configuration LAL has created and maintains a Quattor configuration that allows the installation of a complete StratusLab cloud, similar to that created with the scripted installation with stratus-install. This configuration has evolved with the StratusLab distribution. The most difficult issue encountered has been the switch from CentOS 5.5 to Fedora 14 as the base operating system for the project. Fedora 14 requires many changes in the configuration modules for standard ser-vices and for the generated kickstart files. These are generic changes needed within Quattor to support more modern RedHat-based distributions, but the StratusLab effort is pushing this evolution forward rapidly. As for previous releases, the Stra-tusLab 1.0 release should have a complete, working Quattor configuration.

Task 5.2: Testing of the StratusLab Toolkit

Testing Infrastructure In addition to the development of the registration service, associated hudson jobs have been created that test the service as well as username/password and certificate authentication. As for all of the hudson test jobs, these are run automatically when the code base changes and a full re-installation and test is done daily.

Validation of StratusLab Releases Significant effort has been placed this quarter for testing and validation of StratusLab 0.4. For this purpose we relied heavily on the pre-production nodes residing in GRNET datacenter. This update is quite significant since among others requires the migration of the base OS for the physical nodes from CentOS 5.5 to Fedora 14. Our tests spotted a few problems and incompatibilities that where fixed promptly in coordination with WP4. In particular a new build of OpenNebula packages had to be performed on Fedora 14. Also the new version of libvirt that comes with Fedora 14 had to be fine tuned in order to allows seamless interoperability with StratusLab distribution.

Investigation of Storage Solutions

During the past months we have gathered important experience from the operation of the reference cloud service over a shared NFS file system. This experience showed that NFS lacks the performance attributes required for the delivery of efficient cloud services, or at least this is the case for the setup in GRNET where a centralized EMC storage server is sharing volumes to the rest of the nodes over NFSv3.

For this reason we decided to investigate alternative solutions for distributed and parallel file systems. In particular we chose to test Ceph9 and GlusterFS10 as two of the most popular offerings nowadays. Both file systems were installed in the pre-production service and were used for sharing the images of VM instances managed by OpenNebula (i.e. residing under /var/lib/one folder). Unfortunately, the results we got were not particularly encouraging. Ceph appears to be still very unstable and in our opinion can not be trusted yet for production-level services. GlusterFS on the other hand did not gave us the expected performance gain. Actually our tests showed that in most cases the system exhibits worse or similar performance with NFS v3. For this reason we decided to retain the existing setup based on NFS but to continue seeking more efficient shared storage solutions.

Task 5.3: Virtual Appliances Creation and Maintenance

Marketplace During this quarter the initial version of the appliance repository, using a WebDAV enabled Apache web server, has been maintained. The initial set of reference images has been updated as required. In collaboration with WP2 activity two Bioinformatics specific appliances have been created based on a CentOS 5.5 base image.

In this quarter the focus of the task has been on the continued design and implementation of the Marketplace. A reference implementation of the Marketplace has been made available for use in testing and for development of the client tools. The Marketplace implementation has been extended to include new features such as SPARQL querying of the metadata, new search functionality, and browser-based upload of metadata files. During the next quarter the existing appliance repository will be phased out, to be replaced by the Marketplace.

Hudson jobs have been created to test the deployment, configuration, and functionality of the Marketplace. These run regularly and flag any errors that have been introduced in the code or the deployment procedures.

In addition, a test server has been deployed for the HEPiX Virtualization Working Group. This will allow them to test how the Marketplace can help them to publish and to share virtual machine images between their grid sites. A concerted testing effort by the HEPiX people is expected to take place in the latter part of June 2011.

Creation of Standard Base Images

A new update contextualization recipe has been introduced and will be applied from v0.4 and forward. This requires all images to be re-configured or re-generated in order to be compatible with the cloud services build with StratusLab 0.4 and later. The largest part of this effort is managed by WP4 although members of WP5 have contributed with new versions for a number of appliances.

PXE Support Almost all operating systems provide a mechanism for automated installation via PXE. Reusing this mechanism to automate the production of virtual machine images offers the possibility to document the contents of a particular image (through kickstart or other description files) and to regenerate them easily to incorporate regular security updates of the operating system. LAL has adapted the mechanisms used in Quattor for the automated installation of machines via PXE to provide a similar mechanism for cloud users. In this case, cloud users provide the network boot file and the machine description to create the virtual machine. A prototype of this works, but needs to be integrated into the main StratusLab distribution and command line interface.

Issues and Corrective Actions

Quattor Support of Fedora 14 The switch from CentOS 5.5 to Fedora 14 has required more effort than foreseen, pulling effort away from other tasks like the testing of iSCSI as an image store. Despite the need for additional effort, it is expected that a full Quattor configuration will be available for deploying the Stra-tusLab 1.0 release.

Need for Storage Resources for LAL Pre-production Service Local users of the LAL pre-production service have already encountered limitations stemming from the lack of storage capacity. Foreseen funding has unfortunately not appeared, so storage hardware will likely be purchased with StratusLab funds instead. This will mean a reduction in the overall funded human effort, but believe that this will better serve the aims of the project and allow faster evolution of the StratusLab distribution.

Use of Resources

The level of resource consumption showed a slight underspending during Q4. As depicted in Table 4.1 the total manpower consumed was 24% less than anticipated. The main reason for this is the underspending that occurred within the leading WP5 partner (GRNET) that in turn has to do with the way payments are arranged by the partner (not following a string one-month cycle). Thus the level of resource usage is expected to go back to slightly overspending in Q5 is it was the case for Q3.

WP6: Innovative Cloud-like Management of Grid Services and Resources

WP6 develops advanced technology/features for deployment on existing cloud infrastructures through automatic deployment and dynamic provision of grid services as well as scalable cloud-like management of grid site resources. During this period, a grid site has been deployed in the testbed by using Claudia. Component scalability has been carried out considering virtual hardware resources (e.g. VM CPU) and some work in being done towards the scalability driven by Key Performance Indicators (KPI). On the other hand, some work has been done in order to adapt OpenNebula to the typical operations of a grid site (like virtual resource placement heuristics, cloud-aware image management techniques and management of cloud-aware networks).

Progress Towards Objectives by Task

T6.1: Dynamic Provision of Grid Services

Grid Site Specification Formalized in OVF The grid site features has been defined and specified by the OVF format. The OVF is a DMTF standard which provides a portable packaging mechanism to specify the service requirements to be deployed in the cloud. To this end, OVF has been used to define the VMs involved in a grid site (CE, SE, WN, APEL), their hardware resources (CPU, RAM, etc.), and the software installed in each VM together their configuration parameters. In addition, some OVF extensions has been included for scalability issues (to define the scalability rules, the KPI which drives the scalability, and the number of replicas which can scale).

Grid Site Deployment by Using Claudia The grid site, specified in the OVF format, has been deployed in the cloud by using the service manager (Claudia), using the deployment functionality that Claudia provides (following the TCloud specification). Claudia processes the OVF and performs requests via the virtual machine manager (OpenNebula) to deploy networks and virtual machines. The configuration information arrives at the VM by using the contextualization mechanism. By using scripts and contextualization information, all VMs execute the YAIM tool against a set of inferred configuration files. As a result, all the VMs from the grid site have been deployed in the cloud are interconnected by a public network, configured correctly and the grid site is up and running.

Grid Site Contextualization

The OVF language can provide a mechanism to provide the contextualization information for the software installed in the VMs. In the OVF, it is possible to specify configuration parameters, certificates needed and so on. The generated file by Claudia (ovf-env.xml) is in the deployed VM. A script, also passed by contextualization, is in charge of configuring each VM.

Grid Site Scalability Grid applications deployed over cloud technologies can benefit from scalability at the service level that Claudia provides; this conceals low level details from the user. This means that the number of WNs can vary depending on the virtual hardware resources (like CPU) and the KPI like job queue utilization.

Probe Development A typical approach to apply grid site elasticity is to vary the number of WNs in the site depending on the job workload at a given moment. This requires the constant monitoring of the number of jobs in the local LRMS queues. During Q4 we experimented with a basic probe module developed in Python that calls the pbsnodes and qstat command line tools of Torque LRMS (which has been installed in HG-07-StratusLab), parses the output and generates the percentage of job queue utilization. This module is currently being evolved to act as a proper demon service that will communicate the the above mentioned KPI value (queue utilization) to the Service Manager using the its REST API.

Load Balancer Support The CE of a grid site has to be aware of all of the WNs to be deployed and un-deployed on the site. To achieve this, close cooperation between the service manager and the load balancer (LB) component that is running as a service on the CE is requried. LB is running constantly waiting for notifications from Claudia. Work on the LB has started in the end of Q4 and is expected to complete in the first weeks of Q5. The LB service is developed with Python using the web.py11 module.

Static IP Support by Claudia A requirement taken from the grid site is the need for static IP addresses. Some VMs in the grid site require a digital certificate which corresponds to a concrete IP address. This certificate has a duration of a year for the duration of which we have to keep the associated IP addresses. Due to this, Claudia has evolved to manage the static IP for customers and specify a way that the user can utilize it to configure the grid site in the OVF.

Feeding Claudia with Virtual Hardware Monitoring Information OpenNebula provides Virtual Machine hardware monitoring information obtained from Ganglia. Claudia needs to obtain this virtual hardware information to scale grid services. Thus, this monitoring information collected by OpenNebula has to be provided to the Claudia optimization module.

T6.2: Scalable and Elastic Management of Grid Site Infrastructure

Development of Image Repository for Image Management in OpenNebula

The Image Repository system allows OpenNebula administrators and users to set up images, which can be operative systems or data, to be used in VMs easily. These images can be used by several VMs simultaneously, and also shared with other users. Users can manage the image repository from the command line interface with the oneimage command.

Support for Multiple Storage Backends to Access Persistent Images in the Image Repository VM disk images can be provisioned using two approaches: block devices and files. The image repository has been architected to support these two approaches and to easily incorporate different technologies in each area.

Evaluation of Additional VLAN Models for Virtual Network Management

We are evaluating different alternatives for virtual network management. For example, Open vSwitch12 or host-managed 802.1Q VLANs. In this case, the Open-Nebula network manager creates bridges and tagged interfaces for VMs as needed, when the VM is booted; and remove them upon VM disposal.

Automatic Setup of Simple TCP/UDP Firewall Rules for VMs Each network interface defined in a VM can include simple filter rules based on destination ports. This is implemented by dynamically creating iptables rules that captures the FOR-WARDING packets and setting up custom iptables chains for the VM interfaces.

Using Virtual Network Information and Image Information in VM Contextualization Variables from the image template and virtual network template can now be used in the CONTEXT parameter of the VM description file to provide contextualization information to VMs.

T6.3: Cloud-like Interfaces Specific for the Scientific Community

TCloud as the Claudia API The TCloud API is a RESTful API which allows access to the service manager. The TCloud API has been implemented and released by the tcloud-server project, which is a general TCloud API representation and which is bound to an implementation by a set of drivers, for instance, the OpenNebula driver. The tcloud-server plus the drivers are provided as an RPM package tcloud-server-rpm, which is configured and installed by using the StratusLab sysadmin tools.

Enhancements in OpenNebula OCCI Implementation The OCCI interface has been enhanced to expose more OpenNebula functionality for image management, virtual network management, as well as for fine grained resource specification.

Authorization Based on Groups and Roles in OpenNebula

The authorization system in OpenNebula is being extended to support groups of users and access rules (roles) to manage OpenNebula resources. These groups and roles can be used, for example, to map attributes specified in VOMS certificates.

Issues and Corrective Actions

No major issues related to WP6 have arisen in Q4.

Use of Resources

The effort spent in Q4 has been a 36% higher than expected in order to finalize all the commitments specified in the Dow for year 1 developing all the required functionalities for the release 1. With this increase, WP6 effort in the first year is consistent with expectations in year 1. These are the spent effort figures in personmonths by the partners involved in WP6: UCM 6.92, GRNET 1.00, TID 5.00. This adds up to a total 12.92 person-months.

Project Management

Consortium

The project consortium consisting of six partners (CNRS, UCM, GRNET, SIXSQ, TID, and TCD) has not changed since the start of the project. There have been no changes in the legal status of those partners. The representatives for TCD and TID have changed because of retirements and internal reorganization of activities.

The effort consumed by partner and by work package are shown in Table 4.1. See the "Issues" section for a discussion of the lower expended effort than expected. Effort numbers for UCM include unfunded effort and hence will be larger than claimed in the Form C for reimbursement from the EC.

Management Tasks

Meetings Tables 4.2, 4.3, 4.4, and 4.5 contain a list of the meetings by quarter that have been planned to foster collaboration between the project participants. Not listed are the planning meetings for each development sprint and the daily standup meetings.

Metrics Table 4.6 contains the metrics for the project. The table groups related metrics together. The first group aimed towards dissemination show steady interest in the project; an open question is how to encourage and manage discussion with the community. The second group concerns the integration processes; all of the metrics show good progress, which is reflected in the regular releases of the distribution. The third group concerns the operations and deployments. The metrics show that the quality of the software is good. However, more effort needs to be made by the project in having external users deploy and use the distribution in production. The release of the StratusLab v1.0 release will help improve these metrics. In the fourth group, the resources provided by StratusLab are steady. Storage services have just been added in the v1.0 release, so related metrics can now be collected. The last group shows that the maintained appliances and the Marketplace are well used. Further growth in these metrics are expected in Y2.

Deliverables and Milestones

Issues

Underused Effort The effort numbers for Q4 are very near to the expected values except for WP1 (management) and WP5. The effort for WP1 is about 50% of what was foreseen because the project has been less difficult to manage than expected, although with the year end deliverables and first review the numbers are likely to increase in Q5%. The deficit in WP5 is largely an artifact of GRNET's internal accounting schedule; the actual effort is expected to be compensated in Q5.

Planning

Objectives for Next Quarter

• Solidify the v1.0 StratusLab cloud distribution through increased testing and hardening of existing services.

• Support for a second operating system to ensure the portability of the distribution.

• Survey of the users and system administrators to see if the requirements have evolved from those already collected in Y1.

• Update and expand the target reference architecture for the distribution.

• Continued dissemination of project results.

• Continued operation of reference infrastructure and support to users and system administrators.

• Expansion of the number of users and sites using StratusLab.

Roadmap

The roadmap remains essentially the same as decided in the Lyon Face-to-Face meeting. The PMB in Q3 gave its formal approval of the following changes to the overall work program:

1. The tasks regarding having a public (user-visible) cloud and an associated cloud API have been moved from Y2 to Y1, largely because of interest from scientific communities and resource centers wanting to provide public clouds.

2. The tasks about hybrid clouds will be expanded to include also cloud federation models. This will be moved to Y2 to balance the change above. Also having a solid release will make these investigations easier.

3. As foreseen in the TA, the appliance repository consists of a single service that contains appliance metadata, appliance storage, and services for changing appliance formats. This has been split into different services. The Marketplace will handle appliance metadata. Storage will take place with normal cloud storage or outside of the cloud. Instead of providing a service for appliance format changes, client tools will be provided instead.

These changes have been made and followed at the technical level for sometime; they are now also agreed at the management level. The architecture and roadmap will be re-evaluated early in Q5 to define the detailed work plan for the second year of the project.

WP

Figure 3 . 1 :

 31 Figure 3.1: Visits for Q4.

  StratusLab members from CNRS, GRNET, TCD, UCM participated in the EGI User Virtualization workshop in Amsterdam on 12-13 May to map out the future of EGI as a federated virtualized infrastructure.SARA HPC Cloud StratusLab members from CNRS, GRNET, TCD, UCM met representatives of the SARA HPC Cloud project in Amsterdam on 11 May to see how StratusLab could work with SARA on future developments of their HPC

  FLoros, GRNET) at the EGI User Forum, Vilnius, Lithuania 2011-04-11 "StratusLab Tutorial" at the EGI User Forum, Vilnius, Lithuania 2011-04-12 "StratusLab Collaborations" at the ECEE Workshop during the EGI User Forum, Vilnius, Lithuania 2011-04-13 "Sharing Virtual Appliances with the StratusLab Marketplace" at the HEPiX Workshop in Darmstadt, Germany 2011-05-05 "Panel Discussion" sharing knowledge on StratusLab's experience in applying agile (and Scrum) in academic context and FP7 contracts (Marc-Elian Bgin, SixSq) at XP2011, Madrid, Spain 2011-05-11 "StratusLab use cases" at EGI User Virtualisation Workshop, Amsterdam, Netherlands 2011-05-12 "Cloud Computing & StratusLab" at Journée Grille de production EGI, Lille, France 2011-05-17 "StratusLab: Enhancing Grid Infrastructures with Cloud and Virtualization Technologies" at the TERENA Conference in Prague, Czech Republic 2011-05-19 "StratusLab: Enhancing Grid Infrastructures with Cloud and Virtualization Technologies" at Journes GRISBI in Lyon, France 2011-05-26 "Grid-Ireland Operations Centre / TCD presentation" featuring StratusLab at Lero Cloud Computing seminar Cork, Ireland 2011-05-26

  Tables 5.1, 5.2, and 5.3 list all of the documents. In addition, these are available from the project website. Milestones MS8 and MS11 as well as Deliverables D6.2, D2.2, D3.2, D3.3, D4.2, D4.3, D5.3, and D6.3 have been produced in this quarter.

Table 1 . 1 :

 11 StratusLab Information and SupportMore information about the StratusLab project can be obtained from the sources listed in Table1.1. Individual partners can also be contacted to obtain more specific information about their contributions to the project. Table1.2 contains the list of StratusLab partners and relevant contacts.

	Website	http://stratuslab.eu/
	RSS Feed feed://stratuslab.eu/feed.php?ns=news&linkto=page
	Twitter	@StratusLab
	YouTube http://www.youtube.com/user/StratusLab
	Support	support@stratuslab.eu
	1.4 Contact Information

2 Project Objectives for the Period 2.1 Objectives 2.1.1 Quarter 1

Table 3 .1: Talks

 3

	Title / Event

Table 3 . 2 :

 32 WP5 Infrastructure Services

	Reference Cloud Service	GRNET https://cloud-grnet.stratuslab.eu
	Pre-production Infrastructure	GRNET https://62.217.120.158
	Pre-production Infrastructure	LAL	https://onehost-4.lal.in2p3.fr
	Project Tools (Hudson Server) GRNET http://hudson.stratuslab.eu:8080
	Appliance Repository	TCD	http://appliances.stratuslab.eu
	App. Repository Mirror	GRNET http://appmirror-grnet.stratus.eu/images
	App. Marketplace	TCD	http://appliances.stratuslab.eu/marketplace/
	Test Infrastructure	LAL	https://onehost-2.lal.in2p3.fr

Table 3 . 3 :

 33 Reference Cloud Service Usage Statistics

	Number of external users	26
	Number of external projects supported	8
	Number of countries represented	10

Table 4 . 1 :

 41 Effort (in Person-Months) by Partner and by Work Package Memoranda of Understanding The project has signed Memoranda of Understanding (MoU) with the EGI and EDGI projects; an MoU with VENUS-C has been concluded but not yet signed. The project has decided not to pursue an MoU with the ERINA+ project because of the large additional effort required by Stra-tusLab and incompatibilities with the timelines. MoUs with EMI and IGE are still under negotiation.

	Partner/	Q1		Q2		Q3		Q4			TOTAL
	WP	Actual Exp. Actual Exp. Actual Exp. Actual Exp. Actual Exp. Diff. (%)
	CNRS	6.84 10.13	6.61 10.13	7.49 10.13	9.70 10.13	30.64 40.50	-24
	UCM	4.40	5.13	5.50	9.41	6.93	9.41	12.31	9.41	29.14 33.36	-13
	GRNET	1.75	6.75	6.68	7.46	7.40	7.46	4.86	7.46	20.69 29.14	-29
	SIXSQ	5.10	6.38	3.29	6.38	4.14	6.38	6.63	6.38	19.16 25.50	-25
	TID	0.10	3.00	2.70	7.29	7.30	7.29	7.90	7.29	18.00 24.86	-28
	TCD	1.50	3.00	3.50	3.00	3.30	3.00	3.24	3.00	11.54 12.00	-4
	WP1	0.98	1.50	0.93	1.50	0.55	1.50	0.55	1.50	3.01	6.00	-50
	WP2	5.38	6.00	4.46	6.00	4.96	6.00	6.25	6.00	21.05 24.00	-12
	WP3	0.70	4.38	3.92	4.38	2.94	4.38	4.14	4.38	11.70 17.50	-33
	WP4	9.00 12.00	5.29 12.00	8.03 12.00	12.21 12.00	34.53 48.00	-28
	WP5	3.63 10.50	8.43 10.50	11.28 10.50	8.57 10.50	31.91 42.00	-24
	WP6	0.00	0.00	5.25	9.29	8.80	9.29	12.92	9.29	26.97 27.86	-3

Table 5 .3: Milestones

 5

	Lead Bene-	Diss. Due	Actual

http://www.cesga.es/index.php?lang=en

http://www.fcsc.es/index.php?option=com content&view=frontpage&Itemid=1&lang=es

http://stratuslab.eu/doku.php/claudia

http://www.isgtw.org/

http://www.stratuslab.eu

http://stratuslab.eu/doku.php/presentations

http://stratuslab.eu/lib/exe/fetch.php/documents:gridovercloud-v1.0.pdf

http://gstat-prod.cern.ch/gstat/site/HG-07-StratusLab/

http://ceph.newdream.net/

http://www.gluster.org/

http://webpy.org/

http://openvswitch.org

StratusLab is co-funded by the European Community's Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI-261552.

The information contained in this document represents the views of the copyright holders as of the date such views are published.