Ulrich Margull
email: ulrich.margull@1mal1.com

Michael Niemetz
email: michael.niemetz@continental-corporation.com

Gerhard Wirrer

Quirks and Challenges in the Design and Verification of Efficient, High-Load Real-Time Software Systems

Keywords: Scheduling, real-time systems, highly efficient systems, robustness, automotive power train

Existing concepts for ensuring the correctness of the timing behavior of real-time systems are often based on schedulability analysis methods using exact proofs. Due to the complexity of the scheduling problem, today typically worst case approximations are used to judge the reliability of the timing behavior in software systems. In industrial practice, however, this leads to large safety margins in the design of products which are commercially unacceptable in many application domains. For such highly-efficient systems, schedulability analysis methods that are too pessimistic are of limited benefit. As a consequence, penetration of real-time analysis is suboptimal in the industrial software development, which possibly leads to insufficient quality of the developed products. Therefore, new approaches are needed to support the design and validation of high-load real-time systems with an average CPU load of 90% or above to improve the situation.

I. Introduction

In safety critical real time devices, the design and test of the timing behavior plays a central role. With the increasing complexity of devices during the last years, this fact became more and more a consensus in the embedded systems community. Still, in several application domains for embedded devices the penetration with concepts resulting from research activities is quite low, especially when compared to the desktop and server domain. This is astonishing, as embedded devices are typically facing higher requirements in terms of real-time and reliability. While a state of the art schedulability analysis based on worst case execution times and maximum interrupt rates is able to provide the proof that the timing behavior of the system is guaranteed under all circumstances for each calculation instance, this leads typically to a normal load of the system which is around 50-60% of CPU calculation power. This safety margin, which will probably never be needed during the lifetime of the system, increases the cost of the system significantly, being especially problematic in case of higher volumes. In the automotive domain, for example, huge volumes are combined with the need for safe systems plus In this paper we will show as an example a nontrivial and significantly event driven embedded system, which has been proven to work reliably with CPU loads above 90%. Afterwards, we will discuss some aspects where an improved theoretical understanding is required in order to design and verify such systems systematically.

II. System Description

Engine controls units (ECU) have strong real-time requirements. More than 80% of all timing requirements are faster than 10ms. In specific system scenarios, e.g. car going downhill with continuous engine overspeed, an extreme high average CPU load is achieved which can reach even more than 90%. Figure 1 shows the load caused by calculations having been assigned to a certain deadline 1 in relation to the deadlines in a typical application. The upper part of Figure 1 shows the typical load associated with time-triggered (Load TT), with enginesynchronous (Load ENG) and with interrupt associated calculations. Only about half of all calculations are time-triggered, the rest is event-based, either dependent on the engine position or on other event sources. On the bottom part of Figure 1 the same load is shown, this time accumulated over the deadline. On the right side the total load of approximately 95% is shown. Please note that this is the average load, measured over many seconds, not only a short peak! From an overall system point of view many of these deadlines are really hard deadlines. For example, missing an injection or ignition point in time can lead to engine misfire, missed combustion cycles or even engine damage and is therefore not acceptable. Being able to reach such high loads under those requirements shows to what elaborate performance the industry is driven to in the quest for efficient, affordable systems.

III. Challenges for System Design and Verification

When designing such a system, one faces a set of contradicting requirements: the system has to fulfill safety requirements, it must be reliable, and highly efficient all at the same time. There are several strategies to implement these requirements, and they must be well-balanced against each other. The real-time behavior of the system is strongly influenced by the scheduling approach and the distribution of the calculation loads. In the automotive industry, operating systems compliant to the OSEK standard are used in most control units. It provides a simple priority-based scheduling with priority-ceiling protocol. However, fast calculations with deadlines of 100µs or below are often implemented as interrupts. The main challenges in system design is the partitioning of the calculations and how to distribute them on different priority levels. One critical point is to determine whether a given system will work correctly with the chosen loads and priority levels. Using a classical response time analysis [START_REF] Audsley | Optimal priority assignment and feasibility of static priority tasks with arbitrary start times[END_REF], one would determine the worst-case execution time 1 We are using relative deadline, i.e. the time duration from activation of one job instance until its calculation is finished.

(WCET) for all calculations, model the maximum event stimuli and event rates, for calculating the feasibility. However, this approach is by far too pessimistic. For example, for a typical ECU system, such an analysis easily yields a requirement for 130% to 200% of the available CPU resources! In contrast, the industrial practice shows that the systems are working correctly and are robust and fail-safe in operation. The consequence is that a WCET schedulability analysis has no real meaning for productive systems: if it is feasible, then it is not efficient enough, and if it is efficient, the analysis yields a "not feasible" result! A rather pragmatic proceeding to overcome this dilemma is what we call the optimistic approach. Here, several aspects of pessimism are replaced by more optimistic approaches. For example, Figure 2 (top) shows an allegory of the system state of an electronic control unit. The bubble in the middle represents all the state space that is explored in operation with the "center of mass" denoting an average CPU utilization of about 60%. In order to guarantee schedulability, a classic analysis of this system has to contain some amount of pessimism which is indicated by the box around the bubble. Using an optimistic approach, we remove the pessimism where possible, thus concentrating on the core of the state space while neglecting some of the rarely used corners (Figure 2 middle), which in the end allows higher average system utilizations (Figure 2 bottom), e.g. up to 90%. However, since we no longer guarantee schedulability in all possible circumstances, the system has to be designed quite differently. It must be robust in such a way that sporadic overload situations are handled gracefully, as well as encapsulating safety relevant or other important functions in areas that can be guaranteed to be scheduled correctly (or are completely independent of the scheduling, e.g. by implementation in hardware). A classic feasibility analysis contains different kinds of pessimism that can be questioned. One way to reduce this pessimism is to use the average execution times (AET) instead of the WCET normally used in classic feasibility analysis. Our experience shows that when using AET we are able to model the system realistically even at very high CPU loads. On the other hand there is the risk that an AET based analysis might give a too optimistic picture of the system, so the analysis results need to be used carefully and require backup of measurements from the real system. In order to improve the analysis, we add different levels of stochastic behavior to the system: variation of stimuli or variation of runtime (e.g. like [START_REF] Manolache | Schedulability analysis of real-time systems with stochastic task execution times[END_REF]). Another useful analysis strategy which we have successfully applied are stress-tests, where the system is driven artificially into a crisis by adding a dummy load according to a certain pattern. Stress tests help to Fig. 2. The overall system load typically has a complex dependency on the current system state. The details of this dependency are often not known in advance due to the complexity of the system. In the figures above, the space of valid system states is represented by the ragged blob-like shape assuming for further simplification that every position within the shape has the same probability of occurrence. Ignoring small areas (with a consequently small probability) during the design of the system allows a more efficient hardware usage. However, this leads to timing violations in some dedicated operation states which need to be known and handled safely.

understand the system dynamics in critical conditions: At which point does my system break down? How does it break-down, what are the weak points leading to a break-down? The analysis of our systems is further complicated by the fact that our systems are too complex to be fully understood in theory. For example, they are not rate-monotonic, not even strictly deadline-monotonic, they are partly time-triggered, partly event-based, the scheduling is to large parts cooperative, not preemptive, data dependencies cannot be modeled completely due to the complexity of the software (more than 150.000 dependencies), etc. No analysis method seems to be available that takes all restrictions into account, or is able to handle such complexity. As a pragmatic approach we use simulation methods: a model is created which is abstract enough to allow easy simulation, but still detailed enough to reflect the relevant timing aspects of the system (see [START_REF]chronSim[END_REF]- [START_REF] Davis | A survey of hard real-time scheduling algorithms and schedulability analysis techniques for multiprocessor systems[END_REF]). The advantage is that nearly all constraints can be modeled in short time, enabling us to evaluate a system in a fast, flexible way. The main disadvantage of a simulation approach lies in the fact that one carefully has to prepare the simulation, otherwise the worst-case scenarios are not triggered, and the result might then be too optimistic.

IV. Need for Theoretical Understanding

As it was shown in the previous chapters, it is obviously possible to design systems where the state of the art methods of schedulability analysis can not deliver a valid result while the system is behaving well. Today, several state of the art techniques like e.g. EDF scheduling are available to be introduced in the automotive powertrain domain [START_REF] Margull | Improved Schedulability using EDF in Automotive Powertrain Software[END_REF] for coping with the the challenges described above. Besides the application of existing concepts also new approaches are required. In order to manage the development and quality control for such systems, several additional research fields would be of interest. First of all, design principles need to be developed which allow to design robust systems that are able to work up to very high processor loads without failure due to execution time problems. This may for example involve special scheduling strategies, design patterns for algorithms, a more flexible approach for classifying and handling of deadlines, e.g. by allowing sporadic, i.e. non successive, deadline misses, or concepts how to distribute processes to tasks or CPU cores. Also research to evaluate requirements regarding their consequences for the robustness of the timing behavior would be extremely helpful, as this may help to influence the engineering of the system in a very early phase. Depending on certain properties, some functional requirements will not only consume the resources needed for their implementation, but will also increase the required resources (typically calculation power) that have to be kept as a reservoir to ensure a reliable and safe operation. Finally, new concepts for validating systems are needed, e.g. by introducing probabilistic testing approaches, or by defining metrics that show the degree of dynamic robustness of the system. Such research would also allow to compare different project configurations and thus enable the definition of quality thresholds and successive improvement of the software as described in [START_REF] Boers | Application specific performance indicators for quantitative evaluation of the timing behavior for embedded real-time systems[END_REF]. In general, for domains like automotive powertrain new strategies are required for creation of efficient, high load systems above 90% average CPU load that are at the same time safe, reliable and efficient. The focus needs to be on robust systems that are capable of handling small disturbances and a certain amount of "glitches" in the system gracefully, rather then on proving 100% error-free and "feasibility under all possible conditions". Here, maybe, the embedded systems need to learn from biological systems which are far from being perfect -but are often astonishingly reliable, robust and efficient.

Fig. 1 .

 1 Fig. 1. Partial (top) and accumulated CPU load (bottom) of timetriggered, engine-related and ISR-based calculations

Acknowledgment

We thank Denis Claraz and Annette Kempf for inspiring discussions and helpful comments.


Christian Knüchel

Michael Rudorfer

Stefan Voget

Stephan Eberle

Romain Sezestre

Aldric Loyer

Artop -an ecosystem approach for collaborative AUTOSAR tool development

Keywords: Artop, AUTOSAR Tool Platform, AUTOSAR tools, Eclipse, open source ecosystem

A successful approach to develop and evolve complex technologies is to establish an ecosystem around such a technology. The AUTOSAR Tool Platform (Artop) builds on this idea of an ecosystem and adapts it to the automotive domain in the field of AUTOSAR tool development. This paper explains how this idea has been implemented with Artop. Artop is an implementation of common base functionality for creating tools used in designing and configuring AUTOSAR compliant E/E systems and electronic control units (ECUs). The platform is jointly developed in an active community of AUTOSAR members and partners. Similar to Eclipse, a well established open source ecosystem, Artop propagates an ecosystem that is based on four principles which are: (1) low entry barriers; (2) commercially friendly licensing; (3) clear technical focus; and (4) awareness for the competitive differentiators. Essentially the Artop ecosystem takes the well known AUTOSAR mission "Cooperate on standards, compete on implementation" one step further to: "Collaborate on commodities -compete on differentiating implementations".

Introduction

In the IT industry today a variety of successful ecosystems have been established around complex technologies to develop and evolve such technologies. Lately this has also been a strong trend in the field of mobile communications and operating systems with the appearance of ecosystems like the Open Handset Alliance (Android), Symbian or Moblin/Meego. With Artop the concept of an ecosystem has been applied to the field of AUTOSAR tool development.

The idea of such a tool platform was first presented at the European Conference of Embedded Real-Time Software (ERTS) in 2008 with a platform which was then called ADP [START_REF] Heinecke | Enabling of AUTOSAR system design using Eclipse-based tooling[END_REF].

The term ecosystem as it is used throughout this paper refers to a concept where the value of the products and services of the ecosystem participants complement each other. A requirement for such ecosystems is that common technologies and standards are used. This paper will explain how the ecosystem approach has been realized with Artop. Section 2 gives an overview of Artop itself and the underlying AUTOSAR standard. Section 3 presents the Eclipse ecosystem and its influence on Artop. Section 4 explains the four main principles of the Artop ecosystem. Finally the current challenges and an example for the successful exploitation of the approach are described. Some of the Artop Design Members are also active in the EDONA project [START_REF]EDONA project of the System@tic Paris Région Cluster website[END_REF] which had an influence on the technical background of Artop. In addition to the contributions of the Artop Design Members more content of the platform is provided by the Contributing Members which currently are itemis, OpenSynergy and Tata Elxsi.

Artop and AUTOSAR

The third role of Artop User Group participants are the Artop adopters who consume the Artop platform but do not contribute back at this point of time.

AUTOSAR

AUTOSAR is a standardized software architecture for complex E/E systems. AUTOSAR has been launched by the automotive industry to handle the increased complexity of E/E systems needed to provide the functionality expected in modern vehicles. AUTOSAR defines standards on how to describe and configure software that runs on ECUs in the automotive domain. For the ECU description and configuration, formal models containing large amounts of data have to be handled by specialized tools.

The description of AUTOSAR systems is a multi step process that is described by the AUTOSAR methodology. Each step in the methodology involves the manipulation of the formal models which needs to be supported by an appropriate tool. Since each step operates on the same formal model, i.e. the AUTOSAR meta-model, there is a certain amount of commonality among the tools needed for each step.

Artop provides the common parts of AUTOSAR development tools that are outlined by the AUTOSAR methodology.

The Eclipse ecosystem as a blue print

Eclipse [START_REF]Eclipse Foundation: Eclipse website[END_REF] Eclipse also supports collaboration in vertical markets. Using Eclipse all players within one industry can work on one technical platform. This allows them to concentrate on their expertise and try to co-develop the rest, even if this means collaborating with competitors in that area. There are already working groups in Eclipse for the healthcare industry and mobile phone industry as well as others. An industry working group for the automotive industry is about to be founded.

The field of AUTOSAR in the automotive industry can also be seen as a vertical market and a corresponding ecosystem has been established by the AUTOSAR development cooperation. Even though AUTOSAR heavily depends on tool support the AUTOSAR development cooperation does not support the development of a tool platform implementation. This leaves room for an ecosystem approach for collaborative AUTOSAR tool development.

The Artop ecosystem principles

The Artop ecosystem approach is based on several principles which are crucial for the cultivation of a successful ecosystem and the attraction of a large community around Artop.

Low entry barriers

Artop is open to all AUTOSAR members and partners. That means Artop, including its source code, is available free of charge to the AUTOSAR community. Developers, engineers and other individuals in the AUTOSAR community can easily obtain Artop via the Artop website [START_REF]Artop User Group: Artop website[END_REF] after a simple registration step that is necessary to verify the AUTOSAR membership status. There is no extra formal membership or written agreement necessary to use Artop besides the acceptance of the Artop software licenses. Also the initial hurdles to participate in the development of Artop are low. Artop uses a transparent development process where all development artifacts like source code, bug reports documentation and build results can be accessed by all registered Artop users. This allows users to observe the development progress, to report bugs and feature requests and also to join development related discussions.

Commercially friendly licensing

The Artop software licenses (ASL) define the terms under which Artop can be used and distributed. It also defines how contributions back to Artop are handled. The licenses are modelled after the Eclipse Public License (EPL). Just like the EPL, the ASL allows the inclusion and exploitation of software developed by the Artop User Group in both commercial and non-commercial products. Separate Artop licenses are necessary to fulfil the terms and conditions of the licenses of AUTOSAR itself.

Since the ASL also regulates the contributions back to Artop it is also the basis for the collaboration in the Artop User Group. Essentially the ASL defines the terms that apply when a member in the Artop User Group contributes source code to Artop. Therefore the Artop User Group can easily accept contributions without additional regulatory steps.

The evolution and maintenance of Artop from a technical point of view, including feature expansion, increased quality and robustness, depends on the input by the users of Artop. To encourage such input the ASL asks licensed users to contribute modifications to Artop itself back to the Artop community. Of course this only applies to modifications and not to complementary additions or separate software modules that are shipped with Artop in a commercial product.

Clear technical focus

Artop and the Artop User Group was launched with a clear definition of the technical content of the tool platform. With this definition all members of the Artop User Group agreed on the technical direction of Artop and all development effort has been aligned to this direction.

A main aspect in the definition of the technical content has been the focus on platform functionality as opposed to end-user functionality. In the context of AUTOSAR tools the end-users are engineers that are directly involved with AUTOSAR-based system or ECU development. Platform functionality more precisely means APIs, frameworks and other infrastructure that tool developers need in order to build AUTOSAR tools for end-users.

In a different way the technical content [START_REF]Artop Whitepaper[END_REF] The development of a platform like Artop that is based on a collaborative development approach, is driven by continuously adding new contributions. Where these contributions can either enhance existing functionality or add new functionality. Basically each contribution has to be evaluated to determine if it is aligned to the technical content of Artop. This alignment ensures that the platform always provides a consistent set of features that is transparent to the Artop community.

The definition of Artop's technical content also helps contributors to plan their products as well as their contributions to the platform. More generally the Artop ecosystem participants are able to adjust their technical strategies to fit with the focus of Artop.

Awareness for competitive differentiators

As described in the previous section the technical content of Artop focuses on platform functionality. This has also been a consequence of the common understanding of the Artop User Group members that the competitive differentiator in the field of AUTOSAR tools is actually not the platform functionality.

An AUTOSAR tool differentiates itself with the value it represents towards the end-users, i.e. the AUTOSAR system and ECU engineers. Ideally this value is achieved with: (1) ergonomics and usability;

(2) close support of the end-user's development practices and processes; and (3) finally mastering complex model transformations and processing steps. In the field of AUTOSAR examples for such steps are: (1) the creation of ECU extracts; (2) the system mapping; (3) RTE generation; and (4) even automated system generation. Also current series projects are often hindered by missing or incomplete integration to proprietary and non-proprietary backend systems and development practices used by OEMs and Tier-1s. For example:

• Integration with other frequently used exchange formats used for coding, diagnosis and bus system design like CAF, ODX and Fibex.

• Integration with other engineering tools used for application development like Ascet and Matlab/Simulink.

• Coupling with other E/E description databases and PDM systems, that are often proprietary and OEM specific. Since Artop provides the common parts of an AUTOSAR tool, the Artop ecosystem participants, like tool vendors and service providers, can focus their efforts on the mentioned differentiating enduser features. This is possible since the effort to develop the platform is shared among the Artop User Group members. Also the effort for the maintenance of Artop which heavily depends on the evolution of the underlying technologies is spread across many shoulders. The work to keep up with new versions of the AUTOSAR standard can be accomplished once in the community instead of several times in each company.

Challenges

In the previous chapter we showed the advantages of an ecosystem approach for AUTOSAR tools. Of course there are also challenges that need to be overcome.

Perception of commodity

The ecosystem approach builds on the thesis that a company should clearly define its core competencies and focus on that part and develop the rest of its product in a community approach. Of course the delimiter between commodity and differentiating feature is not a sharp line but can vary from company to company as well as it can move over time. Today's differentiating feature can easily become tomorrow's commodity.

On the one hand, this puts pressure on a company to constantly build new unique features that give their products an advantage over the products of its competitors. On the other hand, the rapid evolution of technology results in a constant need for new differentiating features which speeds up development and drives innovations. This separation between commodity and differentiating feature is an important strategic decision. Artop attempts to provide guidance through a clear purpose and aim of the tool platform. However, the decision on what one should contribute to the Artop community ultimately rests with the contributing company. The decision consequently affects the success of a product and a company.

Different technical backgrounds

The decision to use an existing platform as a basis for a product or tool is quite easy if one starts developing a new product from scratch. One can simply evaluate candidates for such a platform and select the most suitable one from today's point of view. This is often the case for small niche products that are developed without a history. But a lot of companies are already on the market with their tools for quite some time, which means they have already invested in their differentiating features as well as in functions that may be considered as commodity.

These existing solutions can be on a completely different technical background than Artop in terms of different programming language, different set of tools and libraries or simply different concepts for the realization of the base functionalities.

Of course the more an existing product differs from a considered platform in the technical background, the harder it to switch to such a platform. The "low entry barrier" is not that low in that case. This does not necessarily mean that it does not pay to switch to a community platform. In general it still can be an advantage to consider changing the technical platform of a product to participate in development that is done in an ecosystem. Several large companies in the IT industry, like SAP [START_REF]SAP Extends Commitment to Open Source, Increases Eclipse Membership Status and Joins Additional Eclipse Projects[END_REF] or Hewlett Packard [START_REF]Eclipse forms independent organisation[END_REF] have announced that they successfully did so. This is also true for some of the Artop members like Continental who switched their existing products to Artop although some major changes to their technical platform had to be done.

Artop Exploitation

For Artop the benefits of an ecosystem do not only exist in theory. The investment into Artop is already paying off for the members of the Artop User Group and also for some of the other Artop adopters. Several commercial tools exist on the market that are based on Artop.

One example of such a commercial tool is Cessar-CT offered by Continental Engineering Services [START_REF] Voget | How the concepts of the Automotive standard 'AUTOSAR' are realized in new seamless tool-chains[END_REF]. Cessar-CT has recently been migrated to use Artop. This migration goes along with added support for the new AUTOSAR 4.0 release. This support was not developed from the ground up but uses the existing broad AUTOSAR meta-model support in Artop. The example shows that a collaboratively developed, common base platform like Artop helps to keep up with the evolution of an underlying technology like AUTOSAR.

Cessar-CT is also a good example for another benefit enabled by the Artop ecosystem: tool interoperability. The end-user tools provided by Cessar-CT are focussed on specific development steps, which are basic software development and ECU integration. With Artop it is possible to tightly integrate other end-user tools with Cessar-CT, e.g. for application component development. Due to the nature of Artop this integration will be realized beyond the XML file level but on an in-memory model level which allows close interaction of the complementary integrated tools.

Another example of the exploitation of Artop is within the EDONA research project and by the commercial product AUTOSAR Builder from Geensoft. Here Artop is at the time used as the base platform for the research project and for a commercial tool. Within EDONA Artop is considered as the reference platform for all components that deal with AUTOSAR, e.g. timing extensions for AUTOSAR 3.0; or EAST-ADL2 transformer tools. AUTOSAR Builder realizes an end-user tool suite which is used in series projects. It includes among others, tools for application component development, for generic ECU configuration and for RTE generation.

Conclusion

In this paper we presented the approach of developing tools for AUTOSAR in a new way by establishing an ecosystem around a platform for these tools. We introduced the ideas of an ecosystem and gave an example of a successful existing ecosystem, namely Eclipse.

Artop is a similar approach for creating an ecosystem for AUTOSAR tools. We described the success factors for Artop, but also mentioned some of the challenges currently faced by the Artop designers and potential adopters.

Artop is already used in commercial products like Cessar-CT or AUTOSAR Builder. This indicates that an increasing number of companies are ready to exploit Artop as the benefits of this exploitation exceed the required effort. Also several other companies are already using Artop or are seriously investigating to adopt it.

We observe that the Artop community is growing. The more players decide to adjust their strategy to use the Artop platform, the more each individual company will gain from the expanding ecosystem.

Acknowledgement

Figure 1 :

 1 Figure 1: Relation between AUTOSAR and Artop 1.1 Artop and the Artop User Group The Artop User Group is a group of licensed users of the AUTOSAR standard. It was launched in October 2008 and provides Artop, the AUTOSAR tool platform. Currently, there are approximately 500 registered individual Artop users from 120 different companies. It is run by the Artop Design Members which currently are BMW Car IT, Continental Engineering Services, Geensys and Peugeot Citroën Automobiles.

  has also been defined to match the common needs of the different tools referenced by the AUTOSAR methodology. The basic common needs are: AUTOSAR meta-model APIs, AUTOSAR model XML serialization and workspace management. More advanced needs are support for model validation and different types of model viewers and editors, e.g. tree-based, form-based or diagrambased.

Figure 2 :

 2 Figure 2: Distribution of end-user value

  to model transformations, code generation and many more which makes Eclipse the first choice in model based software development (MBSD). As AUTOSAR requires a lot of these modelling support features, Eclipse was predestined as a basic building block for AUTOSAR tools.

	open source community that is the basis of an
	ecosystem of complementary products and services.
	The Eclipse Foundation has around 200 member
	companies, about 1000 committers and consists of
	more than 100 separate projects. It is estimated that
	Eclipse is used by more than 4 million developers
	worldwide.
	Eclipse consists of many technology subprojects.
	One of them is the Eclipse Modelling Project, which
	supports model based development with powerful
	frameworks for graphical modelling, textual
	modelling, model
	is a prominent example of a successful
	ecosystem. It is an extensible open source platform
	for developing all kinds of software. It started as an
	IDE (integrated development environment) for Java
	programming in 2001 and through continuous
	extension and evolution it has emerged to a platform
	consisting of various open source frameworks for
	building almost every kind of software.
	The Eclipse platform is provided under the Eclipse
	Public License (EPL), an open source licence that, in
	contrast to some other open source licenses like the
	GPL, allows users to build their software on top of
	Eclipse and commercially distribute it.
	Eclipse was established in 2001 and from 2004 it
	has been managed by the Eclipse Foundation, a
	non-profit organisation that develops, hosts and
	maintains the Eclipse platform and drives an active

Acknowledgments go to:

Paul Hoser (BMW Car IT), Robert Kiss (Continental), Ali Akar (Geensys) and Yannick Didierjean (Geensys) for driving the technical architecture of Artop.

Ralph Müller and Mike Milinkovich (both Eclipse) for insights into the Eclipse ecosystem.

Simon Fürst (BMW) for spreading the Artop idea in AUTOSAR.

Eliane Fourgeau (Geensoft/Geensys) and Harald Heinecke (BMW Car IT) for valuable advice on how to develop the Artop idea.


A Sophie Stern

B Olivier Guetta

Manage the automotive embedded software development cost by using a Functional Size Measurement Method (COSMIC)

Keywords: Software metrics, Software functional size, Software development workload estimation process, Productivity models, COSMIC. Functional users Software Or peripheric Or another software Boundary

Year after year, more and more cars' functionalities are performed by software: electrical vehicle, multimedia, connectivity with the outside world and so on. As its software development costs are increasing, Renault decided to develop metrics and an estimation process in order to be able to predict its software costs early in the vehicle or power-train project. At the same time, Renault is working with its major Electronic Control Units suppliers to contract with them on the basis of software metrics. After different studies, Renault chose the COSMIC method as its embedded software metric. COSMIC is for COmmon Software Measurement International Consortium, and is also the name of a functional size measurement method, ISO standard since 2003.

Introduction

Year after year, more and more cars' functionalities are performed by software: electrical vehicle, multimedia, connectivity with the outside world and so on. Software development costs are more and more important in the whole project development costs. Depending on the Electronic Control Units (ECUs), Renault subcontracts their development partly or entirely. But if Renault is very used to estimate accurately the development cost of physical parts such as harness or electronic components, it was quite lacking in for software development cost estimation. So the Renault embedded software group was asked to develop a software workload estimation process that had to be usable for all suppliers. The main goal of this presentation is to give a feedback on the Renault practical experimentations with the COSMIC method on its real-time embedded software. We will first explain productivity models and our choice of the COSMIC method to measure the software size. Then we will present the COSMIC method and how we tested it on industrial cases. Along the way, we will give advices stemming from our own experience. Each of them will be flagged by the symbol ☺.

The software development productivity models

Introduction

The final target of having a software development workload estimation process usable for all suppliers requires the knowledge of the relationship between a software characterization and the corresponding development workload, either for the whole software or either for a piece of software. When the relationship is clearly established, the software development workload prediction becomes possible.

What are productivity models?

Productivity models are really the key of our estimation process, they are obtained by statistics methods.

A software productivity model is a linear relation between a functional software size and the corresponding development workload.

Figure 1: A software development productivity model

The software functional size is independent from the kind of implementation. The number of lines code, still used in a lot of organizations, is not pertinent, especially for embedded software. Everyone knows that for saving place, an engineer may spend a lot of time to compact its code. Furthermore, with Automatic Code Generators, the number of lines is more important but the spent time is less than with manual coding. The linear relation between the functional size and the software development workload is obtained with data set on past projects by using linear regression methods. Each datum is representing the whole A productivity model has to be constructed on a coherent perimeter. We explained in more details this notion in § 5.4 to 5.6.

2.3

What are the uses of software productivity models?

As we said before, productivity models are the key of our estimation process. We exploit them in different uses, the major ones are:

• Estimation of development workloads (and deduction of costs, delays), • Benchmarking of suppliers productivity,

• Managing of suppliers' annual productivity.

To be pertinent, benchmark must be performed on a coherent perimeter and with the same conditions. Productivity models may be used as a basis of discussion with each supplier about its software development tooling for example. Notice that development productivity must be related to the quality level of the associated software.

The choice of the Functional Size Measurement Method

What are functional size measurement methods?

To construct a productivity model, you need to measure first the software functional size. Functional Size Measurement (FSM) methods have been used since the early eighties. FSM methods enable to measure software independently from its implementation, they measure the software functional wealth.

Each FSM method has its own unit, Function Points (FP) for the IFPUG method, COSMIC Function Points (CFP) for the COSMIC method. The software functional size is an intrinsic metric of software, as the length in meters and centimetres for the size of a car.

Once we decided to define productivity models, the first step was to choose the FSM method.

The different FSM methods experimented

Renault experimented the COCOMO method a few years ago in the ECU diagnostic and in the Engine Control Module departments, but with unsuccessful results.

The IFPUG method has been used for several years in the Renault Information System department.

The possible application of IFPUG on embedded software had to be checked with an evaluation. Furthermore, we decided to experiment the COSMIC method as it was announced to be well adapted to real-time embedded software.

Our first experiments started on the Engine Control Module (ECM) in 2008 with the IFPUG and the COSMIC methods. The ECM is modular and each of its modules is a set of model-based specifications.

The effort supplier invoice is available for each module.

Functional size measurements were realized on the same modules with the two studied methods, and then results were compared.

In our experimentation, the COSMIC method suited well for embedded software whereas the IFPUG method appeared not pertinent especially when the software functional size was increasing. Furthermore, the measurements with the COSMIC method seemed more repeatable and faster than the ones with the IFPUG method. Last but not least, as we are more and more involved in the model based design process for our ECUs, it appeared to us that COSMIC could be automated in a further step on models realized with a simulation tool.

So we decided to pursue the experimentation for embedded software with the COSMIC method on several types of ECU and for different suppliers. Our results have been very encouraging since the beginning. Let's see now more deeply how we applied COSMIC and the productivity models on embedded software.

The COSMIC application on embedded software

The COSMIC method is an ISO standard (19761,2003). The whole COSMIC documentation is free and available on Internet, cf [START_REF]COSMICON[END_REF].

The software functional size measurement according to the COSMIC method is based on data movements measurement: either through the boundaries between functional users and software, either data movements forwards or from memory device. So data movements are: entries, exits, reads and writes.

Figure 5: The mapping of COSMIC concepts on a view of software functional requirements All COSMIC concepts are detailed in [START_REF]Common Software Measurement International Consortium, The COSMIC Functional Size Measurement Method Version 3.0.1, Measurement manual[END_REF].

In this chapter, we will explain with some highlights how we applied the COSMIC method on embedded software with an industrial example: our study on the Engine Control Module (ECM).

The COSMIC method is composed of three phases:

• The measurement strategy phase.

• The mapping phase.

• The measurement phase. ☺This phase has to be realized with the customers of the productivity models.

Determine the purpose of the measurement

The purpose of the measurement was defined with the purchasing and the ECM development departments.

The purpose was multiple:

1. To be able to estimate the function software development cost as soon as its specification is written to decide to implement or not the function. 2. To predict the cost of the software functions development in order to negotiate if necessary with suppliers. 3. To benchmark and manage the productivity of our suppliers.

☺The purpose of measurement may be different from one ECU to the other one.

Determine the scope of the measurement

The ECM is developed schematically depending on the following architecture: hardware, basic software, specific software depending on the platform, portable software independent from the platform. The measurement scope in 2009 included all suppliers developing portable software, since 2010 it also includes specific software developed by major suppliers.

☺It is important for you to know that it is possible to start working only on some pieces of software and not on the whole software.

☺ In a general way, we decided to restreint our COSMIC studies to ECUs which are model based.

Determine the functional users

For the ECM, functional users are interfacing peer software, in other words, when we consider one module, functional users are the others modules in contact with it.

Determine the level of granularity

choice of the granularity level is very important, it may be very different from one ECU to the other one. For the ECM, we chose the granularity level of the software modules. There are two reasons for this choice, the first one is that it is already the level of costs negotiation with suppliers, the second one is that this choice enables to have quite a welldimensioned data set on past projects for constructing productivity models.

The COSMIC mapping phase

This second phase includes the identification of functional processes, data groups and data attributes.

Identify the functional processes

A model-based specification is composed of blocks and data flows that the COSMIC method interprets as functional processes and data movements, or not. It is important to understand that the COSMIC method doesn't take into account the specification's architecture. The important point is to identify parts of specification which are not only structure but which are active parts.

Identify data groups

We measure variables and calibrations as data groups with the COSMIC method.

Identify data attributes

☺At the present time, we do not take into account this step of the COSMIC method in our mapping.

The COSMIC measurement phase

Identify data movements

This is the identification of data which are going through the boundaries between functional users and functional processes (Entries or Exits), and the identification of data read from or written in the memory (Readings and Writings). Once you identified functional processes and functional users, you are able to identify data movements.

Apply measurement function

When data movements have been identified, you just have to count one COSMIC Function Point (or 1 CFP) for each data movement, whatever its nature.

Aggregate measurement results

The last step is to aggregate the COSMIC Function Points on the measurement.

☺For the ECM, a module is described by several specifications files. We measure each specification and aggregate the results in each specification and then for the whole module.

4.4

Software enhancements COSMIC measurements

Introduction

With the arrival of ECU's architecture standards as AUTOSAR, there is now the possibility of developing the same code for different hardware targets. In this context, more and more software is reused from one ECU to another one. So beside new software developments there are more and more software enhancements.

The functional size measurement process is different for a new development or for an enhancement.

4.4.2

Highlight points for enhancements measurements A software enhancement functional size measurement is obtained by comparing the two specifications releases, what we called release N and release N-1. The COSMIC method measures differences between the enhanced specification and the previous one.

When modifications affect functional processes, data movements or elementary blocks in the release N, they are caught by COSMIC measures.

There are three types of modifications:

• Addings, • Modifications, • Removals.
When an element is added, we measure its size.

When an element is deleted, we measure its size before its deletion.

When an element is modified, we measure the size of the modification.

Elements modifications caught by the COSMIC method are for example a variable renaming, a modification of a variable size.

The three types of enhancements are measured, and functional sizes are added at the end.

Industrial examples and key points

Obtained results

We already studied and obtained results on several projects, on several kinds of ECUs, for new developments and for enhancements. For example, we have COSMIC results on:

• The Body Control Module, or BCM.

• The Engine Control Module, or ECM.

That means we realized several productivity models for each ECU and for different suppliers.

We are studying other types of software.

An example of cost prediction

We applied our BCM COSMIC new developments productivity models during a Request For Quotation with suppliers for a new BCM in order to have a target software development cost on the applicative software. As soon as the specifications were written, we predicted the development effort for each BCM function within a prediction interval, we also predicted the development effort and the associated prediction interval for the whole applicative software. At the same time, the suppliers realized their own estimation each of them with their home made method. Then we compared their estimations and ours before negotiating. The graph below shows the comparison between our COSMIC predictions and the estimations of one supplier after the first technical and economical round and after the second round of the RFQ. We discussed with suppliers on software modules for which their estimations were very far from our predictions based on COSMIC and on our productivity model.

BCM RFQ: COSMIC predictions versus Supplier estimations

☺ Having factual data on software development workloads was a very good basis of discussion and a good lever of negotiation.

COSMIC measurement's Return On Investment and Model Based Design

Whatever the reason why you are realizing COSMIC measurements, one day or another you probably have to check the Return On Investment (ROI) of your software measurement. So you will have to manage and to limit the cost of your COSMIC measurements. The best way to reduce this cost is having the more automated measurement process as possible. Even if the measurement is performed by humans, the more standard the specifications are, the less interpretable they are, the easiest and the quickest the measurement will be. If specifications are written in natural language, with different formalisms, with residual errors, the measurers will have to be experimented people as functional measurement experts.

The COSMIC method may be applied on non model based specifications, for example in natural language, but there are several brakes to realize very performing functional size measurements in these conditions. The COSMIC application is really favoured by Model Based Design (MBD) because specifications are already consistent for the measurer, the formalism is always the same and it is easy to train new measurers who are not necessary FSM experts. Furthermore, automation of COSMIC measurements on MDB specifications seems possible.

Productivity models and outliers

Remember, productivity models are realized on data set on past projects with statistical methods as linear regression. A productivity model is defined for a perimeter, at least for the couple ECU and supplier. By trying to realize a correlation on data set for a coherent perimeter, there may be some points outside the regression curve, these points are called outliers. There are two major reasons of existing outliers: either the outlier corresponds to software intrinsically different from the other ones of the data set, either there are other influent factors than the functional size. Influent factors may be: restricted delay to develop the software, high experienced people, and so on. As you can see, influent factors may reduce or increase the development workload.

Either outliers will go in another productivity model, either they will be stored in a outliers checklist: let's see on a real industrial case.

You can see below the different steps to detect outliers and to take them into account. On this first figure, you can see the whole data set on past projects for BCM for one supplier named X. ☺ Before removing the outliers, you must absolutely find the explanation for each of them otherwise your estimation process will never be strong. The ECU development team has probably this explanation.

In our case, one point was corresponding to an automatically coded software, and that is the reason why we began to split productivity models in function of the coding type, manual or automatic. The two other points had been coded manually by an expert of this software, so we put these points in the outliers checklist.

After removing these three outliers, the data set is the following.

Productivity models split, influent factors

There is not one software development productivity model for all ECUs, all suppliers and all situations.

The influent factors we have already seen are: the ECU's type, the supplier, manual or automatic coding, new developments or enhancements. You can treat influent factors by separating models in several models or consider multi-factors approaches.

For the time being, we have preferred to separate models because it is easier to understand and to explain simple linear regression models.

☺Do not confuse software functional size which is a software intrinsic metric and productivity models. For example, the software functional size is the same either the software is manually coded or automatically generated, but there will be two productivity models on one perimeter with different slopes and ordinates at the origin according to the coding type.

☺ People often have presupposed ideas on influent factors and on the best productivity models split. For example, before realizing the ECM productivity model, the development team thought there will be several productivity models: one for interfaces modules and one for algorithmic modules. Our principle was, let's try to put all modules on one model and then we will see.

You can see below the data set on past projects for one supplier of the ECM. You can see that all points are in the confidence interval. The confidence interval is a factual boundary to detect which points are outliers.

Contracts with suppliers based on productivity models

Large organizations are already used to establish contracts with their software companies based on the fixed price of a Function Point with the IFPUG method.

We found productivity models more interesting than the Function Point fixed price because they take into account the difference between large software and small ones. There is a scale factor. Nevertheless, the habit of contracting on metrics with a supplier is a very mature way of working and we can copy paste this way from Information Systems to Embedded Software, two different worlds but with interesting similarities. So we decided to share with several Renault major suppliers the COSMIC method mapping and their productivity model on some ECUs.

5.8 Where using COSMIC, inside our outside?

FSM methods as the COSMIC method is, productivity models, can be applied inside or outside the company, depending if the software development is totally or partially outsourced.

Possible organizations for software workload estimation process

To construct a complete and strong software workload estimation process, it is a necessity to have in the organization a core measurement team with different skills, especially: informatics and statistics.

Once the process is constructed and strong, there are several types of possible organizations for functional size measurers and productivity models specialists. Either there is only a core team, either the measurement is completely disseminated, either there is a merge of these two ways. Automation may have a role in disseminating the COSMIC measurement task in the whole organization.

Cooperation with the embedded real-time software community

Our Embedded Software group has written rules to map all the COSMIC concepts on simulation tools used by the ECUs' development Renault teams.

Our first objective is to have the same rules for all our suppliers. Our final goal is to have in the real-time embedded software community shared rules for the mapping of COSMIC.

☺ Renault is co-managing with the Ecole de Technologie Supérieure (ETS) of Québec the production of the official COSMIC guideline for sizing real-time system software. Its Web publication is planned for the end of 2010, cf [START_REF]COSMICON[END_REF].

☺ Renault is also involved in the French organization CG2E (Club des Grandes Entreprises de l'Embarqué), in the working group "Productivity of the development chain", and sponsors the COSMIC method in order to have the feedback of the community.

Conclusion

When we began working on COSMIC in 2008, our first objective was to put in place a software development workload estimation process. With our results, we are confident and we are pursuing on the same track. What we have found along the way is: if COSMIC is a very good metric to help us to predict workloads of software projects, COSMIC might be also our software reference metric and the basis for a lot of new uses.

  software or only a part of it, like a software module for example.

Figure 2 :

 2 Figure 2: The construction of a productivity model with a data set on past projects

Figure 3 :

 3 Figure 3: Comparison of software productivity models for different suppliers

Figure 4 :

 4 Figure 4: Part of the Renault comparison between the IFPUG and the COSMIC FSM methods

Figure 6 :

 6 Figure 6: The measurement process in the COSMIC measurement manual ([2])

Figure 7 :

 7 Figure 7: COSMIC use for costs predictions and costs negotiations

Figure 8 :Figure 9 :

 89 Figure 8: The data set on past projects for BCM

Figure 10 :Figure 11 :

 1011 Figure 10: The BCM data set on past projects, outliers removed

Figure 13

 13 Figure 13: A ECM productivity model for one supplier

Acknowledgement

We want to acknowledge here the contribution of Professor Alain Abran, teacher at the ETS, Montréal, Québec, who is one of the COSMIC father and who is always there to answer to our questions.

Glossary

BCM:

Body Control Module

CFP:

COSMIC Function Point COSMIC:

Common Software Measurement International Consortium ECM:

Engine Control Module FSM method:

Functional Size Measurement method IFPUG:

International Function Points Users Group MDB:

Model-Based Design ROI:

Return On Investment


G Carrozza

A Marongiu

L Benini

M Loffreda

Bringing the Embedded Systems Industry Towards Open Source: the SHARE project experience

Keywords:

published or not. The documents may come

Introduction

Over the last decade industry has shown an everincreasing interest in Open Source Software (OSS). Nowadays large, medium and small companies are becoming more and more aware of the actual benefits that OSS provides in terms of reduced license and development costs, license flexibility, increased number of suppliers and market innovation. However, although many OSS products have reached maturity during the last few years, scepticism about open source as a viable option to develop commercial products, or to support critical business functions, still hampers the industrial world transition to open source. The decentralized and distributed development model of OSS makes quality assessment hard to achieve, thus making OSS integration within complex industrial-strength applications a risk. In fact, companies find hard to believe that open solutions are capable of delivering the same reliability level of commercial offerings and that they can actually fill the requirements gap better than commercial products. For large projects and mass market products, the situation is alleviated by the availability of a plethora of OSS software projects, supported by large communities of developers and users. Such a broad user base contributes to increase the know-how, and often makes large open source projects even more reliable than commercial solutions. This does not hold in the case of specialized markets which have a narrow set of product users, and whose requirement for specialized development expertise reduces the size of developers communities. This is especially true for embedded solutions, which are increasingly being used for developing mission and safety critical applications, e.g., for transportation systems, medical devices, critical infrastructures, and mobile computing. These systems are usually developed and tested in house by leading firms and impose strict dependability and real-timeliness requirements, as well as the need for achieving international certifications. Hence, assessing the quality of eligible OSS products to integrate into proprietary industrial solutions becomes a major concern. So far, qualitative and quantitative evaluation methodologies have been lacking (see [QUA] for a comprehensive comparison), as well as approaches which actually address niche market sectors that can be leveraged by the embedded systems industry. Both industry and academia are currently concentrating their efforts on how to enable the embedded systems industry to move toward OSS, all over Europe. This is also witnessed by the increasing number of initiatives and research programmes focusing on embedded systems, like ARTEMIS or European Commission objectives in the framework of 7th Framework Programme (FP7) calls. The SHARE project, funded under the 7th Framework Programme, aims to facilitate and promote the use of OSS software in the embedded systems domain, and to foster cross communications among several classes of stakeholders, especially in the fields of e-health, nomadic applications and mission critical systems. In order to help industry in the delicate task of choosing among several available solutions the one which best fits their requirements, a qualitative methodology for OSS evaluation is proposed which takes into account the specific needs of products targeting embedded systems.

A practical implementation of this evaluation methodology is also provided. The O4S toolavailable on the SHARE web platform -allows a wide community of users and developers with expertise in embedded systems to evaluate and rate OSS solutions. This in our view helps to create an increasing awareness about OSS products, as well as specific know-how on available technologies. Moreover it strengthens the effectiveness of feedback and support actions coming from small communities. The main novelty of the SHARE evaluation methodology -with respect to other previously proposed similar approaches -consists in the set of considered evaluation criteria. Indeed, due to the lack of mass interest in the targeted fields, most criteria encompassed by the SHARE O4S toolwhich actually focus on the peculiarities of embedded systems -have not been considered before.

Through the development of a web platform, the project aims to become a virtual meeting point for the embedded system community where people can share knowledge, as well as to encourage cross domains collaborations. It also aims to promote the adoption of open standards in the embedded systems field, which would give a boost to the use and development of OSS in charge of complying with certifications requirements. This paper will thoroughly describe the tools and the methodology developed in the framework of the SHARE project, as well as the lessons learned by the achieved results.

Benchmarking OSS

In order to enable and drive the evaluation of OSS to integrate into proprietary industrial solutions, or to be included into the industry development process, a benchmarking methodology, as well as a support tool, have been developed by the SHARE project which are described in the following sections.

The SHARE Methodology

The SHARE benchmarking process is an extension of the Qualification and Selection of Open Source software methodology (QSOS) [QSOS], aiming at creating a process tailored for the kind of systems and software benchmarking falling into the SHARE focus. As it is shown Figure 1

The O4S tool

The tool developed to perform evaluations has been created as an extension of the O3S tool 1 to which database facilities and several functionalities have been added to fit the defined methodology.

In this section we describe the tool, as well as the steps it enables to perform OSS benchmarking according to the methodology described in section 2.1.

The evaluation process leverages a set of XML templates, which constitute a skeleton built on top of the identified evaluation criteria. A different template is defined for each target software family, and is structured in two main blocks. The first block is shared among families, as it contains a common set of criteria which describe generic features of the product (e.g. software maturity, availability of documentation and community support.). The second block includes specific criteria to each family, thus differentiating the evaluation based on true key features of different OSS. Criteria can be scored within a range of three values (0 to 2) which typically have the following meaning: 0. the target feature is not supported 1. the target feature is partially supported 2. the target feature is fully supported

In case this classification does not hold, and the score assumes different, criterium-specific meaning, the template provides descriptive comments to aid the evaluation process.

1 http://www.qsos.org/o3s/

The tool, which can be used by the front-end of the project website 2 , allows an iterative benchmarking process consisting of the following steps:

• CONTEXT AND FAMILY SELECTION -Users have to select the area in which they are going to use the product (near real time and mission critical; nomadic and multimedia, e-health and generic applications) and the family of the product (e.g, compilers, operating systems, databases). • WEIGHTING -Users express the weight that they want each criterion to assume thus "formalizing" somehow the requirements he is looking for the specific product.

• EVALUATION -Provides feedback on if and how a given product is capable of fitting user needs. By selecting among already evaluated tools (stored into the SHARE database) users get a comparative evaluation of several tools. • DISPLAY -Plots the benchmarking results in the form of a graphs or tables.

It is worth to note that users are also allowed to propose and evaluate software products that have not been already evaluated. This increases the advance in knowledge sharing, thus pursuing one of the main objectives of the project.

The Share Platform

The Share project aims to enforce relationships between embedded systems industry and OSS communities involving researchers, developers, end users and companies, as well as to promote collaborations favouring an osmotic process of knowledge transfer. This contamination process has been realized through the development of a SHARE WEB SPACE 3 , which is close in spirit to the OSS philosophy as it considers nothing better than web, and the Intranet communication tools, for sharing ideas and information. It follows the line of virtual workspaces which result from the complete integration of traditional information systems and the old idea of Intranet.

Through enhanced communication services and knowledge management tools the SHARE web space provides a working environment in charge of supporting the daily needs of people. It was designed both to provide a virtual meeting point for stakeholders interested in OSS, and to be an effective dissemination means for OSS usage in several fields, from business organizations to communities of developers. This virtual workspace was designed to overcome spatial and temporal barriers to cooperation, and to flatten roles gap.

The WEB space is meant to provide effective support to different figures involved in the OSS software, especially in the context of embedded systems which are becoming ubiquitous in our life. Additionally, it provides the whole lot of necessary facilities to support the Share consortium, such as tools and mechanisms for internal project management procedures. iii. the Collaboration Tracker, which is a framework for supporting and keeping alive collaborations promoted by the projects. This has been particularly appreciated by users that have been provided with dedicated files repository, maling tools and calendars to manage their activities.

Share platform organization and facilities

Within the Knowledge Framework, besides the already described Benchmarking Tool a Review and References section has been included. In fact this is a documentation area, where documents (related to OSS initiative review, user guides, papers and best practices) can be uploaded and stored by the users. This section is continuously updated to keep the community alive and always aware of current progresses in the state of the art.

The spirit of community has been actually realized by delivering the Knowledge Sharing section of the platform where several there have been provided

i. a code sharing framework for allowing users to share their OSS projects ii. a forum and newsletters for encouraging discussions

iii. best practices and recommendation for those OSS and embedded systems practicioners which are going to select products to use.

A Content Management System has been chosen as the technical means for accomplishing these functionalities, and to allow all the users to have editing rights, according to given privileges. This way, the SHARE community can grow continuously thanks to users contributions. The Joomla framework Joomla!, version 1.5.7 4 Production/Stable, has been chosen to develop the site. It is a powerful Open Source Content Management Systems, which can be extended both via installable add-ons authored by the developer community and by customized solutions that users can develop by their own.

The O4S tool and several other extensions are going to be released in the form of Joomla add-ons publicly available. This represents a concrete outcome of the project activities.

Project outcomes

This section describes the main achievements of the SHARE project, regarding the OSS benchmarking activities, as well as the achieved results in terms of community constructions and knowledge sharing.

Benchmarking results

The main aim of the benchmarking process discussed in this paper is that of supporting the creation of active communities of embedded OSS users and developers. The O4S tool is meant as the main technical means to share and spread knowledge and information among the community.

For each of the three main market sectors considered as the basis for OSS classification within the project, several evaluations have been carried out. A total of 80 products have been evaluated by selecting representative applications in the mission critical, nomadic and mobile multimedia applications, and e-health thematic areas.

These evaluations are stored in a database, available for consultation through the web platform.

Users can also provide their feedback (i.e., their own evaluation) to the SHARE community. The tool allows for a convenient and readable display format, where the user can decide whether to display the scores of a single target evaluation (i.e. a single product), or a comparison of all the available evaluations as shown in Figure 3. Similarly, a comparison can be done among several products belonging to a same family. We believe this is a valuable feature for users which are looking for a particular application template, but do not have insight on the specific technical features of different available OSS matching that template. The evaluation criteria are organized as nested subsets of related features (Figure 4). These can be easily displayed at once by simply navigating the criteria tree at different depths through the graphical interface. An example of the exploded view of two nested criteria levels is shown in Figure 5.

Share community and created knowledge

During the project, partners have been strongly committed into dissemination activities, thus increasing the visibility project activities and interim results. This has been particularly helpful to allow an incremental community construction. This section summarizes the achieved results in terms of SHARE community results, knowledge base created and collaborations that have risen exploiting the SHARE support.

At the time of this writing, the SHARE web community counts more than hundred registered users, coming both form universities and industries from several European countries, and almost 30.000 visits.

The SHARE consortium is proud of this results as well as the actual collaborations that have been created thanks to the support provided by SHARE.

An interesting case study comes from the joint collaboration that SESM5 and STM Products s.r.l 6 , an Italian SME active in embedded software development, have created. The collaboration was about the evaluation of an industrial solution (from STM) that was to be tested and assessed in terms of quality and reliability. The collaboration has been particularly fruitful and it will continue over the end of the project. Interesting results have also been achieved in terms of users' interest in the feedback about funding opportunities provided and continuously updated through the Share website. This activity has led to an increasing number of partnerships created through SHARE and aiming to deliver project proposals for the incoming research calls in this field.

Encouraging results also came from the experience that the SHARE consortium put in place with academia. Two real cases are worth to cite. First, a strong collaboration has risen with the University of Naples and the CINI (the Italian Interuniversitary Consortium for ICT) which is witnessed by the 3rd International Workshop which has been jointly organized focusing on embedded systems and OSS 7 . Second, a collaboration has been created with Roma Tre University: students from a master Embedded Software course have been invited to test and use the O4S tool, in the context of a set of seminars held by SHARE partners. Feedback and opinions from students are believed crucial in terms of future enhancement of the Platform, and are going to be published in the form public deliverables on the SHARE website 8 .

Recommendations and best practices

At the end of the project activites, some guidelines for each of the three main areas of interest have been derived. The scope of such achievings can be broadened and generalized to other application fields.

The sense of what quality means with respect to OSS is clearly stated in a short sentence by a famous Linux kernel maintainer: "It takes time to make things work in Linux, but it takes even more to make the code clean and acceptable by the community".

In fact, the lack of attention to the generation of "high quality" code is the main reason for manpower effort requirements planning underestimations and deadline misses, which make OSS adoption still dangerous. Remaining in Linux kernel domain, one of the most evident and important Open Source Software success stories, it has been recently noticed that more and more companies asking for Linux kernel base ports are expecting code deliveries to be fetched from public repositories and are not going to accept local branches or custom patches over vanilla kernels. This is because having the code integrated into the mainstream Linux tree is a guarantee to the customer for excellent code quality and reliability. This is a clear symptom of the need for a renewed development process in line with the OSS philosophy, even if it could somehow be in contrast with organizations' internal development procedures.

The use of standards in OSS becomes then crucial for high quality products development, also in terms of development process. In fact, the high variety of products and the need for a short time to market make the reusability of the software elements mandatory.

So far, OSS leverage two kinds of standards, namely industry driven standards, mainly delivered by industrial consortia like Khronos [KHR], and de-facto standards which are related to widely used OSS which become in fact a standard due to their large diffusion.

Industries accepting to adhere to both these kind of standards would be really allowed to deliver cross-platforms products, i.e., to sell the same software for different platforms, as well as to reuse the code in different products.

This is the reason why the SHARE project is committed to encourage any process driving a software into a standard especially in the fields of mobile and nomadic, real time and e-health applications.

In the context of mobile application development the need for open standards is steadily increasing. Nowadays several mobile devices are available on the marketplace, as well as several software development solutions, which are based on a number of wide-spread operating systems such as Linux, Windows Mobile, Symbian. Besides proprietary systems such as Windows and Symbian -which do not even claim to embrace open source (although many components of their SDKs are based on OSS), many Linux-based systems cannot be considered "open", being ad-hoc solutions, tailored on the characteristics of a specific device. Indeed, just as it happens with other proprietary platforms, not all the features of the device are accessible from the software stack due to the lack of specific APIs. A representative initiative aimed at the adoption of open standards in the mobile area is represented by the Open Handset Alliance (OAH) [OAH], a business alliance composed of 65 participants from several areas including mobile operators, handset manufacturers, software, semiconductor and commercialization companies. The goal of the alliance is that of allowing faster and cheaper development of innovative mobile solutions, which is achieved through "a commitment to openness, a shared vision for the future, and concrete plans to make the vision a reality". Android [AND] software stack for mobile devices is one of the initiatives started and promoted by OAH. Another remarkable initiative in this field is the OpenMoko community [OPM], which supports a completely open solution, from the hardware 9 to the software stack, which includes a rich set of OSS to develop mobile applications. The OpenMoko community is strongly committed to develop Linuxbased solutions for mobile programming that ensure interoperability and shared technologies.

In the field of mission critical and near real time systems, the need for standards is even more exacerbated by certification issues. Companies developing critical systems often encounter serious difficulties in satisfying reliability requirements, in many cases imposed by certification standards, at competitive and acceptable cost and time. In fact, engineering disasters such as the 'Three Mile Island' and the 'Therac-25' incidents let software in this area move from a disordered development field to one with tight, prescribed methods and controls. Standards such as MIL-STD-2167 and DO-178B/C give guidelines for producing safe and reliable software which if followed with well defined methods do have a marked improvement on process quality. Industries delivering solutions for critical domains like aerospace, air traffic management or safety critical systems are required to develop software products exhibiting high Software Assurance Level (SWAL) to be compliant with the these standards for certification. As for the nomadic application area, industrial communities pursuing standard compliance for software products also exist in this area. Open Source Real time Operating Systems (RTOS) (e.g., OSEK 10 ) standardization boards go in this direction trying to encourage the delivery of OS operating systems that could be used in several mission critical and real time scenarios, providing the same reliability level than commercial alternatives. This way they can be integrated into a variety of industrial solutions without mining the overall quality of delivered products and reducing the time to market. Although these standards are not the only factor in producing high quality software, following a well defined process (similar to processes in other engineering disciplines) unquestionably improves the end product.

The best recommendation, then, is for industries to actively participate to these joint initiatives not only to gain visibility but, more important, to put on the table their requirements and open needs than can be shared and accepted by the community.

Regarding e-health applications, the unquestionable improvement that the new technologies have produced within the medical sector has accomplished the inclusion of ICT in diverse medical tasks which, of course, strongly rely on embedded systems. Even if this is a younger field of application, if compared to the nomadic and mission critical domains, there exists a lot of outstanding activities related to OSS. Great attention has been devoted to medical images and vital signs processing which have encouraged the development of several OSS applications. Since signals and image processing have large application, several initiatives are already ongoing to promote OSS, in different engineering fields. What has been observed in the course of the SHARE project, is the lack of OSS specifically supporting the treatment of a given illness (e.g, cardiovascular issues) and, more in general, of highly specialized scientific software for medical diseases.

The benefits that OSS could bring to these areas is great but it has not being exploited so far due to the lack of common initiatives and cross fertilization among different companies delivering proprietary, closed, solutions.

The future of SHARE

It is always a tricky task to quantify the real success of research projects. In the case of SHARE we believe that several measures can give an idea of the real impacts that the project actually got like the collaborations that have been created thanks to the project support, the size of the community that has risen around the project, the number of shared OSS projects. However, what we are really concerned about is the future of the SHARE community, and which ways have been paved for future initiatives. Of course, the trickiest issue will be that of keeping the community alive once that project has finished. The great attention that the SHARE consortium devoted to exploitation is a concrete evidence of such a concern. Even if each member of the SHARE consortium has a different perspective with this respect, all partners agreed that "exploitation" actually means which strategies to adopt to have the SHARE initiative survive and grow. SME partners within the consortium are mainly interested in reusing SHARE results and methodologies in their production chain to improve development process and OSS selection activities. Academic partners are mainly interested in training and further research that may arise from the project outcomes. As for large industries, project results constitute a starting point to promote standardization and gain visibility in international scenarios as well as to lead industrial research and business in their fields of interest. Several actions are foreseen from the whole consortium which, taking into account feedback gathered though technical questionnaires and user forums, will be undertaken to try and keep the community active:

Figure 2 Figure 2 :

 22 Figure2gives an overall view of the architecture, functionalities and services which have been implemented in the SHARE Web Space.

Figure 3 :

 3 Figure 3: Graphical exploded view of the Technical Features criteria subset for the Build Tools software family. Comparison of 6 evaluated products.

Figure 4 :

 4 Figure 4: A subset of the criteria tree. Each level has an associated graphical plot.

Figure 5 :

 5 Figure 5: Exploded graphical view of two levels of the criteria tree.

Functional coverage, i.e., how a given product is able to fit context requirements. This is the "dynamic" part of the evaluation model based on context-related criteria. ii. Users risks, which do not depend on the context and which are weighted

		b. software context (e-health, mission-critical,
		nomadic and others);
		c. functional grids to describe each domain;
		d. Free/Libre/Open Source Software (FLOSS)
		licenses classified according to Ownership,
		Virality, Inheritance criteria;
		e. Communities, which have been identified
		in	insulated	developer,	Group	of
		developers, developers organization, Legal
		and, commercial entities.
		2. Evaluation, during which each version of the
		identified software products is actually evaluated.
		To this aim, each software is described by means
		of an evaluation sheet containing (i) a "software
		id card" reporting general information about the
		software, and (ii) software evaluation criteria.
		These are further divided in:
		i. during
		evaluation step
		iii. Providers risks, i.e., risks viewed by a
		provider of services based on FLOSS.
		3. Qualification, aiming at defining filters that
		translate requirements and constraints related to
		the selection of FLOSS in a specific context. Here
		is where the user's content is qualified. Filters are
		possible	on	software	identity card	or
		functionalities, as well as on users and providers
	, the process is composed of	risks.
	four steps:	4. Selection which allows selecting software
	1. Definition, aiming to define the basic concepts enabling the evaluation: a. software families;	fulfilling user's requirements and comparing different products belonging to the same family. The QSOS methodology proposes two alternative ways of accomplishing this fourth step: strict
		selection, that

means to eliminate software which Figure 1: SHARE and QSOS methodology

  These criteria are suitable for evaluating any OSS or system regardless of the operational field of interest. Specific criteria, instead, have been introduced to perform custom evaluation: they take into account different parameters for each evaluation context.

	do not fulfil users' request, and loose selection
	that requires, instead, to use filters built in step 3
	to "weight" choices. This step results in the
	comparison of products in a given family.
	The idea of "context" represent the added value
	provided to the existing QSOS methodology. It
	provides a means to specialize evaluations with
	respects to the particular area of interest and
	encompasses the definition of common and specific
	criteria which have been used to perform evaluations
	and comparative analysis.
	The common criteria provided by the QSOS
	methodology	follow	a	nested	structure
	encompassing four groups at the first level ( intrinsic
	durability,	industrialized		solution,	technical
	adaptability and strategy) .

• Collaborations with other research project and initiative raised during the project life will be kept alive (hArtes [HRT], FLOSSMETRICS [FSM], QUALIPSO [QUA] among the others).

  Interesting collaborations rose with MORFEO[MRF], which is a significant open collaborative framework where industries and academic organizations have a common point to share knowledge as well as software. Through the fusion with this large community, interesting input will come to the SHARE project. With particular emphasis on embedded systems, the SHARE project devoted great attention to ARTEMIS 11 initiatives which aims to help European industry to consolidate and reinforce its world leadership in embedded computing technologies.

The natural evolution of these actions will be the creation of larger communities with focus on Open Source and embedded systems, as well as the formulation of larger research projects that will consider the SHARE experience as a starting point.

  Dissemination actions performed during the project, aimed at enlarging the network of contacts will be exploited and particularly helpful to create new partnerships and research consortia.• The

creation of a network of excellence aiming at fostering cooperation among research groups and spreading funding opportunities awareness.

  Since there are many research projects and initiatives focusing on open source, both in academia and industry, creating larger alliances is a powerful way to spread knowledge even across different fields and scientific communities, as well as to increase raising the attention on common topics and needs.•

The creation of joint collaborations with standardization organizations. This will give a wider visibility to the SHARE outcomes, reaching all the practitioners in the field of OSS.

http://www.share-project.eu/index.php?option=com_osbenchmark&Itemid=15

http://www.share-project.eu

http://www.joomla.org/

http://www.sesm.it

http://www.stm-products.com/

At submission time the event still to come. Reports and results will be made available on the Share platforms after April 9th, 2010-03-27

At submission time the activities are still on going. Results will be made available on the Share website at the end of the project(April 30th, 2010)

http://osek-vdx.org

Code Sharing Code Sharing Code Sharing Code Sharing Community Community Best practices and Recommendations Best practices and Recommendations Code Sharing Code Sharing

Community Community

Best practices and Recommendations

Best practices and Recommendations

KNOWLEDGE FRAMEWORK KNOWLEDGE SHARING PROJECT MANAGEMENT

SHARE WEB SPACE CORE

Review & References

Review & References

Collaboration lab

Collaboration lab

Benchmarking Suite

Benchmarking Suite

Review & References

Review & References

Collaboration lab

Collaboration lab

Benchmarking Suite Benchmarking Suite


Antonio Monzón
email: antonio.monzon@military.airbus.com

Bi-directional Mapping between CMMI and INCOSE SE Handbook

Keywords: CMMI, INCOSE SE HB, ISO15288

The complex normative context in the field of industrial areas like aerospace and defense require appropriate guidance to distinguish the purpose of the standards involved and specially the overlapping and borders among them. The aim of this paper is to present a practical mapping between CMMI-DEV v1.2 (ML3) and INCOSE Systems Engineering Handbook v.3.1. Summaries with bi-directional mappings are provided as well as a detailed mapping.

Introduction

Industry has a growing interest on standardization of those activities under the scope of the Systems Engineering discipline. Several initiatives have been put in place since late 60s from the Mil-Std-499 until last version of ISO/IEC15288:2008 [5] (see Fig. 1).

In parallel to this international standardization mainstream, the International Council on Systems Engineering (INCOSE) has produced a handbook [3] conceived as the practical guidance reference for Systems Engineers. In addition to these initiatives, the Software Engineering Institute (SEI) has promoted over the last decade the de facto standard CMMI [1][2] (Capability-Maturity Model for Integration) as a reference framework to be used by organizations developing systems (embedding or not software) to measure their maturity as productive entities. Although CMMI comes from the software world, this version "for integration" considers general aspects of systems production, regardless they apply particularly to software. This movement towards system's world makes CMMI an interesting complement of SE standards to cover those process management aspects these other standards are lacking. In such a complex normative context it is always difficult to know which are the borders among norms and references, and therefore practical mappings In addition to the mapping summary in both directions, a detailed mapping is provided in the annex, and those aspects not covered in one reference with respect to the other are also highlighted.

Assumptions

INCOSE SE HB v3.1 has no formal processstructure (like ISO12207 or ISO 15288). The handbook is organized in a documental way by processes and within each process there are no explicit subsections per activity. Activities are included in the process diagrams and described in a specific section with no additional breakdown. The comparison has been made under the assumption that process diagrams included in the HB can be interpreted as follows: Process X uses inputs, produces outputs and has some main activities inside. The activities in the diagrams are the basic elements to be compared. The rest of elements used in the HB have been considered as informative. From CMMI side the model elements used for comparison purpose are the specific practices (SPs). Generic practices have not been considered explicitly as their main role is to handle the institutionalization of SPs. The percentage values provided in the comparisons represent the number of atomic elements in one reference covered in the other with respect to the total number of elements within the aggregated element (process in the HB or SPs in CMMI).

It is also worth to mention that not all HB processes have counterparty in CMMI (e.g. Operation, Maintenance or Disposal). This partial mismatch is reasonable as long as both frameworks are conceived for different purposes. The opposite statement is also true: CMMI considers additional aspects not taken into account in the HB (e.g. training, process improvement or measurement). This mapping was performed against version 3.1 of SE Handbook. Version 3.2 [4] (published in January 2010) includes a high level mapping with CMMI for Development model not present in the previous version.

CMMI to INCOSE SE HB Mapping

As a general summary, an organization holding a quality system aligned with the INCOSE SE HB will cover about 60% of the CMMI L3 specific practices. With respect to the CMMI L2 practices, the coverage is slightly better: 66%. A first observation from this mapping is that there are very few CMMI process areas for which all the specific practices are covered in the HB (only Supplier Agreement and Technical Solution are fully covered). This fact contrasts with the opposite mapping, where most of the processes in the HB are fully covered by CMMI practices. The results per process area can be summarized as follows (see Table 1):

Measurement and Analysis (MA) is not covered at all. In fact, there is no explicit measurement process in the handbook. The HB is not oriented to process improvement but to process definition.

In Organizational Training (OT) only 14% of the practices are covered. Therefore the HB does not focus particularly on skills improvement. Although PP and VER show a relatively high coverage rate, they should be in the upper band due to their impact in the project performance.

The main reason why VER is not in the upper band is the lack of peer reviews in the HB. A special emphasis is given to this verification technique in CMMI.

Concerning PP, the reason for its lack of coverage is that CMMI requires reviewing and tracking the plans.

INCOSE SE HB to CMMI Mapping

From the analysis performed, the main result extracted is that an organization with a CMMI accreditation of maturity level 3 covers 76% of the activities included in the INCOSE SE handbook. Taking into account this figure, it could be concluded that CMMI is wider than INCOSE HB, but it is also true that not all the processes in the HB are considered in CMMI. Another interesting observation from the Table 2 is that most of handbook processes are fully covered by some practice of CMMI: 16 out of 25 processes. The results per process type can be summarized as follows (see Table 2): Operation, Maintenance, Disposal and Supply are not treated in CMMI at all. The reason can be that these processes are typically related with the operation of a system once created. CMMI for Development is more focused in the production cycle, regardless the operation of the system.

Comparison with ISO15288

In Aerospace and Defense field, additional standards are required to be fulfilled in order to get systems certified for operation. Standards like SAE ARP4754 or RTCA DO-178B are some examples of such norms. The quality systems of the organizations in this field must be compliant in addition with special regulations which include aspects related with the operation of the systems. Particularly the AQAP160 is a NATO norm, applicable at software level, which is composed by a combination of ISO12207, ISO9001 and specific military requirements, is largely compliant with ISO15288 requirements. The additional processes Supply, Operation, Maintenance and Disposal are present in this norm. Additionally, the Tailoring Process is used to instantiate the quality system to particular projects.

ID

Figure 1 .

 1 Figure 1. Systems and Software Engineering Standards.

  Project Management and Causal Analysis & Resolution) have been excluded from this analysis because they are very much concerned with the performance of an organization in contrast with the process orientation of the low maturity process areas.

  As expected, the better coverage rate corresponds to the technical process areas and the worst to the process definition and organizational process areas.

	The rest of process areas are largely covered by
	the HB, and the discrepancies are only
	attributable to the particularities of CMMI:
	o Configuration Management (CM): 86%
	o Decision Analysis & Resolution (DAR): 83%
	o Project Monitoring & Control (PMC): 80%
	o Requirements Management (RM): 80%
	o Validation (VAL): 80%
	o Product Integration (PI): 78%
	It is also worth to highlight the aggregated coverage
	level per process area types:
	Process Management: 40%
	Project Management: 59%
	Engineering: 71%
	Support: 52 %

Table 1 .

 1 CMMI to INCOSE SE HB Mapping.

	The rest of technical processes are fully	Finally, it is interesting to remark that the coverage of
	covered. Although no explicit process area exist	the INCOSE SE HB by CMMI L3 is of 92%, if the
	in CMMI for transition activities, they are covered	system operation aspects are excluded. This
	in several practices of the PAs Technical	basically means that CMMI can be considered
	Solution, Product Integration and Validation.	largely compatible with the system development part
	Concerning Project Processes of the HB, the	of the INCOSE SE HB.
	comment is that only one aspect of the Project
	Assessment is not considered in CMMI:
	monitoring of new technologies. The rest of
	aspects are fully covered.
	Main discrepancies fall on the Enterprise and
	Agreement Processes of the HB. Although
	percentages are diverse, the discrepancy is just
	one activity per process. The overall figure has
	been calculated with respect to the total number
	of activities within each process:
	o Resource Management: 88%
	o Acquisition: 88%
	o Enterprise Environment Management: 83%
	o Investment Management: 83%

Table 2 .

 2 INCOSE SE HB to CMMI Mapping.

One of the main differences between versions 3.2 and 3.1 of the INCOSE SE HB is the one-to-one mapping in their sections (see Table 3). The last version of the HB has in fact the same process structure than the ISO15288 (and the ISO12207 as well).

Table 3. ISO15288 vs. INCOSE SE HB v3.2

Conclusions

The main conclusion obtained is that CMMI and INCOSE SE HB are not competing but complementary approaches. The INCOSE handbook fully covers all the activities relevant from the perspective of the Systems Engineer practitioner. It is not specially focused on process measurement or improvement. On the other hand, CMMI is a reference framework for process improvement purposes. A recommended approach could be to define a quality system according to the recommendations provided by the INCOSE HB and complement it with additional processes to cover the process improvement aspects. The alignment with CMMI is strongly recommended when a mature organization aims to improve its efficiency while creating their products. Nevertheless, CMMI can be counterproductive in very immature organizations as it involves many control activities which could make fail the whole productive process.