Jakob Engblom

Guillaume Girard

Bengt Werner

Testing Embedded Software using Simulated Hardware

Keywords: Verification, Validation, Simulation, Business models

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Simulation as a tool for testing and debugging software has a long history going back to the very first electronic computers [START_REF] Gill | The Diagnosis of Mistakes in Programmes on the EDSAC[END_REF]. Simulating a system has always carried the advantage of increased insight and flexibility, at a cost in execution speed and timing fidelity visavi the real machine. However, until recently, use of simulation technology for largescale embedded systems software development and testing has been fairly limited. Hardware designers for processors, supporting chip sets, systems-on-chip, and servers have always made use of simulation in order to model hardware early. Simulation is used for performance evaluation, to test various ideas for implementation, and to validate that a system works as intended [START_REF] Magnusson | Simics: A Full System Simulation Platform[END_REF] [START_REF] Alameldeen | Simulating a $2M Commercial Server on a $2K PC[END_REF]. Providers of software development systems (especially for 8-bit and 16-bit processors) have always provided instruction-set simulators (ISS) for the target systems. However, such solutions have been limited to only simulating the processor and not the surrounding hardware, making them suitable for simple initial software test but not for running operating systems or software that interacts with hardware. Initial firmware bring-up and ports of operating system codes to new embedded and other computers is quite often performed using simulation tools, as the real hardware is typically not available early enough [START_REF] Stetter | IBM eServer z990 improvements in firmware simulation[END_REF] [START_REF] Connell | Early Hardware/Software Integration Using SystemC 2.0[END_REF]. Some software work can be performed on the developer workstation using API-level simulations of the embedded operating system [START_REF] Möller | A Simulation Technology for CANbased Systems[END_REF]. For final verification of functionality, it is necessary to use the actual binary that would be used in the real system, which is not possible in an API-level simulation. Overall, however, for the volume work of embedded software development, embedded developers have relied on development boards and instances of the real target hardware boards 1 . We introduce a simulation tool that can replace the use of hardware to a large extent, by providing a simulation model that is faithful enough that all software for the target can run on it, and fast enough that it can be used in daily work. This paper describes this tool, Virtutech Simics, and how it is applied to a number of tasks in embedded software testing and development, with a focus on testing.

Simics

Our simulation tool, Virtutech Simics (see www.virtutech.com), is capable of simulating large computer-based systems and complete networks. The simulation approach used is full-system simulation, where the processor, memories, peripheral devices, and environment of a target computer system are all simulated in such detail that the target software cannot tell the difference from a real target system. The full-system model runs the same binary software as would run on the real target, including device drivers and firmware, as illustrated in Figure 1. At its core, Simics is an event-driven simulator where processors are treated specially for performance reasons. All I/O and other peripheral devices on a board (and thus all interconnects between boards in a system) are simulated in a transaction-based style. This means that accesses to devices are handled as a synchronous unit, rather than simulating the actual bus traffic required to perform the request in the real hardware. Networks are simulated on a packet level, where entire packets are sent and received as units. This simplification is key to gaining simulation speed.

Simics is based on very fast instruction-set simulators (ISS), which are bit-accurate in the results of all instructions, including supervisor-level operations, floating-point operations, and model-and hardwarespecific registers. The use of an ISS provides the user with total virtualization of the target system; For example, Simics can execute operating systems and applications for PowerPC or MIPS targets on an x86 PC or a SPARC workstation, as illustrated in Figure 2. Simics handles endianness and word length differences, and can simulate a 64-bit little-endian system on a 32-bit big-endian host. Simics does not require any special hardware or particular operating system; it is a pure software application that can run on regular Solaris, Linux, and Windows workstations and servers. This also makes the virtual (simulated) hardware future-proof. As illustrated in Figure 3, the same simulated hardware will continue to be available as host machines change and evolve, as long as Simics is available for the new host. The behaviour of the simulated hardware remains the same over time, allowing for maintenance and development of code for a platform for extended time frames. Unlike real hardware, the simulated hardware will not break or become unavailable.

Simics is designed to be a fast simulator, and can currently achieve speeds exceeding 2000 MIPS when running single-processor workloads on top of full simulated systems with a real operating system (measured when simulating a single PowerPC 750 processor running on a 2GHz Opteron PC). Such high execution speed allows for real workloads to be run in simulation, including operating systems, network stacks, and complex applications. This is a crucial enabler for using simulation in testing large real-world codes and real-world test suites, in a completely virtual environment [START_REF] Magnusson | The Virtual Test Lab[END_REF]. Simulation in Simics is guaranteed to be deterministic -from the same initial state, the simulation will simulate the exact same execution path every time the simulation is run. If variation is desired (in order to test multiple code paths that depend on target timing, for example), it can be provided by adjusting simulation parameters for a particular run. Simics supports checkpointing, where the complete state of a system is written to disk. This checkpoint can later be restarted at the precise instant where it was saved. Together with determinism, this allows for executions to be repeated any number of times, using any workstation running Simics. When simulating multiple processors and/or machines (for example, in a distributed system), Simics provides global synchronization and stop. If one part of one machine is stopped, the entire simulation is stopped. This makes it feasible to single-step interrupt handlers and perform deterministic debugging and analysis of multi-processor and distributed systems [START_REF] Engblom | Developing Embedded Networked Products using the Simics Full-System Simulator[END_REF]. Scripting in Simics is very powerful, with a full Python language interpreter as well as an extensible command-line interface. Scripts can react to output from the simulated machine and to events inside the simulated machine (breakpoints, exceptions, device accesses, etc) and provide scripted intelligent input to the simulation. All target state is accessible without probe effects, including information which is hidden and hard to get to on real hardware, such as TLB tags and the contents of supervisor-level registers. This ability to observe also provides the ability to trace and log everything that happens in the system. Tracing can be used to profile and perform code coverage analysis without having to instrument the target code. Even very detailed code coverage analyses like decision and condition coverage can be implemented transparently to the code being executed; it is all handled by looking at the execution trace and noticing which instructions are executed (and which are not). All target state can be manipulated. If the state can be observed it can also be changed. For example, for fault injection, transient and permanent faults can be easily simulated [START_REF] Myhrman | Studying Fault Injection in WCDMA Base Station Processors Using Simics Simulator[END_REF] [START_REF] Bastien | A Technique for Performing Fault Injection in System Level Simulations for Dependability Assessment[END_REF]. Simics supports source-level debugging of software running on the simulated machine, including firmware and operating systems. Any code can be debugged, as the simulator has complete control over the state and execution.

Network Simulation

As noted above, Simics simulates not only individual machines but also networks of machines. Each machine in this instance runs a complete software stack, including operating-system drivers for the network devices. In network simulation, network traffic is sent as entire packets. It is possible to inspect (and modify or destroy) network packets as they travel across the network, as illustrated in Figure 4. There are no limitations as to how machines can be combined in a simulated network: it is possible to combine many different machines of different types and speeds, and Simics simulates the relative execution speed of the different machines. For more information on Simics network simulation, we refer to [START_REF] Engblom | Developing Embedded Networked Products using the Simics Full-System Simulator[END_REF].

Simulation Timing

In order to gain simulation speed, Simics simplifies the timing of the target system. In the basic model, processor instructions are assumed to have a fixed execution time, and device accesses provide simple time models for when transactions complete. The function is identical to a real machine (which is necessary in order to run real binaries), but the timing might be different. This makes Simics suitable for testing the functional correctness of code, and coarse-grained timing. Simics is not intended to analyze or predict the precise cycle timing of processor pipelines or caches. Since building precise timing models of real hardware is very difficult, such detailed timing analysis has to be validated on the real target platform [START_REF] Engblom | On Hardware and Hardware Models for Embedded Real-Time Systems[END_REF]. Simics provides a single global virtual time. All processors and device models are synchronized to this time base, across processors and machines in a simulated network. The progress of this time in relation to the real-world time is variable. It is quite possible for a simulation of a slow system to be many times faster than the real world. Also, if a system is mostly idle, simulation can run very quickly. For example, we have run a network of 100 small sensor nodes2 at five times real-world speed. In the tested case, the software tested had the sensor nodes spend about 99% of their time sleeping, making it possible for the total simulation to run faster than the real world, as very little processing was done in each node. The simulated system might also run slower than a real system if the simulation is heavily loaded or contains many processors. It is obviously very hard to simulate ten high-speed processors as fast as real-time on a single host processor. In the most extreme case, simulation execution can be paused for some reason, in which case time does not advanced at all. Real-time simulations where simulated boards are mixed with real-world systems are possible, as long as the simulated system is slow enough that the simulation will always run faster than real life. Then, the simulation can be stalled when it runs ahead of real-time, creating a simulation which runs at realtime speed [START_REF] Engblom | Time Accurate Simulation: Making a PC Behave Like a 8-bit Embedded CPU[END_REF].

System Model Creation

A key part of using Simics for a particular target system is creating the model of the target hardware. As noted above, Simics models the full system hardware, and thus more than just a processor ISS is needed in order to create a working simulated system. First, any components already available for Simics can be reused. In processor arena, Simics has fast models available for all the most common embedded processor families, including PowerPC, MIPS, MIPS64, ARM, x86, and SPARC, along with system controllers, FLASH memories, IDE disks, I2C buses, network controllers, serial ports, timers, and other common components. Second, any new components present in the system in question have to be modelled. This task can be performed by Virtutech, the Simics user, or by a third party. In order to speed and simplify the modelling of new systems, a domain-specific language called DML has been created for writing Simics device models. DML is many times more productive than using C or C++ for model construction.

Simics Uses

Simics is currently in use at many commercial customers and more than 1000 universities worldwide. Simics has been used for a large spectrum of activities, including computer architecture research & design, firmware development and test, operating system ports to new hardware, application development for distributed systems, and system testing for telecoms systems. Simics has been a commercial product since 1998. Version 3.0 shipped in October of 2005 [START_REF] Mcgrath | Virtutech system-level simulator features Hindsight technology[END_REF], and development and enhancement of the product continues. The technology has proven very flexible and scalable, being used for simulation of everything from single aerospace boards to networks of storage boxes to multiprocessor database servers and rackbased telecom systems with tens of processors. As an example, Figure 5 shows a screenshot of Simics running a network containing one x86-based PC with Linux, one PowerPC 750-based embedded computer with Linux, and one PowerPC 440GP-based development card running Linux. Each target machine has its own text console available for interaction, and there is also the main Simics window from which the simulation is controlled. In the remainder of this paper, we will primarily address the uses of Simics in testing of embedded software, with an eye towards general embedded software development.

Unit Testing

A straightforward use of Simics is to replace real target hardware for unit testing of programs. This is done either with or without deploying an operating system on the target machine, depending on the characteristics of the code and system being tested. In the simplest case, a simplified target system is setup containing only a single processor ISS with memory (to store code and input data), and a special test output device. The program under test has to be able to run on a naked machine with no operating system, and to write its results to a "port" in memory. Typically, the input data is loaded into the simulated machine together with the program, while the result is a stream of characters which is collected into a file on the host machine, and then analyzed in a separate post-processing phase. The proposed setup is illustrated in Figure 6. In this setting, simulation is used as a replacement for development cards used to perform unit testing of binaries for a specific target architecture, without using any operating-system calls or particular input or output. The goal is to ascertain the correct function of a small unit of compiled target code, using some a test bench compiled into or with the program.

An alternative setup is to run an operating system on a complete simulated machine, mimicking the setup of a real development board. In this case, the simulated target will be connected using simulated networking to the driver application on the host, and interfaced and used just like a real development board. If the test cases use operating-system calls, such a setup is obviously necessary. An illustration of this setup is given below in Figure 7.

Benefits of simulation

Using simulation for unit testing has several benefits compared to using real hardware. First of all, loading and executing tests is more convenient than on real hardware. Code to be tested is put in place when the simulation starts by writing it directly to simulated memory. There is thus no need to download code over a serial line or network and running a monitor on the target.

Function Testing

Broadening the scope from unit tests, the next step is typically to test a complete function consisting of several software units working in concert. Here, we assume that such testing is carried out for a single target board at a time (testing networked systems in a network configuration is addressed later).

For function testing, Simics is used with a full simulated machine running an operating system (and various supporting libraries or middleware systems, where such are employed), as illustrated above in Figure 1. Thus, the program under test is expected to be using OS and library API calls. Interrupt occurrences and OS scheduling will behave as on a real machine, allowing the test of tasks prioritization and execution modes. Memory management and usage can be tested, as well as the interaction between parallel tasks or threads in a system. Note some care regarding timing is needed when mixing real-world testers with simulated machines. Timeouts for determining when a crash has occurred in the program under test are usually set in terms of real-world time, and this is not always appropriate for a simulated system. As noted above in Section 2.3, the timing of the simulation is different from that of a real system. Thus, for best results, testing tools should use a time feed from the simulated system to determine time-out conditions based on the virtual time of the simulation [START_REF] Engblom | Developing Embedded Networked Products using the Simics Full-System Simulator[END_REF].

Benefits of Simulation

All the benefits of simulation that apply to unit testing also apply to function testing. In addition, some new benefits apply.

System Testing

Beyond the testing of individual units or functions, simulation can also be used to test the functionality and correctness of a complete system involving multiple processors and networked boards [START_REF] Magnusson | Simics: A Full System Simulation Platform[END_REF][3][7] [START_REF] Engblom | Developing Embedded Networked Products using the Simics Full-System Simulator[END_REF].

In this case, full-system models are constructed for all nodes in a network, and several target board models connected using a simulated network, as discussed in Section 2.2. The machine models themselves are the same as used for function testing, but typically using multiple instances of a particular type of simulated machine, with identical or different software loads.

In system testing, the simulation is used to execute a complete real software load, including multiple programs running in parallel and software relying on communicating with other machines in a network. A complete distributed system is easily simulated, providing a realistic environment for the software on each node in the system. For embedded systems controlling physical systems, mechanical or physical simulations of the environment can be interfaced to the Simics simulation of the computer system. Such models rely on the virtual time in Simics, computing the evolution of the environment in the same time-domain as Simics executes the software. In order to enable the simulation of large systems, Simics can distribute the simulation of a network across multiple host processors or host machines. This lets large simulation take advantage of added CPU cycles and memory resources to handle really large simulated systems [START_REF] Engblom | Developing Embedded Networked Products using the Simics Full-System Simulator[END_REF].

A recent example of system testing with Simics is the simulation of Iridium satellites. Using virtual satellites, software is developed and tested on the ground in an environment corresponding to what it would meet in an actual satellite in orbit [START_REF] Waters | Iridium Simulates Space Software with Simics[END_REF].

Stimulus

When doing full-system testing, the problem of how to inject stimulus to the system becomes very important. The goal is to execute the software in use cases taken from the real world, and this is realized by providing relevant stimulus as input to the system.

The system configuration itself can be the stimulus. One example of this is a real-world case where Simics was used to test self-configuration of large networks by simulating several hundred network nodes connected in various ways. Booting up all the machines on the network tested that they correctly elected leaders, obtained network addresses, and established communication.

A simulation of the physical environment provides good test input for control software, as discussed above.

In network simulation, stimuli can be provided by traffic generators in the classic sense that provide a stream of packets from a given distribution. It is also possible to use behavioural models of network nodes, i.e. small programs describing the network behaviour of a node without all internal details. Existing network testers can also be interfaced with the simulated system. Figure 8 shows how these types of stimuli are combined with regular Simics simulations running complete software stacks. Since simulation provides full control over all data exchange and input and output of a system, it is also possible to record and replay input and output (network traffic, user interaction, other external events) in a simulation. This makes it possible to perform isolated error diagnosis on a single machine in a multi-machine setup, by recording the whole system and replaying the external events for a single machine in the system. Figure 9 illustrates this idea. Machine 2 in the threemachine network has all its input and output recorded, later to be replayed using just a simulation of machine 2. This can also be done for all machines simultaneously, allowing for analysis of the networked behaviour of each machine in isolation following a single networked simulation run.

Simics Simics Simics Simics

Fault Injection Testing

Simulation can also be used to test the behaviour of a system in the presence of faults [9][10]. Fault injection in simulation requires the creation of appropriate fault-injection modules inside the simulated system, modules that are attached to buses, networks, memories, devices, and processors, and perform the actual injection of faults. These fault-injection modules have to be tailored for each component, but it is easy to reuse models for memory components, processors, networks, and similar common devices. In addition, it is necessary to establish a way to specify which faults occur when, so that fault injection campaigns can be specified and executed using a single point of control. Usually, a master fault injection module is used which reads a file specifying which faults are to be injected where and when. Some faults can only be reasonably studied on real hardware, for example electrical effects of pulling cables and cards out of racks and radio disturbances caused by interference from electrical systems. Similarly, radiation effects on chips have to be tested in irradiation chambers. However, once physical studies of fault behaviour have been performed, simulation can be used to replicate the effects on the computer system. Data from the physical experiments, such as the frequency and nature of transient and permanent faults, can be used to guide the faults to be tested in simulated fault injection campaigns. Fault injection in simulation is typically used to check that fault detection and recovery mechanisms work as designed. This might mean testing voting mechanisms or redundant failover, or just that a system correctly logs errors. Fault injection can also be used to diagnose problems -it allows an engineer to test a hypothesis as to which hardware problem causes a particular software error.

Benefits of Simulation

Simulation has a range of advantages for fault injection studies compared to using real hardware. Simulated fault injection is non-destructive. The system under test does not suffer permanent damage from being tested with faults, unlike physical experiments where hardware components are quite often damaged (intentionally and unintentionally) in testing. Simulation offers repeatability of fault injection, as replaying the same fault injection script will inject the precise same faults and the precise same point in time. Achieving precise repetition with physical fault injection is very difficult. Fault injection in simulation will allows increased fault-injection coverage, as faults are easy to program and introduce, and fault-injection campaigns are simple to execute; just run the simulator, no special lab needed. Since faults are easy and precise to program and tailor, corner-case testing is enabled. With the help of simulation, it is thus possible to script and repeat known hard cases in the system, such as multiple board failures close in time and babbling idiot failures.

Business Aspects

Using simulation for testing in a real production environment requires considering the business aspects of simulation as a testing tool.

Costs

The costs involved in introducing simulation consist of four parts. First, the development of the simulation framework, which should not be needed as it can be bought off-the-shelf. Second, the cost of developing the necessary hardware models to model the system or systems of interest. Depending on the nature of the system, many standard parts are typically available from the simulation vendor. Developing new standard parts of custom parts requires a one-time investment by the vendor or the customer. Third, there is a licensing and continued support cost associated with the simulation tool. Fourth, there is an initial need to train users on the simulation tool and introduce the simulation into the workflow. As we have already noted, simulation in testing can often slip into an existing workflow, using the same methodologies and tools as used with real hardware. Full-system simulation really does provide virtual hardware that can be used instead of real hardware.

Benefits

The benefits from introducing simulation are both direct costs savings, and indirect economic benefits from faster development and better quality products. First, buying a single simulation license is usually cheaper than buying a single development card or custom product board. Since several boards can be simulated in a single simulation instance, there are obvious cost reductions from simulation. These cost savings can either translate to lower overall costs or more systems deployed at the same cost. Simulation also provides increased flexibility. There is no fixed inventory of available boards. Any users can setup any board or combination of boards, without allocating physical boards. Any workstation can be used to run target code for any target system, greatly improving hardware availability for developers and test engineers. There is no need to ship hardware around to supply each developer with the particular hardware needed at a given moment. Second, simulation makes it possible to develop software in parallel with the hardware, and to deploy more hardware earlier in the development cycle. This translates to a shorter time to market for new products, as the software teams can start working earlier than if they have to wait for hardware. Removing the dependence on hardware also means that more developers can work in parallel on the same project, as hardware availability is no longer a bottleneck. Third, simulation brings a number of technical benefits for the actual execution of tests, as detailed above. These cumulative benefits lead to better product quality thanks to more and better testing and easier fault identification and correction. Shorter setup times and turn-around times, as well as more convenient execution environment makes test engineers more efficient. Our experience is that the cost of introducing a simulation are usually far outweighed by the advantages it offers, especially for companies building and integrating complex embedded systems based on custom hardware and with large value deriving from the software running on these systems.

Summary

This article has presented the use of simulation in general (and the Simics simulation product in particular) for testing and developing embedded software. Simulation does not replace all testing on real hardware, but it offers a very good complement to real hardware, with potential for great improvements in development and test process efficiency and overall cost. Simulation can be used within existing work flows and tool environments, making such benefits quite easy to realize. We have reviewed a number of typical usages of simulation for embedded software testing, and for each case, how the simulation would be setup and the resulting benefits.

Glossary

Figure 1 :

 1 Figure 1: The concept of full-system simulation

Figure 2 :

 2 Figure 2: Simics provides full virtualization

Figure 4 :

 4 Figure 4: Network simulation with Simics

Figure 5 :

 5 Figure 5: Screenshot of a Simics session

Figure 6 :

 6 Figure 6: Setup for simple unit testing

Figure 7 :

 7 Figure 7: Network test on a single machine

Figure 9 :

 9 Figure 9: Record and replay in a network

  The testing can be driven by test bench code loaded with the program under test, or it can use external testing tools that communicate with the program. In both cases, testing is performed in the same way it would be on a real target board. The simulation does not necessitate changes to methodology or tool use for testing, it just changes the means used to execute the code. For example, a networked target machine can be tested by connecting existing network testing tools to the simulated network, communicating with the simulated board just like with a real board. Both in the real and simulated cases, the testing tool is given a network address to communicate with (typically, an IP address + port number), and is thus oblivious to the nature of the target (whether it is real or simulated). The test program and simulated board with the program under test can both be run on the same host workstation, as illustrated in Figure7.

	Program under test
		Network Network
	OS	testing testing
		program program
	Simulated Target HW	Network simulation
	Simics Simics Simics Simics	Simics Simics
	Host operating system Host operating system Host operating system Host operating system
	Host hardware Host hardware Host hardware Host hardware

In this paper, we use the term "board" to mean any complete computer unit used in an embedded system. This can be a rack-mounted card, a standalone box, or some other package.

A "Telos Mote" is a small wireless sensor node containing an 8Mhz 16-bit processor[14].

Acknowledgement

The authors would like to thank all our colleagues at Virtutech for having created a great tool and applied it to real-world problems.


Bernard Guichoux

François Dubuc

CMMI -AN ANSWER TO THE SOFTWARE CRISIS?

Keywords: Software Crisis, CMM, CMMI, processes, Systems Engineering 1. Software Crisis

As regularly mentioned in the current "software crisis", the software must be developed while the system definition is not yet frozen. In 2001, EADS SPACE Transportation started a CMM approach to improve its software development process. But even if the projects became more deterministic, the software developments still remained depending on system evolutions, and continued to appear very costly. So it was decided to use the CMMI maturity model, adapted to our industrial domain and now applied by many companies. We really think that the CMMI approach allows to improve both the system and software engineering processes:

-The CMMI management processes are mandatory to efficiently plan, control and manage complex projects. -The technical processes must contribute to well organize the project. The CMMI project has just begun, so the expected benefits and savings are not yet visible. In parallel, other improvements are also expected from the software R&D activities presented in the paper.

The paper will present:

an overview of the "software crisis", -a presentation of SW-CMM and CMMI, -The main improvements achieved through the SW-CMM project and expected from the CMMI project, -The software R&D roadmap at EADS SPACE Transportation.

-53% of those were completed cost an average of 189% of their original estimates. -of those 53%, only 42% had their original set of proposed features and functions. -Only 9% of the projects were completed on time and on budget. Spectacular examples or failure are well known as the crash of Mars Polar Lander in December 2002. To improve software development in spaces programmes, ESA crystallised again this problem by organising a panel in the framework of the DASIA conference 2001 in Nice; in the same time CNES initiated a similar analysis in their projects. After, several workshops were performed by the agencies with industrials. The accepted findings [START_REF]Astrium contribution to the analysis about "Solving the SW-Crisis[END_REF] [START_REF] Hjortnaes | ESA Software Initiative[END_REF] is now that the software engineering process has to be itself mastered, but also that software is often a part of systems engineering process and so this process must be mastered as well. The software crisis in not only a space / aerospace problem, but affects the total software industry. But the effects on the space software products are very critical and that is why this problem is especially followed by agencies as CNES, ESA, NASA or NASDA. Indeed, the following space projects critical aspects were especially formally identified:

system complexity is underestimated, -co-engineering tasks between system team and SW team are not well coordinated and reviewed, -SW development life cycle is more an incremental life cycle than a classical "V" cycle, -new methods and tools are not used enough to allow automating the SW production and test, -Organisation and role of SW engineers are not so clear within the projects. So, several opportunities for improvement are coming from these previous topics:

use of SW modelling methods to reinforce systems engineering process, -participation of SW engineering experts in systems engineering team to address the software system architecture and stable software requirement baseline, -better definition of incremental software life cycle and definition of SW architecture that allows incremental development. Maturity models can be particularly useful to implement such improvement opportunities.

SW-CMM and CMMI

The Capability Maturity Model (CMM) initiative [START_REF] Carnegie Mellon | Capability Maturity Model for Software[END_REF] started at the Software Engineering Institute (SEI), in November 1986, in order to develop a process maturity framework that would help organizations to improve their software process. This effort was initiated in response to a request to provide the U.S. federal government with a method for assessing the capability of its software contractors.

The initial release of the CMM, Version 1.0, was delivered in August 1991 and then used by the software community. This current version of the CMM, Version 1.1, delivered in February 1993, is the result of the SEI studies and of feedback from the software community.

The SW-CMM

The CMM is a baseline of key practices that should be implemented by any entity developing or maintaining a product which is completely or partially software. The CMM includes also an assessment approach. The CMM can help to:

-Provide guidance for improving processes performance and capability, -Provide a reference point for assessing current practices, -Provide a framework for benchmarking among and across organizations.

The CMM assessment model includes five levels of software process maturity: initial, repeatable, defined, managed, and optimizing. The next level can only be reached when a previous level has been satisfied.

The five levels of software process maturity, based on an organization's support for certain key process areas (KPAs). Level 1 (Initial) describes an organization with an immature or undefined process. Level 2(Repeatable), Level 3 (Defined), Level 4 (Managed), and Level 5 (Optimizing), respectively, describe organizations with successively higher levels of software process maturity.

The CMMI

After the Software CMM model and its successful adoption and use in many domains, other CMMs were developed for other disciplines and functions such as Systems Engineering, people, integrated product development, software acquisition, and others.

Although many organizations found these models to be useful, they also struggled with problems caused by overlap, inconsistencies, and integration. Many organizations also confronted conflicting demands between these models and ISO 9001 audits or other process improvement programs.

The CMM Integration (CMMI) Project [START_REF] Carnegie Mellon | Capability Maturity Model Integration (CMMI)[END_REF] was conceived as an initiative to integrate the various CMMs into a set of integrated models. The source models that served as the basis for the CMMI (see Figure 1 The CMMI includes two representations: Continuous and Staged. The Continuous representation is well adapted to improve or assess an organisation, process area by process area. The Staged representation is adapted to improve the maturity of organisations and globally the systems / software engineering process. We will speak now in this paper only about the Staged representation. The Staged CMMI describes the same 5 distinct levels of maturity as the CMM (see Figure 2), even if the words are a little bit different.

Process characterized for the organisation and proactive

Process measured and controlled

Focus on process improvement

Level

Requirements Management (REQM) Project Planning (PP) Project Monitoring and Control (PMC) Supplier Agreement Management (SAM) Measurement and Analysis (MA) Process and Product Quality Assurance (PPQA) Configuration Management (CM)

Basic PROJECT management

Requirements Development (RD) Technical Solution (TS) Product Integration (PI) Verification (VER) Validation (VAL) Organizational Process Focus (OPF) Organizational Process Definition (OPD) Organizational Training (OT) Integrated Project Management (IPM) Risk Management (RSKM) Decision Analysis and Resolution (DAR) Integrated Teaming (IT) IPPD Integrated Supplier Management (ISM) Organization Environment for Integration (OEI) IPPD

PROCESS standardization

Organizational Process Performance (OPP) Quantitative Project Management (QPM)

Quantitative management

Organizational Innovation and Deployment (OID) Causal Analysis and Resolution (CAR)

Continuous process improvement

Process Areas Focus

Differences between SW_CMM and CMMI

As issued from an important project, CMMI includes several improvements wrt CMM. The main ones are following:

-CMMI covers software practices and systems practices, Many companies accepted to be named in the SEI Web site as users of CMM / CMMI: Airbus, Boeing, Bosch, Honeywell, Hughes, Siemens VDO, Thales, Toshiba…

-

CMM / CMMI at EADS SPACE Transportation

For more than 30 years, EADS SPACE Transportation has developed software for space domain, as Industrial Architect of Ariane, and for defence domain, as prime contractor for French missiles development. Software development process applied a strong reference system based on principles of GAM-T17 (French Defence) and DOD-2167A (US Defence); this reference system is equivalent to the ECSS-E-40 and ECSS-Q-80.

The SW-CMM project

Even if these software products don't have quality problems, software projects had often problems of cost and / or schedule. That is no more acceptable today, so the French part of EADS SPACE Transportation (named AEROSPATIALE Lanceurs) began, already before 2000, to look at improvement approaches such as CMM and SPICE / ISO 15504. When we became prime contractor of ESA for the ATV programme, it was necessary to start of formal software improvement approach to reinforce the mastering of software development process and to increase our credibility to the company managers and to the customers; we have also to consider that the number of SLOC of ATV flight control is very higher (X 10) than these of A5 flight control. Furthermore, SPACE Transportation is now also the prime contractor of ESA for Ariane 5. The CMM approach was chosen because more largely used than SPICE approach in the world and more adapted to the prime contractors.

The CMM project started mid 2001 in order to improve the productivity and the control of SW development activities by generalising best practices especially for estimates, risk management and project management. The involved perimeter covered all software engineering activities related to the main programmes in the Company; the main SW products are various: flight control software, launch preparation software, control software, simulation software and ground facilities software. The project was scheduled in 3 main steps:

first, an awareness to the involved people and an assessment of the current practices, -then, an update of the reference system based on the assessment findings, -and finally a deployment of the new practices in the software project teams before a formal assessment. The first assessment included all the KPA of CMM Level 2 and Level 3, because it was the right way to get, in the same time, a global status of engineering, management and support processes. After a delay due to priorities on business projects, the new reference system was updated and reviewed mid 2003. Then, the deployment of practices was done in all the software project teams, including more 200 people, and it was decided to perform as soon as possible a formal CMM assessment at Level 2. Finally, the CMM Maturity Level 2 was formally attested by a CMM lead assessor in May 2004.

And now, which are the effects of this CMM approach? Is there still anything to perform? It is obviously possible to improve again the software engineering process, because we are only at CMM Level 2 when the higher Level is the Level 5. Is it realistic for the Company to reach the Level 5? Is it interested? A part of the answer is given below in "the CMMI project". Therefore, the benefits of the CMM approach are very significant for the Company, on several aspects:

-Communication aspect: the public image of the Company is largely improved and reinforced towards the customers and the current or future partners; inside the Company, The SW engineers are now better recognized. -Economical aspect: the cost, spent from the kick-off of the project to the formal assessment, was recovered before end of 2004; savings were visible from the beginning of deployment phase. Now the CMM positive effects go on and also the saving, that is very appreciate by the financial managers. -Technical aspect: many positive points are recorded, the main ones are detailed below.

Technical improvements concern both the global software engineering process and specific subprocesses: -SW reference system / policy: it was updated according to the results of first assessment and to comments coming from process reviews; 13 procedures or guidelines was added, especially related to detailed project planning and project management, to requirement management, to metrication and project final assessment. SW project teams have been trained to this reference system and it is really applied: that is checked through regular process reviews. -Requirements management: specific tools have been developed to take into account the fact that the software development cycles are very evolutive, i.e. incremental and iterative. The requirements are now managed very accurately, in particular when they evolve. Every requirement definition or change is reviewed with the system team in term of feasibility, testability, necessary effort, schedule impact, comprehensiveness, coherence with the other requirements, etc… A lot of requirements changes is always formally associated with a predefined software version, in association with the user needs (in term of technical and schedule needs). This permits to plan every software version much more accurately, in particular to plan the need for resources and to anticipate potential problems much better.

The figure 5 gives an example of how the requirements are measured in order to better master the cost and schedule aspects. -Project management: key indicators are now better defined, correlated, analysed and included in a dashboard, so the project manager has more resources and information and the frequency of progress meetings and reportings could be adapted to the project context; risks are more regularly updated and impacts estimated. In accordance with requirements management practices, building an incremental and optimised schedule is now possible. As a consequence, estimations at completion are more realistic and the software is more and more delivered in time, and in line with the expected cost and quality level. Therefore, as said previously in "Software Crisis", this improvement of software development process stressed again the fact that, as software is a part of systems engineering, software projects are directly depending of outputs and data managed at system level; the changes frequently done in these outputs or data have negative consequences on mastering of software project schedule and cost. So, EADS SPACE Transportation considered having a similar approach at system level to look at opportunities of systems engineering improvement.

Requirements stability metric

The CMMI project

After the positive experience with SW-CMM, we must continue our process improvement and, given the number of CMM trained people, it was mandatory to continue with this approach. As the SEI decided to replace CMM by CMMI and to leave soon the CMM formal assessment, we decided obviously to replace formally CMM by CMMI because the CMMI model keeps the principles of the CMM, but weak points have been updated. The real deal was either to continue only on the software engineering with a Level 3 CMMI target or to extent the improvement domain to systems engineering. EADS SPACE Transportation is not a software editor, but a system designer and a provider of launchers or miscellaneous aircrafts: the SW product is always included in a system. It appeared quickly that it would be more profitable to the Company to improve also systems engineering than to continue only to increase the maturity level of software engineering. So, it was decided to apply the CMMI approach for both software engineering and systems engineering in order to be assessed at Level 2 SE/SW as soon as possible. Furthermore, EADS Space Transportation gathers French and German components, for 2 years, and we saw through a CMMI approach an opportunity to go to a common language in the engineering teams.

So, the CMMI project started in March 2005: it concerns systems engineering and software engineering in the two sides France and Germany.

The involved perimeter includes more 800 people of many activities: mission analysis, algorithms and SW, aerodynamics, mechanical analysis, propulsion and control, avionics, ground systems, platforms and ground facilities…These people are managers, projects managers, subcontractor managers, quality assurance managers and engineers.

The CMMI project finality is the same as the CMM project: to improve the productivity of systems engineering and software engineering, the project logic is also the same including the 3 previous main steps. The project objectives are both general and specific to systems engineering.

General goals:

-Software engineering: to continue the effort given by the CMM project, -Systems engineering: to apply the best practices already efficient in the software domain, especially in project management and requirements management, -Policy and knowledge: in engineering domain, to converge to a common language and understanding in France and Germany, and to a common reference system compliant with CMMI and ECSS standards.

Systems engineering specific goals:

-To improve the specific significant weaknesses seen by the SW-CMM project, -To correct the other main weaknesses that could be seen through the first assessment.

The SW-CMM project has shown that software development, because it is strongly linked with system development, is disturbed by 2 main facts:

the frequency and the impact of system changes: systems changes must be managed as well as SW changes and SW engineers have to participate to systems engineering process, from the beginning to the end; -the system data definition and management:

the SW project has to implement a lot of data required formally or implicitly by the system architecture (all the mission data, but also many data for telemetry only): the size of data declarations is, for example, more than 50% code of ATV flight control; the definition process of these system data is very heavy and must be improved.

The CMMI engineering process areas concerned by the previous facts are typically Requirements Definition and Technical Solution (at Level 3) and Requirements Management (at Level 2); the management and support process areas of Level 2 are obviously all concerned. So, the process areas included in the first CMMI assessment are: -all the Level 2 process areas, -the Level 3 engineering process areas.

The CMMI project is now in progress; it is early to give other informations about it.

The software R&D roadmap

To improve the process description is not sufficient to have a good productivity, it is also necessary to improve methods and tools. So, EADS SPACE Transportation built a R&D roadmap in order to improve also in another way cost, quality and schedule. This roadmap, compliant with the company strategy, is implemented first inside the company, but also through research programmes at Group level, from EU or ESA.

The main drivers retained on this topic are costs savings, risks mastering, emerging technologies monitoring and evaluation for future systems developments. Especially, improvement of digital "system to SW" engineering process and associated methods and tools are key aspects. In this field, several actions have already started in 2004 and 2005 through "ASSERT" (EU co-funded project) and "ACG" (Automated Code Generation, ESA Technological Research Project). Both projects are conducted through collaborations between French and German internal teams.

The overall technical context is depicted on the following diagram (figure 6).

ASSERT mainly deals with the digital system requirements capture, the avionics architecture and the sharing between hardware and software, the use of advanced tools and modelling techniques. The formalization of the system requirement via AADL models shall allow developing innovative and proved software architectures. These new architectures will assure for instance a complete independence between the physical implementation level and the functional needs (and then a decrease of the functional application development costs, which would not have any more to take into account the digital architecture constraints). They will also facilitate the reuse of building blocks from one application to another. The use of models shall allow proving early in the development cycle the correctness of the system design. The results of the ASSERT study will be a set of user requirements for the academic institutes, new modelling languages (based on AADL and UML profile), proof tools to be developed by the industrial partners and a new development process.

ACG completes the scope of ASSERT, dealing with the "SIMULINK to SCADE" process and the automatic code generation derived from SCADE. In the ACG process, a modelling activity replaces the phases of specification and of software architectural design. The code is then automatically generated, allowing suppressing detailed design and coding and in the best case unitary tests and integration tests. The classical "V" development cycle is then transformed in a "Y" development cycle (see figure 7).

The results of the ACG study will be an evaluation of ACG tools and a methodology of use of these techniques. Over 2006-2008 period, the roadmap also covers tooling aspects to support Model Driven Approach required for potential "Virtual Spacecraft" activities and "Abstract Interpretation" subject on the track of ACG project. Abstract interpretation techniques shall allow decreasing software bugs discovered in the validation phase and suppressing the unitary tests even in case of a non-certified code generator.

Case studies under ASSERT and ACG will give the opportunity to experiment process, method and tools relying on proof continuity approach, model based environment, automated coding & Test

Conclusion

In spite of the methodological and technological efforts, the "Software Crisis" still persists today, given the increasingly part of software within large and complex space systems. Several studies especially led by ESA showed that the Software Crisis is mainly a "System Crisis" because Software Engineering is a part of Systems Engineering process, but the "software size" is underestimated in the early phases of the projects (A/B).

The CMM model has been largely used by the software community to assess the maturity of the software processes applied by companies and agencies to develop improvement plans or as a reference book. It is going to be replaced now by the CMMI model, which allows in addition assessment and improvement of Systems Engineering processes. One can say the SW-CMM and the CMMI have become "de facto" standards. Furthermore, Level 2 (or 3) of CMM/CMMI is now required by call for proposals.

Even if the improvement of the systems engineering process -by means of a CMMI model-is mandatory to tackle the software crisis phenomenon, it is not sufficient: it is also essential to improve the technologies and particularly to improve the "system to software" tools and methodologies. As shown in this paper, some innovations and techniques are available and must be studied now in order to try to shorten the study and development cycles of the next space programs.

In this context, CMMI is being used within EADS SPACE Transportation to support systems and software process improvement and to develop a common vision of practices at company level.

Figure 1 :

 1 Figure 1: Source models of CMMI

Figure 2 :

 2 Figure 2: maturity levels of Staged CMMI

Figure 3 :

 3 Figure 3: process areas of staged CMMI

Figure 5 :

 5 Figure 5: Requirements stability metric

Figure 6 :

 6 Figure 6: R&D technical context

Figure 7 :

 7 Figure 7: ACG target The next picture (figure 8) locates the different activities along the successive phases of a development. The emphasis is put mainly on the early phases where most cost and time reductions are expected without neglecting the improvement.

CMM V1.1 (1994) • ISO 12207 (1995) SW Life Cycle Processes • ISO 15504 (1998 SPICE) SW Engineering Process Assessment • SE-CMM V1.0 (?) • IPD-CMM V0.98a Integrated Product Development • EIA/IS 731 (1998) Systems Engineering Capability Model SW-CMM V2.0 Draft C (1997) CMMI V1.1 (2002) Continuous Representation Staged Representation

	) include: CMM for Software V2.0 (Draft C),
	EIA-731 Systems Engineering, and IPD CMM (IPD)
	V0.98a.
	The CMMI project participants were coming from
	SEI, US Army, US Navy, but also American
	industrials as Boeing, AT&T, IBM, Honeywell,
	Lockheed Martin, Motorola… and European
	companies: BAE Systems, Ericsson, Q-Labs,
	Thales…
	The first version of CMMI, Version 1.02 was
	delivered in November 2000, the current version,
	Version 1.1 SE/SW extended with IPPD and
	Supplier Sourcing in March 2002.
	Software	Systems Engineering
	• SW-

  the engineering process area of Level 3 is now detailed in 5 engineering process areas:

	Requirements	Development,	Technical
	Solution, Product Integration, Verification
	and Validation;
	-specific process areas have been added
	(Measurement and Analysis at Level 2, Risk
	Management and Decision Analysis and
	Resolution at Level 3);
	-for subcontracts and suppliers, the Supplier
	Agreement Management process includes
	now the COTS;
	-the internal structure is also clarified (see
	Figure 4): specific practices categorize now
	practices that are unique to the process
	area, generic practices are very similar
	across process areas. Furthermore, each
	activity description includes a list of "typical
	work products" (outputs of this activity)
	which allows a better understanding of the
	model.
		Maturity Level
	Process Area 1	Process Area 2	Process Area n
	Specific		Generic
	Goals		Goals
				Common Features
		Commitment	Ability	Directing	Verifying
		to Perform	to Perform	Implementation	Implementation
	Specific		Generic
	Practices		Practices
	Figure 4: Staged CMMI model structure
	From 2001 to June 2005, 1,613 SW-CMM appraisals
	were reported to the SEI [5], for 7,991 projects led by
	860 companies (65% non-USA). 1,982 appraisals
	were done in USA, 399 in INDIA, more 140 in
	France and also in United Kingdom, more 160 in
	Japan, more 280 in China.
	From April 2002 to June 2005, 868 CMMI / SCAMPI
	appraisals were reported to the SEI [6], for 3,250
	projects led by 438 companies (59% non-USA). 365
	appraisals were done in USA, more 100 in INDIA, 26
	in France, 26 in United Kingdom, 100 in Japan, more
	60 in China.