Franc ¸ois Bremond Luis Patino

Monique Thonnat
email: monique.thonnat@inria.fr

Luis Patino Vilchis

Online learning of activities from video

The present work introduces a new method for activity extraction from video. To achieve this, we focus on the modelling of context by developing an algorithm that automatically learns the main activity zones of the observed scene by taking as input the trajectories of detected mobiles. Automatically learning the context of the scene (activity zones) allows first to extract a knowledge on the occupancy of the different areas of the scene. In a second step, learned zones are employed to extract people activities by relating mobile trajectories to the learned zones, in this way, the activity of a person can be summarised as the series of zones that the person has visited. For the analysis of the trajectory, a multiresolution analysis is set such that a trajectory is segmented into a series of tracklets based on changing speed points thus allowing differentiating when people stop to interact with elements of the scene or other persons. Tracklets allow thus to extract behavioural information. Starting and ending tracklet points are fed to a simple yet advantageous incremental clustering algorithm to create an initial partition of the scene. Similarity relations between resulting clusters are modeled employing fuzzy relations. These can then be aggregated with typical soft-computing algebra. A clustering algorithm based on the transitive closure calculation of the fuzzy relations allows building the final structure of the scene. To allow for incremental learning and update of activity zones (and thus people activities), fuzzy relations are defined with online learning terms. We present results obtained on real videos from different activity domains.

Introduction

Scene understanding corresponds to the real time process of perceiving, analysing and elaborating an interpretation of a 3D dynamic scene observed through a network of sensors (including cameras and microphones). This process consists usually in matching signal information coming from sensors with a large variety of models which humans are using to understand the scene in the form of activities carried out by people observed in the scene.

Despite some success stories, scene understanding systems remain limited and usually can function only under restrictive conditions (those defined by the models which humans have defined for specific activities of interest). Here, the real challenge is to discover the activity patterns (called also actions, situations,behaviours, scenarios, scripts and chronicles) by exploring efficiently spatial and temporal information of detected objects. Such automatically learned activity patterns allow uderstanding the scene.

Furthermore, here we state that learning meaningful activity patterns rely on the maintenance and update of the activity throughout time. However, not all learning/modelling techniques are well adapted to perform on-line adaptation. On-line learning is indeed an important capability required to perform scene analysis on long-term basis. In this work we aim at designing an unsupervised system for the extraction of structured knowledge from large video recordings that can perform online. We show how meaningful scene activity characterisation can be achieved through trajectory analysis. By employing clustering techniques, we define the context (activity zones) characterising the scene dynamics. Employing learned zones we extract people activities by relating mobile trajectories to the learned zones. The activity of a person can then be summarised as the series of zones that the person has visited. For the analysis of the trajectory, a multiresolution analysis is set such that a trajectory is segmented into a series of tracklets based on changing speed points. Starting and ending tracklet points are then fed to a simple yet advantageous incremental clustering algorithm to create an initial partition of the scene and find an initial set of activity zones. Similarity relations between resulting clusters are modeled employing fuzzy relations. These can then be aggregated with typical soft-computing algebra. A clustering algorithm based on the transitive closure calculation of the fuzzy relations allows building the final structure of the scene. To allow for incremental learning and update of activity zones (and people activities) fuzzy relations are defined with online learning terms. We present results ob-tained on real videos from different activity domains.

The remainder of this paper is structured as follows. We give first a short overview of the related work (section 2). In section 3 we present an architectural overview of our approach. The detailed methodology for activity extraction is presented in section 4. How we evaluate the proposed approach is explained in section 5. Finally, Section 6 draws the main conclusions and describes some perspectives.

Related Work

Extraction of the activities contained in the video by applying data-mining techniques represents an emerging field in computer vision. Recently it has been shown that trajectory analysis from mobile objects detected in videos can give meaningful activity information. Trajectory analysis has become a popular approach due to its effectiveness in detecting normal/abnormal behaviours. Several approaches proposed on trajectory clustering include splitting algorithms [START_REF] Piciarelli | Trajectory clustering and its applications for video surveillance[END_REF], neural trees [5], Hidden Markov Models (HMM) [3,[START_REF] Oliver | A Bayesian computer vision system for modeling human interactions[END_REF]13]. Transforming trajectory points to another space employing for instance PCA [START_REF] Anjum | Single camera calibration for trajectory-based behavior analysis[END_REF], ICA [2] or DFT [START_REF] Anjum | Single camera calibration for trajectory-based behavior analysis[END_REF] has also been employed. The drawback in these approaches is that an activity pattern is basically represented as a single route along which the objects travel repeatedly. Thus the only meaningful information that can be extracted is what are the most common routes. More importantly, no semantic information can be drawn which facilitates activity understanding for end-users. Thus complex activities including changing of route, partially stopping (for interaction with other mobiles or with static objects of the scene) can hardly be captured and meaningfully described to an end-user with these methods.

In order to add semantic information to trajectory clusters, recent approaches describe trajectory information in terms of mobile origin-destination such as Patino et al. [START_REF] Patino Vilchis | Video Activity Extraction and Reporting with Incremental Unsupervised Learning[END_REF],Sankaranarayanan et al. [START_REF] Sankaranarayanan | Learning directed intention-driven activities using co-clustering[END_REF] where only begining/ending points of the trajectory are analysed. Alternatively beginning/ending trajectory points serve to learn the entry/exit zones (also called sink and sources) of the scene [START_REF] Wang | Learning semantic scene models by trajectory analysis[END_REF]8] or any general activity zone [START_REF] Patino Vilchis | Video Activity Extraction and Reporting with Incremental Unsupervised Learning[END_REF]. Zhou et al. [18] on the contrary first manually define rough borders of sink and sources to guide the activity clustering employing Random Field Topic models but the learned activities are still single routes. Recently, Hospedales et al. [START_REF] Hospedales | Video behaviour mining using a dynamic topic model[END_REF] have also proposed Markov Clustering Topic models for more complex activity modeling but the learning of activity zones for better semantic interpretation has not been adressed.

In addition to activity clustering, in order to enable dynamic adaptation to unexpected or newly observed data, we need a system able to learn the activity clusters on line. Online learning is indeed an important capability required to perform behaviour analysis on long-term basis and to anticipate the human interaction evolutions. An on-line learning algorithm gives a system the ability to incrementally learn new information from datasets that consecutively become available, even if the new data introduce additional classes that were not formerly seen. The previous approaches have not demonstrated to work with on-line learning capabilities. Indeed, many popular classifiers are not structurally suitable for incremental learning; either because they are stable [such as the multilayer perceptron (MLP), radial basis function (RBF) networks, or support vector machines (SVM)], or because they have high plasticity and cannot retain previously acquired knowledge, without having access to old data (such as -nearest neighbor) [9]. Specific algorithms that have been developed to perform on-line incremental learning include the Leader algorithm [6], Adaptive Resonance Theory modules (ARTMAP) [4], Evolved Incremental Learning for Neural Networks [15], leaders-subleaders [16].

Our contribution in this work to the state of the art is thus a trajectory analysis-based algorithm capable to extract complex activities with semantic description capable to perform online. We achieve this by automatically learning the context of the scene (activity zones), then relating mobile trajectories to the learned zones, such that, the activity of a person can be summarised as the series of zones that the person has visited. To guarantee that the algorithm works in on-line way, we have set a memory factor allowing retaining previously acquired knowledge, without having access to old data and yet be able to learn from new incoming data.

General overview of the activity extraction system

Our proposed system works off-line and is composed of four modules applied sequentially. Tracklet calculation process: We study the trajectory speed variations in order to distinguish between the mobile in a stationary state (stop points) or moving state. Point clustering: We aim here to automatically discover unknown activity zones. Stop points obtained from the previous tracklet calculation process are clustered together. The different areas occupied by disjoint groups of stop points form the new discovered activity zones. Zone merging:It is possible that some discovered activity zones are partially overlapping. When this is the case, both regions are most certainly part of a bigger activity zone. Our algorithm will attempt to merge those overlapping activity zones. Event discovery and Activity extraction: One displacement through two different zones is marked as a simple event in our system. This module will find out all different simple events occurring in the video and from them extract the different activities defined by the series of zones that the person has visited.

Activity Analysis and Clustering

We understand activity as the interactions occurring between mobile objects themselves and those between mobiles and the environment. We employ trajectory-based analysis of mobiles in the video to discover the points of entry and exit of mobiles appearing in the scene and ultimately deduce the different areas of activity. In a second step, mobile objects are then characterised in relation to the learned activity areas.

Trajectory analysis process: Tracklet calculation

In order to discover meaningful activity, it is of prime importance to have available detailed information allowing to detect the different possible interactions between mobiles. As our system is based on trajectory analysis, the first step to prepare the data for the activity clustering methodology is to extract tracklets of fairly constant speed allowing to characterise the displacements of the mobile or its stationary state.

If the dataset is made up of N objects, the trajectory tr j for object O j in this dataset is defined as the set of points [x j (t), y j (t)] corresponding to their position points; x and y are time series vectors whose length is not equal for all objects as the time they spend in the scene is variable. The instantaneous speed for that mobile at point [x j (t), y j (t)] is then v (t) = ẋ (t)

2 + ẏ (t)

2 1 2
. The objective is then to detect those points of changing speed allowing to segment the trajectory into tracklets of fairly constant speed so that the trajectory can be summarised as a series of displacements at constant speed or in stationary state.

The mobile object time series speed vector is analysed in the frame of a multiresolution analysis of a time series function v (k) with a smoothing function, ρ 2 s (k) = ρ (2 s k), to be dilated at different scales s. In this frame, the approximation A of v (k) by ρ is such that A 2 s-1 v is a broader approximation of A 2 s v. By analyzing the time series v at coarse resolutions, it is possible to smooth out small details and select those points associated with important changes.

The speed change points are then employed to segment the original trajectory tr j into a series of i tracklets tk. Each tracklet is defined by two key points, these are the beginning and the end of the tracklet, [x i j (1), y i j (1)] and [x i j (end), y i j (end)]. By globally reindexing all tracklets, let m be the number of total tracklets extracted, we obtain the following tracklet feature vector :

tk m = [x m (1), y m (1), x m (end), y m (end)] (1)

Context analysis process

Modeling the spatial context of the scene is essential for recognition and interpretation of activity. Because it is not possible to define a-priori all activity zones, manuallydefined contextual zones do not suffice to describe all possible situations or evolving actions in the monitored scene. We thus propose thus to learn the activity zones.

In our approach, we propose to find first high resoluted (small in size) entry/exit activity zones, which in a second step could be further regrouped to obtain broader activity zones (following an hierarchical agglomerative algorithm).

Clustering of tracklet entry/exit points

For this step, we employ the well-known clustering Leader algorithm [6]. It has the advantage to work on-line without needing to specify the number of clusters in advance. In this method, it is assumed that a threshold T is given. The algorithm constructs a partition of the input space (defining a set of clusters) and a leading representative for each cluster, so that every object in a cluster is within a distance T of the leading object. The threshold T is thus a measure of the diameter of each cluster. The algorithm makes one pass through the dataset, assigning each object to the cluster whose leader is the closest and making a new cluster, and a new leader, for objects that are not close enough to any existing leaders.

Let us consider the position of a mobile is L(x,y), its influential zone, Zn, is defined by a radial basis function (RBF) centered at the position L; and the belonginess of a new point p(x,y) to that zone is given by:

Zn(L, p) = φ(L, p) = exp(-p -L 2 T 2 ) (2)
The RBF function has a maximum of 1 when its input is p = L and thus acts as a similarity detector with decreasing values outputted whenever p strides away from L. An object element will be included into a cluster Zn if Zn(L, p) ≥ 0.5; the cluster receptive field (hyper-sphere) is controlled by the learnt parameter T. In this work we have employed the threshold setting value suggested by [START_REF] Patino Vilchis | Video Activity Extraction and Reporting with Incremental Unsupervised Learning[END_REF].

Merging tracklet zones

We find the final activity areas by merging similar entry/exit tracklet zones. We look to establish a similarity relation between the different zones defined by the tracklets. At the end, new zones are given by the fulfillment of different relations. The first relation indicates if zone Zn i overlaps zone Zn j . This relation is defined as follows:

R1 ij : Zone Zn i overlaps Zone Zn j R1 ij = 3 k=1   p(x,y)∈(X ik ,Y ik ) Zn j (L j , p(x, y))   (3) and (X ik , Y ik ) = n (k+1) 3
T cos (θ) + Li o with θ = 0, ..., π 8 , ..., 2π

That is, points (x, y) ∈ (X ik , Y ik ) belonging to Zn i centered at L i are tested to verify the overlap/similarity between Zn i and Zn j .

In order to make the algorithm work with on-line learning capabilities, capable of retaining the previously acquired knowledge but able to learn from new data we have introduced a memory factor, beta, as follows:

R1 t ij = (1 -β) R1 ij + βR t-1 ij (4)
where R1 t ij is the current relationship value and R t-1 ij is the relationship value from a previous analysed dataset.

Similar relations that we have introduced are the following R2 t ij : zone Zn i and zone Zn j are destination zones for mobiles departing from the the same activity zone Zn k R3 t ij : zone Zn i and zone Zn j are origin zones for mobiles arriving at the the same activity zone Zn k R4 t ij : zone Zn i and zone Zn j have about the same number of detected mobiles stopping at the zone R5 t ij : zone Zn i and zone Zn j have about the same mobile interaction time. The mobile interaction time is the mean time a mobile spends in that zone.

All relations can be aggregated employing a soft computing aggregation operator such as

R = R = R1 t ∩ R2 t ∩ R3 t ∩ R4 t ∩ R5 t = max (0, R1 t + R2 t + R3 t + R4 t + R5 t -4) and made transitive with: R • R (x, y) = max z min [R (x, z) , R (z, y]) (5)
R is then a transitive similarity relation with R indicating the strength of the similarity. If we define a discrimination level α in the closed interval [0,1], an α-cut can be defined such that

R α (x, y) = 1 ⇔ R (x, y) α (6)
It is thus implicit that α 1 > α 2 ⇔ R α1 ⊂ R α2 ; thus, the R α form a nested sequence of equivalence relations, or from the classification point of view, R α induces a partition π α = {Zni α } such that α 1 > α 2 implies π α1 is a refinement of π α2 ; that is, learned zones at π α1 are a refinement of learned zones at π α2 .

Event and Activity extraction process

We aim at creating a system for the recognition and interpretation of human activity and behaviour, and extract new information of interest for end-users. Low-level tracking information is thus expected to be transformed into highlevel semantic descriptions conveying useful and novel information. In our application, we transform low-level tracking information into behaviour knowledge by extracting events and activities that give account of mobile displacements through the scene. We do this thanks to the series of tracklets summarising a mobile trajectoty, and thanks to the learned zones and any user-defined contextual zone. Let us assume, we have in total k = 1, ..., K zones, Zn k , on the scene after the zone learning procedure.

More precisely, we define:

a simple Event is the information an object is transferring from one zone Zn k to a second zone Zn k ′ an Activity is the combination of all simple events associated to the mobile object Two different kinds of simple events can then be identified:

• Mobile moving from Zone Zn k to Zone Zn k ′ • Mobile Inside Zone Zn k

Experimental Results and Evaluation

To test our approach, we have worked on the extraction of activities from the video recorded at one entrance hall in the Torino (Italy) underground system. The algorithm for unsupervised learning of activity zones was applied first to a one hour-duration video from the Torino underground system. The final relation R given in equation ( 5), which verifies the transitive closure, is thresholded for different αcut values going from 0 to 0.9 and with a step value of 0.05. The system gives the possibility to the end-user to select the partition containing the number of activity areas which may suit best his needs. Figure 1 presents those learned zones corresponding to the analysed video. Four different activity levels are calculated from the system. The first level groups all the activity in the scene and thus gives the information as a single activity area where individuals are moving around in the station. The second level gives the information of activity occurring in broad areas. The fourth level gives activity information with smaller and more detailed areas. The third level is a compromise between levels two and four. It can be observed that at the different granularities, those areas being the most employed are: The entry/exit areas to the station, the turnstiles area, and the areas corresponding to the vending machines.

To test the on-line learning capabilities we have analysed four different week days (2 hours/day) from the Torino underground system.

We have also worked on the publicly available Edinburgh Informatics Forum Pedestrian Database where we have processed on-line six full working days.

Figure 2 shows how most employed activity zones stabilise with the processing of long-term data and the discovered zones are in agreement with the manually-defined zones of interest.

As mentioned in section 4.3, we achieve behaviour carachterisation by linking low-level tracking to the learned zones. The whole activity observed from the scene can then be reported following the behaviours inferred from the learned zones. Some examples of such activities are: at Zone 8 (Stays at VM1), Zone 1 to Zone 8 (ZE1 to VM1), Zone 4 to Zone 15 (Turnstiles to ZE2). The last two shown in figure 3.

When arranging the occurring events according to their semantic, and looking at their frequency of occurrence , it is possible to observe that the most common activities carried out in the station hall are people detected at the north station entry/exit, and people detected at the turnstiles area; the flow of people between these two areas (representing already 50% of the whole station activity). On the other hand, rare activities are for instance those of people going through the station from one entry/exit to the opposite entry/exit or going from one vending machine to the other one (we have depicted these trajectories in figure 4). To test the validity of the activity extraction we analysed one hour video where each trajectory has an annotated activity according to user defined ground-truth zones. When comparing with our dis- covered activities we obtain the following results: TP:26, FP:3, FN:1, Precision:0.89, Sensitivity:0.96.

Conclusions

In this work we have presented an off-line system for human activities extraction with on-line learning (incremental learning) capabilities. So far we have set up a system which starts in a first step by the unsupervised learning of the main activity areas of the scene. In a second step, mobile objects are then characterised in relation to the learned activity areas: either as 'staying in a given activity zone' or 'transferring from an activity zone to another' or a sequence of the previous two behaviours if the tracking persists long enough. This characterisation has allowed us to obtain already informative statistics on the use of the station, and discover what are the main activity areas and main displacements of people in the station. Applying on-line learning, the system is able to continuously process long-term video recordings. From the analysis of different consecutive days, we have shown in this paper how activity zones can be learnt incrementally and refined. Two different application domains have been explored in this regard. Our current evaluation signals encouraging results. Our future work consists in a more complex characterisation of the activities by including more features other than spatial activity zones.

Figure 1 .

 1 Figure 1. Left top panel: Original underground scene observed by the camera with user-defined areas delimitating the scene. Remaining panels: Learned zones in a 3D top view. They correspond to activity areas as discovered with our algorithm. Different granularity levels allow understanding the activity with different resolutions

(

  http://homepages.inf.ed.ac.uk/rbf/FORUMTRACKING/)

Figure 3 .

 3 Figure 3. Example of two common activities in the Torino underground.

Figure 4 .

 4 Figure 4. Example of two abnormal trajectories in the Torino underground. Left panel: changing vending machine, right panel: going through the station.

Acknowledgements

This work was partially funded by the EU FP7 project VANA-HEIM with grant no. 248907. Figure 2. Two examples of on-line learning with the proposed approach. Activity zones are updated as new data is available. Top panel: user-defined ground-truth zones are given in dotted lines. Bottom panel: ground-truth entry/exit zones are given with arrows.


DMV, MSc Michaël Verset
email: m.verset@envt.fr

DVM Stuart Carmichael

DVM Michael G Conzemius

PhD, diplômé Acvs

en charge chez le chien

Chapô : la mécompréhension du caractère chronique de l'arthrose est à l'origine de difficultés dans sa prise en charge, bien que de nombreux traitements soient disponibles.

Introduction :

L'arthrose est une maladie sous-estimée chez nos Carnivores Domestiques. Elle touche environ 20% des chiens âgés de plus d'un an et une proportion encore supérieure des animaux les plus âgés. La mécompréhension des mécanismes de cette affection conduit souvent à des mauvais résultats thérapeutiques, source d'insatisfaction et de frustration de la part des propriétaires, et la positionne comme une raison fréquente d'euthanasie (cf. : encadré 1).

L'arthrose, maladie chronique

1.1 Etiologie et physiopathologie : L'arthrose est définie comme une affection articulaire progressive, intrinsèquement non inflammatoire, caractérisée par la dégradation du cartilage articulaire et la production d'os néoformé. Chez le chien, l'arthrose est généralement associée à un facteur déclenchant comme la dysplasie de hanche ou de coude (cf. : illustration 1). La dégénérescence précoce du cartilage provoque la libération de médiateurs proinflammatoires, qui entraînent une inflammation aigue (synovite) et accélèrent le processus dégénératif. De nombreux tissus vont être progressivement atteints.

Conséquences pathologiques et cliniques de la chronicité :

En effet, l'arthrose affecte l'articulation dans sa globalité ; la progression des lésions implique toutes les structures intra-et péri-articulaires ainsi que les muscles adjacents (cf. : encadré 2). La chronicité aboutit à une atteinte du cartilage articulaire, une fibrose de la capsule articulaire, une ostéophytose, une amyotrophie et une fibrose musculaire, des altérations de l'os sous-chondral et des lésions du système nerveux central (cf. : illustration 2). La persistance d'une inflammation chronique à bas bruit, avec néanmoins des poussées aigues de temps à autre, participe au caractère incurable de la maladie. L'extension large des lésions et le caractère imprévisible de la progression de la maladie rendent impossible la description précise de l'état clinique d'un patient à un instant donné. En effet, l'implication spécifique de chaque structure varie entre les individus et au cours du temps, en fonction de l'état de la maladie. Ainsi, les effets de la chronicité rendent difficile la prise en charge de l'arthrose et sa réversibilité.

Le caractère chronique de l'arthrose se manifeste essentiellement par des troubles de la mobilité articulaire et de la douleur, dont le contrôle doit être une priorité. La chronicité induit plusieurs effets sur la composante douloureuse : une stimulation douloureuse continue (depuis chacune des structures intra-et péri-articulaires), une réponse sur-régulée et une sensibilisation périphérique et centrale du système nerveux. A un stade avancé, la stimulation douloureuse chronique induit une plasticité du système nerveux central à l'origine de lésions permanentes de celui-ci, d'un état douloureux accru et d'une absence de réponse des traitements antalgiques.

Ainsi, la progression clinique tend vers une aggravation lente, ponctuée de poussées aigues intermittentes (cf. : illustration 3). L'arthrose aboutit, dans son stade terminal, à un problème clinique persistant, une douleur manifeste et difficile à contrôler et des lésions avancées.

Traitement médical de l'arthrose

Diversité des traitements médicaux disponibles

De nombreux traitements sont disponibles et de nouvelles molécules apparaissent régulièrement sur le marché, mais les preuves scientifiques de leur efficacité sont très variables. Parmi ces traitements, seuls les anti-inflammatoires non stéroïdiens (AINS), les acides gras (AG) oméga-3, l'exercice physique / physiothérapie et la réduction du poids font l'objet de nombreuses études reconnues. L'administration de rations riches en AG oméga-3 (provenant d'huiles de poisson) permet de remplacer une partie des AG oméga-6 de l'organisme, qui possèdent un rôle pro-inflammatoire. Une alimentation riche en AG oméga-3 de poissons a entraîné une augmentation de l'appui mesuré sur les membres de chiens atteints d'arthrose, vs alimentation classique [1]. La physiothérapie a permis à des chiens opérés pour rupture de ligament croisé crânial et lésion méniscale de présenter un meilleur appui à 6 mois postopératoires, vs exercice restreint [2]. De même, une réduction du poids de 11 à 18% a permis une diminution des signes cliniques de boiterie chez des chiens arthrosiques [3]. Les autres traitements dont l'efficacité n'est pas prouvée ou remise en cause à ce jour sont : les corticostéroïdes, les cellules souches, l'acupuncture, les analgésiques, les glycosaminoglycanes polysulfatés injectables et les suppléments oraux comme les glucosamines et/ou chondroïtine sulfate.

Action des anti-inflammatoires non stéroïdiens et intérêt du firocoxib :

Les AINS agissent en inhibant la synthèse de médiateurs de l'inflammation par les enzymes cyclo-oxygénase (COX) de type 1 ou 2. La COX-2 est à l'origine de la synthèse des prostaglandines pro-inflammatoires, alors que la COX-1 est notamment impliquée dans le maintien de la fonction rénale et de l'intégrité de la muqueuse digestive. Certains AINS ont une action préférentielle sur la COX-2 mais inhibent partiellement la COX-1. Au contraire, les coxibs inhibent sélectivement la COX-2, tout en préservant l'activité bénéfique de la COX-1. Cela conduit à une diminution du risque d'effets indésirables, notamment digestifs. Comme le dit Mike CONZEMIUS : <<plus la sélectivité COX-2 d'une molécule est élevée, plus elle est efficace et sûre>>. Le firocoxib est le plus sélectif des AINS [4]. Son efficacité et son innocuité ont été démontrées par de nombreuses études cliniques [5] [6] [7].

Conclusion :

Le caractère chronique de l'arthrose conduit à employer nécessairement des traitements de long terme. Ils doivent être entrepris précocement, adaptés au stade de la maladie et choisis parmi ceux reconnus par la communauté scientifique. Une prise en charge multi-modale fournit des résultats plus rapides et plus efficaces, puisque les différents traitements utilisés peuvent agir en synergie et sur différents sites simultanément. Le suivi du patient est primordial et doit comporter une évaluation des résultats thérapeutiques (évolution de la maladie, mobilité, douleur, qualité de vie).

Objectifs pédagogiques :

Comprendre le caractère chronique de l'arthrose et ses conséquences. Connaître les différents traitements médicaux de l'arthrose et leur valeur scientifique reconnue à ce jour.

Points forts :

L'arthrose est une maladie chronique, progressive, complexe et incurable. De nombreux tissus autres que le cartilage articulaire sont affectés, notamment le système nerveux. Le contrôle de l'inflammation et de la douleur est le but primordial du traitement médical. Les seuls traitements médicaux de l'arthrose basés sur des preuves scientifiques solides sont la diminution du poids, l'exercice physique, les AINS et les AG oméga-3. Ces traitements doivent être combinés pour une prise en charge multi-modale, plus efficace.

A lire :

  Tetrick MA, Muir P. Effect of weight reduction on clinical signs of lameness in dogs with hip osteoarthritis. J Am Vet Med Assoc. 2000 ; 216(7) : 1089-1091. 4 McCann ME et coll. In vitro effects and in vivo efficacy of a novel cyclooxygenase-2 inhibitor in dogs with experimentally induced synovitis. Am J Vet Res. 2004 ; 65(4) : 503-512. 5 Pollmeier M et coll. Clinical evaluation of firocoxib and carprofen for the treatment of dogs with osteoarthritis. Vet Rec. 2006 ; 159(17) : 547-551. 6 Hanson PD et coll. Efficacy and safety of firocoxib in the management of canine osteoarthritis under field conditions. Vet Ther. 2006 ; 7(2) : 127-140. 7 Autefage et coll. Long-term efficacy and safety of firocoxib in the treatment of dogs with osteoarthritis. Vet Rec. 2011 ; 168(23) : 617.

	1	Roush JK et coll. Evaluation of the effects of dietary supplementation with fish oil
	omega-3 fatty acids on weight bearing in dogs with osteoarthritis. J Am Vet Med Assoc.
	2010 ; 236(1) : 67-73.
	2	Marsolais GS, Dvorak G, Conzemius MG. Effects of postoperative rehabilitation on
	limb function after cranial cruciate ligament repair in dogs. J Am Vet Med Assoc. 2002 ;
	220(9) : 1325-1330.
	3	Impellizeri JA,


Karama Kanoun

Yves Crouzet

Ali Kalakech

Ana-Elena Rugina

Windows and Linux Robustness Benchmarks With Respect to Application Erroneous Behavior

This chapter presents the specification of dependability benchmarks for general-purpose operating systems with respect to application erroneous behavior, and shows examples of benchmark results obtained for various versions of Windows and Linux operating systems. The benchmark measures are: operating system robustness (as regards possible erroneous inputs provided by the application software to the operating system (OS) via the application programming interface), the OS reaction, and restart times in the presence of faults. Two workloads are used for implementing the benchmark: PostMark, a file system performance benchmark for operating systems, and the Java Virtual Machine (JVM) middleware, a software layer on top of the OS allowing applications in Java language to be platform independent.

Introduction

Software is playing an increasingly important role in our day-to-day life. In particular, operating systems (OSs) are more and more used even in critical application domains. Choosing the operating system that is best adapted to one's needs is becoming a necessity. For a long time, performance was the main selection criterion for most users and several performance benchmarks were developed and are widely used. However, an OS should not only have good performance but also a high dependability level. Dependability benchmarks emerged as a consequence. Their role is to provide useful information regarding the dependability of software systems [START_REF] Tsai | [END_REF][START_REF] Brown | [END_REF][START_REF] Chevochot | [END_REF], Brown et al. 2002, Zhu et al. 2003, Zhu et al. 2003a]. This chapter is devoted to the specification, application and validation of two dependability benchmarks of OSs using two different workloads: PostMark, a file system performance benchmark, and JVM (Java Virtual Machine), a software layer on top of the OS allowing applications in Java language to be platform independent.

Benchmarking the dependability of a system consists of evaluating dependability or performance-related measures, experimentally or based on experimentation and modeling, in order to characterize objectively the system behavior in the presence of faults. Such an evaluation should allow non-ambiguous comparison of alternative solutions. Non-ambiguity, confidence in results and meaningfulness are ensured by a set of properties a benchmark should satisfy. For example, a benchmark must be representative, reproducible, repeatable, portable and cost effective. These properties should be taken into consideration from the earliest phases of the benchmark specification as they have a deep impact on almost all benchmark components. Verification of the benchmark key properties constitutes a large part of the benchmark validation.

Our dependability benchmark is a robustness benchmark. The robustness of an OS can be viewed as its capacity to resist/react to external faults, induced by the applications running on top of it, or originating from the hardware layer, or from device drivers. In this chapter we address OS robustness as regards possible erroneous inputs provided by the application software to the OS via the Application Programming Interface (API). More explicitly, we consider corrupted parameters in system calls. For the sake of conciseness, such erroneous inputs are referred to as faults.

The work reported in this chapter is a follow up of the European project on Dependability Benchmarking, DBench [DBench]. Our previously published work on OS dependability benchmarks was based on i) TPC-C Client performance benchmark for transactional systems [Kalakech et al. 2004b], ii) PostMark [START_REF] Kanoun | [END_REF], and iii) JVM workload [START_REF] Kanoun | [END_REF]].

The work reported in [Shelton et al. 2000] is the most similar to ours; it addressed the "non-robustness" of the POSIX and Win32 APIs (while we are interested in robust and non-robust behavior). P ione e r w or k on robustness benchmarking is published in [START_REF] Mukherjee | [END_REF]. Since then, a few studies have addressed OS dependability benchmarks, considering real time microkernels [START_REF] Chevochot | [END_REF][START_REF] Arlat | [END_REF], Gu et al. 2004] or general purpose OSs [START_REF] Tsai | [END_REF], Koopman & DeVale 1999]. Robustness with respect to faults in device drivers is addressed in [START_REF] Chou | [END_REF][START_REF] Durães | Characterization of Operating Systems Behavior in the Presence of Faulty Drivers through Software Fault Emulation[END_REF], Albinet et al. 2004] and also in Chapter 13.

The remainder of the chapter is organized as follows. Section 2 gives the specification of the OS benchmarks. Section 3 presents benchmark implementation and results related to PostMark workload for Windows and Linux families. Section 4 presents benchmark implementation and results related to JVM workload. Section 5 refines 3 the benchmark results for PostMark and JVM. Section 6 outlines the main benchmark properties that are meaningful to OS benchmarks, and briefly shows what has been achieved to ensure and check them. Section 7 concludes the chapter.

Specification of the Benchmark

In order to provide dependability benchmark results that are meaningful, useful and interpretable, it is essential to define clearly the following benchmark components:

1) The benchmarking context.

2) The benchmark measures to be evaluated and the measurements to be performed on the system to provide the information required for obtaining the measures.

3) The benchmark execution profile to be used to exercise the operating system.

4) Guidelines for conducting benchmark experiments and implementing benchmark prototypes.

These components are presented hereafter.

Benchmarking Context

An OS can be seen as a generic software layer that manages all aspects of the underlying hardware. The OS provides i) basic services to the applications through the API, and ii) communication with peripheral devices via device drivers. From the viewpoint of dependability benchmarking, the benchmark target corresponds to the OS with the minimum set of device drivers necessary to run the OS under the benchmark execution profile. However, for the benchmark target to be assessed, it is necessary to run it on top of a hardware platform and to use a set of libraries. Thus, the benchmark target along with the hardware platform and libraries form the system under benchmarking. Although, in practice, the benchmark measures characterize the system under benchmarking (e.g., the OS reaction and restart times are strongly dependent on the underlying hardware), for clarity purpose we will state that the benchmark results characterize the OS.

The benchmark addresses the user perspective, i.e., it is primarily intended to be performed by (and to be useful for) someone or an entity who has no in depth knowledge about the OS and whose aim is to significantly improve her/his knowledge about its behavior in the presence of faults. In practice, the user may well be the developer or the integrator of a system including the OS.

The OS is considered as a "black box" and the source code does not need to be available. The only required information is the description of the OS in terms of system calls (in addition of course to the description of the services provided by the OS).

Benchmark Measures

The benchmark measures include a robustness measure and two temporal measures.

After execution of a corrupted system call, the OS is in one of the states summarized in Table 1. SEr: corresponds to the case where the OS generates an error code that is delivered to the application.

SXp: corresponds to the case where the OS issues an exception. Two kinds of exceptions can be distinguished depending on whether it is issued during the application software execution (user mode) or during execution of the kernel software (kernel mode). In the user mode, the OS processes the exception and notifies the application (the application may or may not take into account explicitly this information). However, for some critical situations, the OS aborts the application. An exception in the kernel mode is automatically followed by a panic state (e.g., blue screen for Windows and oops messages for Linux). Hence, hereafter, the latter exceptions are included in the panic state and the term exception refers only to user mode exceptions.

SPc:

In the panic state, the OS is still "alive" but it is not servicing the application. In some cases, a soft reboot is sufficient to restart the system.

SHg:

In this state, a hard reboot of the OS is required.

SNS:

In the no-signaling state, the OS does not detect the presence of the corrupted parameter. As a consequence, it accepts the erroneous system call and executes it. It may thus abort, hang or complete its execution. However, the response might be erroneous or correct. For some system calls, the application may not require any explicit response, so it simply continues execution after sending the system call. SNS is presumed when none of the previous outcomes (SEr, SXp, SPc, SHg) is observed 1 .

Panic and hang outcomes are actual states in which the OS can stay for a while. They characterize the OS's nonrobustness. Conversely, SEr and SXp characterize only events. They are easily identified when the OS provides an error code or notifies an exception. These events characterize the OS's robustness.

OS Robustness (POS)

is defined as the percentages of experiments leading to any of the outcomes listed in Table 1. POS is thus a vector composed of 5 elements.

Reaction Time (Treac) corresponds to the average time necessary for the OS to respond to a system call in the presence of faults, either by notifying an exception or by returning an error code or by executing the system call.

Restart Time (Tres) corresponds to the average time necessary for the OS to restart after the execution of the workload in the presence of one fault in one of its system calls. Although under nominal operation the OS restart time is almost deterministic, it may be impacted by the corrupted system call. The OS might need additional time to make the necessary checks and recovery actions, depending on the impact of the fault applied.

The OS reaction time and restart time are also observed in absence of faults for comparison purpose. They are respectively denoted reac and res.

1

In our work we do not address "hindering" errors (e.g., incorrect error codes being returned) as in [Shelton et al. 2000]. This is extremely difficult to deal with, and it is almost impossible to automate their identification.

Note that our set of measures is different from the one used in [Shelton et al. 2000]. Shelton et al. only take into account non-robust behavior of the OS. They use the CRASH scale to measure the OS non-robustness (Catastrophic, Restart, Abort, Silent and Hindering failures). Our aim is to distinguish as clearly as possible the proportions of robust versus non-robust behavior of the OS. Also, we completed the set of measures by adding to it the two temporal measures (Texec and Tres) presented above.

Benchmark Execution Profile

For performance benchmarks, the benchmark execution profile is a workload that is as realistic and representative as possible for the system under benchmarking. For a dependability benchmark, the execution profile includes, in addition, corrupted parameters in system calls. The set of corrupted parameters is referred to as the faultload.

The benchmark is defined so that the workload could be any performance benchmark workload (and, more generally, any user specific application) intended to run on top of the target OS. In [Kalakech et al. 2004b] we have used the workload of TPC-C Client [START_REF] Tpc-C | TPC-C[END_REF], and in this work we use two workloads PostMark [Katcher 1997] and JVM [START_REF] Lindholm | [END_REF]. The two workloads will be discussed further in Sections 3 and 4.

The faultload consists of corrupted parameters of system calls. For Windows, system calls are provided to the OS through the Win32 environment subsystem. For Linux OSs, these system calls are provided to the OS via the POSIX API. During runtime, the workload system calls are intercepted, corrupted and re-inserted.

We use a parameter corruption technique relying on thorough analysis of system call parameters to define selective substitutions to be applied to these parameters (similar to the one used in [START_REF] Koopman | [END_REF]). A parameter is either a data or an address. The value of a data can be substituted either by an out-of-range value or by an incorrect (but not out-of-range) value, while an address is substituted by an incorrect (but existing) address (containing usually an incorrect or out-of-range data). We use a mix of these three corruption techniques 2 . Note that non-existing addresses are always detected. Hence they are not considered as interesting substitution values.

To reduce the number of experiments, the parameter data types are grouped into classes. A set of substitution values is defined for each class. They depend on the definition of the class. Some values require a pre and a post processing such as the creation and the destruction of temporary files. For example, for Windows, we group the data types into 13 classes. Among these classes, 9 are pointer classes. Apart from pvoid (pointer which points to anything), all other pointers point to a particular data type. Substitution values for these pointers are combination of pointer substitution values and the corresponding data type substitution values. Similarly, for Linux, we group the data types into 13 classes among which 5 are pointer classes. We use the same substitution values for basic data types (i.e., integer) both for Windows and Linux. Nevertheless, some data types are system-dependent. Consequently, they have specific substitution values. In Linux, for example, we define a class corresponding to the type mode. A mode is an integer with a particular meaning: read/write modes or permission flags. As the validity domain of this data type can be identified precisely, pertinent substitution values are defined for it. Table 2 reviews the substitution values associated with the basic data type classes.

2

A discussion about the adequacy of this choice is given in Section 6.1.3.

Benchmark Conduct and Implementation

Since perturbing the operating system may lead the OS to hang, a remote machine, referred to as the benchmark controller, is required to reliably control the benchmark experiments, mainly in case of OS Hang or Panic states or workload hang or abort states (that cannot be reported by the machine hosting the benchmark target). Accordingly, for running an OS dependability benchmark we need at least two computers: i) the Target Machine for hosting the benchmarked OS and the workload, and ii) the Benchmark Controller that is in charge of diagnosing and collecting part or all benchmark data. The two machines perform the following functions: i) restart of the system before each experiment and launch of the workload, ii) interception of system calls with parameters, ii) corruption of system call parameters, iii) re-insertion of corrupted system calls, iv) observation and collection of OS outcomes.

The experiment steps in case of workload completion are illustrated in Figure 1. In case of workload noncompletion state (i.e., the workload is in abort or hang state), the end of the experiment is provided by a watchdog timeout as illustrated in Figure 2. The timeout duration is fixed to a value that is three times greater than the largest workload execution time without faults.

To intercept Win32 functions, we use the Detours tool [START_REF] Hunt | [END_REF], a library for intercepting Win32 binary functions on X86 machines. The part of Detours in charge of system call interception is composed of 30 Kilo-lines of code (KLOC). The modifications we carried out on this tool concern i) the replacement of system call parameters by corrupted values (this module is 3 KLOC) and ii) the addition of modules to observe the reactions of the OS after parameter corruption, and to collect the required measurements (this module is 15 KLOC). To intercept POSIX system calls, we used another interception tool, Strace [START_REF] Mcgrath | [END_REF]. Strace is composed of 26 KLOC. Also, we added two modules to this tool to allow i) substitution of the parameters and ii) observation of Linux behavior after parameter corruption (these modules correspond to 4 KLOC together). The reaction time is counted from the time the corrupted system call is re-inserted. Hence the time to intercept and substitute system calls is not included in the system reaction time, as shown in Figures 1 and2.

Figure 3 summarizes the various components of the benchmark environment. All the experiments have been run on the same target machine, composed of an Intel Pentium III Processor, 800 MHz, and a memory of 512 Megabytes. The hard disk is 18 Gigabytes, ULTRA 160 SCSI. The benchmark controller in both prototypes for Windows and Linux is a Sun Microsystems workstation. Once the benchmarking tool is compiled, it is used to identify the set of system calls activated by the workload.

All parameters of all these system calls are then analyzed and placed into the corresponding class. A database of substitution values is then generated accordingly.

Following the benchmark execution sequence presented in Figures 1 and2, at the beginning of each experiment, the target machine (TM) records the experiment start instant tExpStart and sends it to the benchmark controller (BC) along with a notification of experiment start-up. The workload starts its execution. The Observer module records, in the experiment execution trace, the start-up instant of the workload, the activated system calls and their responses. This trace also collects the relevant data concerning states SEr, SXp, and SNS. The recorded trace is sent to the BC at the beginning of the next experiment.

The parameter substitution module checks whether the current system call has parameters. If it is not the case, the execution is simply resumed; otherwise, the execution is interrupted, a parameter value is substituted and the execution is resumed with the corrupted parameter value (tResume is saved in the experiment execution trace). The state of the OS is monitored so as to diagnose SEr, SXp, and SNS. The corresponding OS response time (tResponse) is recorded in the experiment execution trace. For each run, the OS reaction time is calculated as the difference between tResponse and tResume.

At the end of the execution of the workload, the OS notifies the BC of the end of the experiment by sending an end signal along with the experiment end instant, tExpEnd , and then it restarts so that the current experiment does not have any effects on the following experiment. If the workload does not complete, then tExpEnd is governed by the value of a watchdog timer. The BC collects the SHg state and the workload abort/hang states. It is in charge of restarting the system in such cases. When no faultload is applied, the average time necessary for the OS to execute PostMark or JVM is less than 1 minute for Windows and for Linux. We considered that three times the normal execution time is enough to conclude on the experiment's result. Thus, we have fixed the watchdog timer to 3 minutes. If, at the end of this watchdog timer, the BC has not received the end signal from the OS, it then attempts to ping the OS. If the OS responds (ping successful), the BC attempts to connect to it. If the connection is successful, then a workload abort or hang state is diagnosed. If the connection is unsuccessful, then a panic state, SPc, is deduced. Otherwise, SHg is assumed. If workload abort / hang or SPc or SHg are observed, tResponse does not take any value for the current experiment. Thus, the measure Treac is not skewed by the watchdog timer value.

At the end of a benchmark execution, all files containing raw results corresponding to all experiments are on the BC. A processing module extracts automatically the relevant information from these files (two specific modules are required for Windows and Linux families). The relevant information is then used to evaluate automatically the benchmark measures (the same module is used for Windows and Linux).

PostMark Dependability Benchmark Implementation and Results

PostMark creates a large pool of continually changing files and measures the transaction rates for a workload emulating Internet applications such as e-mail or netnews. It generates an initial pool of random text files ranging in size from a configurable low bound to a configurable high bound. The file pool is of configurable size and can be located on any accessible file system. The workload of this benchmark, referred to as PostMark for simplicity, is responsible for realizing a number of transactions. Each transaction consists of a pair of smaller transactions: i) create file or delete file and ii) read file or append file. PostMark is written in the C language.

From a practical point of view, PostMark needs to be compiled separately for each OS. The reaction time is given in Figure 5. Globally, Windows OSs have shorter response times than Linux OSs. The standard deviation is significantly larger than the average for all OSs. Except for the two revisions of Linux 2.4, reac is always larger than Treac, the reaction time in the presence of faults. This can be explained by the fact that after parameter corruption, the OS detects the anomaly in almost 45% of cases for Windows and 75% of cases for Linux, and stops system call execution, returns an error code or notifies an exception.

10 Note that for the Windows family, Windows XP has the lowest reaction time in the presence of faults and for the Linux family, Linux 2.6.6 has the lowest reaction time. For Linux 2.6.6, we notice that reac is almost two times larger than for the other revisions. A detailed analysis of the results showed that this is due to one system call, execve, for which the execution time is 15000 µs for Linux 2.6.6 and 6000 µs for other versions. These values may be considered as outliers. Removing them and re-evaluating the average lead to more representative results.

Another measure of interest in this case could be the median, since it is not sensitive to extreme values.

The restart times are shown in Figure 6. The average restart time without faults, res, is always lower than the average restart time with faults (Tres), but the difference is not significant. Linux seems to be globally faster (71-83s) than Windows (74-112s). However, if we consider only OS versions introduced in the market after 2001, the other OSs rank as follows: Linux 2.2.26 (71s), Windows XP (74s), Windows 2003 server (77s), Linux 2.4.5 (79s), Linux 2.6.6 (82s), Linux 2.4.26 (83s).

Windows family Linux family

Without faults With faults Concerning the Linux family, we note that the restart time increases with new versions or revisions, except for Linux 2.6.6. This progression is due to the increasing size of kernels with the version evolution. The exception of Linux 2.6.6 is justified by the fact that the Linux kernel was restructured in its version 2.6.

JVM Dependability Benchmark Implementation and Results

Java Virtual Machine (JVM) is a software layer between the OS and Java applications, allowing applications in Java language to be platform independent. The specifications of the virtual machine [START_REF] Lindholm | [END_REF] are independent from the hardware platform but each platform requires a specific implementation. The benchmark based on JVM has been applied to three Windows versions (NT, 2000 and XP) and to the four Linux versions considered in the previous section. In this benchmark, JVM is solicited through a small program allowing to activate 76 system calls with parameters for the Windows family and 31 to 37 system calls with parameters for the Linux Family, as indicated in Table 4. These system calls are intercepted and corrupted using the corruption technique presented in Section 2.3, which leads to the number of experiments indicated in the last line for each OS. Robustness is given in Figure 7. As for the PostMark workload, no Hang or Panic states have been observed. It can be noticed that the three Windows versions are roughly equivalent, as well as the Linux versions.

Comparison with Figure 4 shows that the robustness of each family is the same using PostMark or JVM workloads (within 5% discrepancy). Furthermore, we have observed the same robustness for Windows versions using TPC-C as workload in our previous work [Kalakech et al. 2004a]. The three workloads solicit different numbers of systems calls and only some of them are the same. Nevertheless they lead to the same robustness.

Windows family Linux family Their execution times are significantly larger without fault than in the presence of faults. A detailed analysis of the results showed that for these two system calls, most of the corruption experiments ended with an error code (SEr). Thus, we assume that the system calls were abandoned after an early anomaly detection by the OS. Also for Linux, the reaction times in the presence of faults are relatively high. This is due to three system calls (execve, getdents64 and nanosleep). Not including the reaction times associated with these system calls leads respectively to a Treac of 88µs, 241µs, 227µs and 88µs for the 2.2.26, 2.4.5, 2.4.26 and 2.6.6 versions.

Windows family Linux family

Without faults (reac) With faults (Treac)

Results Refinement

The benchmark temporal measures are refined to provide more insights into those presented in Sections 3 and 4. We first consider PostMark, then JVM. For each of them, we mainly detail the temporal measures.

PostMark

Reaction time

Table 5 presents the detailed reaction times with respect to OS outcomes after execution of corrupted system calls (Error Code, Exception and No Signaling). Thus, three average times are added to detail Treac: TSEr, TSXp (the times necessary to return respectively an error code or an exception) and TSNS (the execution time of the corrupted system call, in case of no-signaling state). For the Windows family, it can be seen that for versions 2000, 2000 Server, XP and 2003 Server, returning an error code takes less time than raising an exception. This can be explained by the fact that when returning an error code, tests are carried out on the parameter values at the beginning of the system call code and the system call is abandoned, while the exceptions are raised from a lower level of the system under benchmarking. Nevertheless, in the cases of Windows NT4 and NT4 Server, TSEr is higher than TSXp. The cause of this anomaly lies in the long time necessary to GetCPInfo system call to return an error code when its first parameter is corrupted.

Concerning the Linux family, the averages presented in this table do not take into account execve system call execution time. Cells in grey correspond to high values of the standard deviation and are commented hereafter. We notice the high values of TSNS corresponding to the two revisions of version 2.4, compared to the two other versions. The very high standard deviation suggests a large variation around the average, which is confirmed in Figure 10 that gives the OS reaction time for all system calls leading to the no-signaling state for all Linux OSs. We can clearly see that for Linux 2.4 the average time necessary for executing mkdir is more than 10 times larger than for all other system calls. We have noticed that mkdir has an extremely long execution time when its second parameter (which corresponds to the permissions to apply on the newly created folder) is corrupted. Also, a very large average time to return an error code is observed for Linux 2.2.26, with a high standard deviation. Figure 11 details the times necessary to return error codes for Linux system calls. It is clear that these times are very similar except for unlink system call in Linux 2.2.26, which explains the high TSEr of Linux 2.2.26 compared to the other versions. After discarding the exceptional values corresponding to execve, mkdir and unlink system calls, the average reaction times Treac of the four targeted Linux OSs become very close. The largest difference is of 8µs. Also, the average reaction times with respect to OS outcomes after execution of corrupted system calls (TSEr, TSXp, TSNS) become very close. The largest difference is of 18µs. Furthermore, reac and Treac become very close.

Restart time

Detailed analyses show that all OSs of the same family have similar behavior and that the two families exhibit very different behaviors.

For Windows, there is a correlation between the restart time and the state of the workload at the end of the experiment. When the workload is completed, the average restart time is statistically equal to the restart time without parameter substitution. On the other hand, the restart time is larger and statistically equal for all experiments with workload abort/hang. This is illustrated in Figure 12 in case of Windows NT, 2000 and NT.

For example, the average restart time in case of workload completion is 73 s and 80 s in case of workload abort/hang, for Windows XP. Linux restart time is not affected by the workload final state. Detailing Linux restart times shows high values appearing periodically. These values correspond to a "check-disk" performed by the Linux kernel every 26 Target Machine restarts. This is illustrated for Linux 2.2.26 and 2.6.6 in Figure 13, and induces an important standard deviation on this measure. Also, it is interesting to note that the check-disk duration decreases with the version evolution, while the regular restart time increases. It seems natural for the time needed to complete a check-disk to decrease while the Linux kernel evolves. The increase of the regular restart time may be due to the increasing size of Linux kernels.

JVM

Reaction time

Similarly to Table 5, Table 6 presents the detailed reaction times with respect to OS outcomes after execution of corrupted system calls (Error Code, Exception and No Signaling).

For Windows family, the averages presented in this table do not take into account LoadLibraryA, LoadLibraryExA and LoadLibraryExW system call execution times. As in the case of the use of the PostMark workload, we notice that, for Windows 2000 and Windows XP, TSEr is higher than TSXp, which is higher than TSNS. The standard deviations for these measures are rather small. On the other hand, for Windows NT4 TSEr is lower than TSXp.

Figure 14 details the times necessary to raise exceptions for Windows system calls. It is clear that these times are similar except for the system call FindNextFileW. The execution time of this system call is greater than 13400 µs for Windows NT4 while for the other Windows OSs it is smaller than 75 µs. It is noteworthy that the execution time of this system call is also the cause for the large values for TSE and TSXp in the case of NT4. 15 that gives the OS reaction time for all system calls leading to the no-signaling state for all Linux OSs. As in the case of the use of PostMark workload, for Linux 2.4 the average time necessary for executing mkdir is more than 10 times larger than for all other system calls.

After discarding the exceptional values corresponding to execve, getdents64, nanosleep and mkdir system calls, the average reaction times Treac of the four targeted Linux OSs become very close. The largest difference is of 15µs. Also, the average reaction times with respect to OS outcomes after execution of corrupted system calls (TSEr, TSXp, TSNS) become very close. The largest difference is of 28µs. Furthermore, reac and Treac become very close (if we discard the executions times for sched_getparam and sched_getscheduler system calls).

Restart time

As in the case of the PostMark workload, for Windows, there is a correlation between the restart time and the state of the workload at the end of the experiment. When the workload is completed, the average restart time is statistically equal to the restart time without parameter substitution. On the other hand, the restart time is larger and statistically equal for all experiments with workload abort/hang. This is illustrated in Figure 16. For example, for Windows XP, the average restart time in case of workload completion is 68 s and 82 s in case of workload abort/hang.

As in the case of the PostMark workload, Linux restart time is not affected by the workload final state. Detailing the restart times shows high values appearing periodically, due to a "check-disk" performed by the kernel every 26 Target Machine restarts. This is illustrated for Linux 2.2.26 and 2.6.6 in Figure 17. Also, it is noteworthy that the check-disk duration decreases with the version evolution, while the regular restart time increases.

PostMark and JVM

OS robustness of all OSs with respect to PostMark and JVM workloads is synthesized in Table 7. It shows that none of the catastrophic outcomes (Panic or Hang OS states) occurred for all Windows and Linux OSs. It also shows that Linux OSs reported more error codes (57%-67%) than Windows (23%-27%), while more exceptions were raised with Windows (17%-25%) than with Linux (7%-10%). More no-signaling cases have been observed for Windows (52%-56%) than for Linux (25%-33%). For Linux family, Linux 2.2.26 seems to have the most robust behavior (the smallest proportion of no-signaling) while for Windows family, the differences between OSs are too small to differentiate them clearly. The reaction times in the presence of faults are globally smaller for Windows family OSs than for Linux family OSs (if we exclude a few systems calls for which the reaction time is exceptionally very long). We have noticed that the execution times of a minority of system calls can have important consequences on the mean reaction time values. For instance, the execution time of mkdir system call for Linux 2.4 biases the Treac measure. High standard deviations on this measure are due to a minority of system calls with very large reaction time compared to the majority. If we discard these exceptional values, Treac and reac become very close. Moreover, Treac for Windows and Linux families become close.

The restart times in the presence of faults (Tres) are always higher than the restart times without faults (res). The lowest Tres was observed for Linux 2.2.26 and Windows XP (71 s). For the Windows family, the restart time is higher in case of workload abort/hang both when using PostMark and JVM as workloads. For Linux family, the standard deviation on this measure is high because of check-disks performed every 26 Target Machine restarts. We have noticed that check-disks take a longer time to perform when using JVM as workload than when using PostMark. The regular restart time is the same both when using PostMark and JVM.

The ordering of the OSs with respect to the restart time is the same for Postmark and JVM except for Windows 2000, for which the restart time is better than Windows NT using JVM while it is worse when using PostMark.

Benchmark Properties

In order to gain confidence in dependability benchmark results, one has to check that the key properties are fulfilled. These properties are addressed successively in the rest of this section. We first define the property then we show what has been achieved to satisfy and check it.

Representativeness

Representativeness concerns the benchmark measures, the workload and the faultload.

Measures

The measures evaluated provide information on the OS state and temporal behavior after execution of corrupted system calls. We emphasize that these measures are of interest to a system developer (or integrator) for selecting the most appropriate OS for his/her own application. Of course other measures would help.

Workload

We have selected two benchmark workloads whose characteristics, in terms of system calls activated, are detailed hereafter. Nevertheless, the selection of any other workload does not affect the concepts and specification of our benchmark.

The PostMark workload activates system calls belonging to functional components: file management, thread management, memory management and system information. Most of system calls belong to the file management functional component (62% for Linux, 48% for Windows). However, a significant amount of system calls belong to the thread management (12% for Linux, 32% for Windows) and to the memory management (19% for Linux, 8% for Windows) functional components. In total, 93% for Linux and 88% system calls for Windows are in these functional components. The PostMark workload is representative if the OS is used as a file server.

Let us stress that we chose the workload to be the JVM and not a particular Java application, to focus on system calls activated by the JVM, regardless of the Java application running on top of it. The JVM is solicited through a very small program. The JVM workload activates system calls fairly distributed with respect to functional components (file management, thread management, memory management, user interface, debugging and handling, inter-process communication). Similarly to the PostMark workload, most of the activated system calls (88% for Linux, 73% for Windows) belong to the file management (40% for Linux, 26% for Windows), the thread management (24% for Linux, 36% for Windows), and to the memory management (24% for Linux, 11% for Windows) functional components.

Faultload

The faultload is without any doubt the most critical component of the OS benchmark and more generally of any dependability benchmark. Faultload representativeness concerns i) the parameter corruption technique used and ii) the set of corrupted parameters.

Parameter corruption technique

In our previous work [START_REF] Jarboui | [END_REF], performed for Linux, we have used two techniques for system call parameter corruption: the systematic bit-flip technique consisting in flipping systematically all bits of the target parameters (i.e., flipping the 32 bits of each considered parameter) and the selective substitution technique described in Section 2. This work showed the equivalence of the errors induced by the two techniques. In [Kalakech et al. 2004b] we obtained the same robustness for Windows 2000 using the systematic bit-flip technique and the selective substitution technique.

The application of the bit-flip technique requires much more experimentation time compared to the application of selective substitution technique. Indeed, in the latter case, the set of values to be substituted is simply determined by the data type of the parameter (see Section 2), which leads to a more focused set of experiments.

We have thus preferred the selective substitution technique for pragmatic reasons: it allows derivation of results that are similar to those obtained using the well-known and accepted bit-flip fault injection technique, with much fewer experiments. Our benchmark is based on selective substitutions of system call parameters to be corrupted.

Parameters to be corrupted

The selective substitution technique used is composed of a mix of three corruption techniques as mentioned in Section 2: out-of-range data (OORD), incorrect data (ID) and incorrect addresses (IA). Let us denote the faultload used in our benchmarks by FL0. To analyze the impact of the faultload, we consider two subsets, including respectively i) IA and OORD only (denoted FL1), and ii) OORD only (denoted FL2). For each workload (PostMark and JVM), we ran the benchmarks of all OSs considered using successively FL0, FL1 and FL2. The results obtained confirm the equivalence between Linux family OSs as well as the equivalence between Windows family OSs, using the same faultload (FL0, FL1 or FL2). Note that for each OS, its robustness with respect to FL0, FL1 or FL2 is different but the robustness of all OSs of the same family with respect to the same faultload is equivalent. The same results have been obtained in [Kalakech et al. 2004b], using TPC-C Client as workload.

The number of substitutions (hence the number of experiments) decreases significantly when considering FL1 and F2. By way of examples, Table 8 gives the number of experiments for Windows NT4 and Linux 2.4 for PostMark and Figure 18 shows the robustness of Windows NT4, 2000 and XP with respect to FL1 and FL2, for PostMark. (robustness with respect to FL0 is given in Figure 3). Moreover, we checked the representativeness of incorrect data faults. One could argue that the OS is not assumed to detect this kind of fault as the substitution values are inside the validity domain of the parameter type. The analysis of the execution traces corresponding to experiments with incorrect data substitution that led to notification of error codes in the case of Linux, revealed that 88.6% of the faults correspond to out-of-range data in the very particular context of the workload execution. Consequently, the notification of error codes was a normal outcome in these cases. Incorrect data are thus very useful: they can provide a practical way for generating out-of-range data in the execution context. Note that an enormous effort would be needed to analyze all execution contexts for all system calls to define pertinent substitution values for each execution context.

Repeatability and Reproducibility

The benchmarking of a given system can be based either on an existing benchmark implementation (an existing prototype) or on an existing specification only. Repeatability concerns the benchmark prototype while reproducibility is related to the benchmark specification.

Repeatability is the property that guarantees statistically equivalent results when the benchmark is run more than once in the same environment (i.e., using the same system under benchmark and the same prototype). This property is central to benchmarking. Our OS dependability benchmark is composed of a series of experiments. Each experiment is run after system restart. The experiments are independent from each other and the order in which the experiments are run is not important at all. Hence, once the system calls to be corrupted are selected and the substitution values defined, the benchmark is fully repeatable. We have repeated all the benchmarks presented three times to check for repeatability.

Reproducibility is the property that guarantees that another party obtains statistically equivalent results when the benchmark is implemented from the same specification a n d i s u s ed to benchmark the same system. Reproducibility is strongly related to the amount of details given in the specification. The specification should be at the same time i) general enough to be applied to the class of systems addressed by the benchmark and ii) specific enough to be implemented without distorting the original specification. We managed to satisfy such a tradeoff. Unfortunately, we have not checked explicitly the reproducibility of the benchmark results by developing several prototypes by different people. On the other hand, the results seem to be independent from the technique used to corrupt system call parameters. This makes us confident about reproducibility. However, more verification is still required.

Portability

Portability concerns essentially the faultload (i.e., its applicability to different OS families).

At the specification level, in order to ensure portability of the faultload, the system calls to be corrupted are not identified individually. We decided to corrupt all system calls of the workloads. This is because OSs from different families do not necessarily comprise the very same system calls as they may have different APIs. However, most OSs feature comparable functional components.

At the implementation level, portability can only be ensured for OSs from the same family because different OS families have different API sets.

Let us consider the case of PostMark as an example, for which the first prototype developed was for Windows 2000. It proved to be portable without modification for Windows 2000 Server and Windows 2003 Server (PostMark activates the same 27 system calls with parameters), and with minor adaptations for the others. One system call (FreeEnvironmentStringA) is not activated under Windows NT4, NT4 Server and XP and another system call (LockResource) is not activated under NT4 and NT4 Server. In these cases, the system calls that are not activated are dropped from the substitution values database.

For Linux, the prototype revealed to be portable across all OSs except the interceptor Strace, which is kerneldependent. Consequently, we used one version of Strace for Linux 2.2 and 2.4 and another version for Linux 2.6. Also, PostMark activates the same system calls for Linux 2.2.26 and 2.4 while it activates a supplementary system call (mmap2) for Linux 2.6.6. Consequently, we added this system call to the set of activated system calls and an entry in the substitution values database.

Cost

Cost is expressed in terms of effort required to develop the benchmark, run it and obtain results. These steps require some effort that is, from our point of view, relatively affordable. In our case, most of the effort was spent in defining the concepts, characterizing the faultload and studying its representativeness. The implementation of the benchmark itself was not too time consuming.

Let's first consider PostMark, then JVM (which benefited a lot from the PostMark benchmarks as all benchmark components did exist and we had only to adapt them).

For PostMark, the benchmark implementation and running took us less than one month for each OS family, spread as follows:

• The installation of PostMark took one day both for Windows and Linux.

• The implementation of the different components of the controller took about two weeks for each OS family, including the customization of the respective interceptors (Detours and Strace).

• The implementation of the faultload took one week for each OS family, during which we have i) defined the set of substitution values related to each data type and ii) created the database of substitution values. Both databases are portable on OSs belonging to their family (one database for Windows family and one database for Linux family). However, small adaptations were necessary (see Section 6.3).

• The benchmark execution time for each OS is less than two days.

The duration of an experiment with workload completion is less than 3 minutes (including the time to workload completion and the restart time), while it is less than 6 minutes without workload completion (including the watchdog timeout and the restart time). Thus, on average, an experiment lasts less than 5 minutes. The series of experiments of a benchmark is fully automated. Hence, the benchmark execution duration ranges from one day for Linux to less than two days for Windows (25-27 system calls are activated by PostMark on Windows, while only 16-17 system calls are activated on Linux).

For JVM, the first step consisted in executing JVM for each OS to be benchmarked, to identify system calls activated. The second step was devoted to define, for each system call, the parameters to be corrupted and the exact substitution values, to prepare the database to be used in the Interception/substitution/observation modules. This step took a couple of days for Linux family (activating 31-37 system calls depending on the version considered) and the double for Windows because it activates 76 system calls. Adaptation of the benchmark controller and of the Interception/substitution/observation modules required about one day for each family. The benchmark duration ranges from one day for each Linux OS to less than three days for each Windows OS.

Conclusion

We have presented the specification of a dependability benchmark for OSs with respect to erroneous parameters in system calls, along with prototypes for two families of OSs, Windows and Linux, and for two workloads. These prototypes allowed us to obtain the benchmark measures defined in the specification. We stress that the measures obtained for the different OSs are comparable as i) the same workloads (PostMark and JVM) were used to activate all OSs, ii) the faultload corresponds to similar selective substitution techniques applied to all system calls activated by the workload and iii) the benchmark conduct was the same for all OSs.

Concerning the robustness measure, the benchmark results show that all OSs of the same family are nearly equivalent. They also show that none of the catastrophic states of the OS (Panic or Hang) occurred for any of the Concerning the OS reaction time, results show that globally Linux reaction time, related to system calls activated by the workload, is longer than Windows reaction time. Refinement of this measure revealed a great variation around the average and that a minority of system calls with large execution times skewed the average. When these system calls are not considered, the reaction times of all the OSs of the same family become equivalent.

With respect to the restart time measure, Windows XP and Linux 2.2.26 have the shortest restart times in the presence of faults (71 s). Detailed analysis showed i) a correlation between Windows restart time and the workload final state (in case of workload hang or abort, the restart time is 10 % higher than in case of workload completion) and ii) that Linux performs a check-disk after each 26 restarts. A restart with a check-disk is three to four times longer than the average.

We validated our benchmark paying a particular attention to representativeness of faultload, and to the properties of repeatability, reproducibility, portability and cost effectiveness of the benchmark.

Figure 1 :Figure 2 :

 12 Figure 1: Benchmark execution sequence in case of workload completion

  Figure 4: OS Robustness (%), using PostMark

Figure 5 :

 5 Figure 5: Reaction time (in micro seconds), using Postmark

Figure 6 :

 6 Figure 6: Restart time (in seconds), using Postmark

Figure 7 :

 7 Figure 7: Robustness measures with JVM

Figure 8 :

 8 Figure 8: OS reaction time (in micro seconds), using JVM

Figure 10 :

 10 Figure 10: Linux reaction time in case of SNS (in micro seconds), using PostMark

Figure 11 :

 11 Figure 11: Linux reaction time in case of SER (in micro seconds), using PostMark

Figure 12 :

 12 Figure 12: Detailed Windows restart time, using PostMark

Figure 13 :

 13 Figure 13: Detailed Linux restart time, using PostMark

Figure 14 :Figure 15 :

 1415 Figure 14: Windows reaction time in case of SXp (in micro seconds), using JVM

Figure 16 :

 16 Figure 16: Detailed Windows restart time, using JVM

FL1FL2Figure 18 :

 18 Figure 18: OS Robustness using FL1 and FL2 (%), using PostMark

  Windows and Linux OSs c o n s i d e r e d . L i n u x O S s r e p o r t e d m o r e e r r o r c o d e s t h a n W i n d o w s w h i l e m or e exceptions were raised with Windows than with Linux. More no-signaling cases have been observed for Windows than for Linux.

Table 1 : OS outcomes

 1

Table 2 : Parameter substitution values

 2

  Workstation will be referred to as Windows 2000 and Windows NT4 respectively. Four Linux OSs (Debian distribution) are targeted: Linux 2.2.26, Linux 2.4.5, Linux 2.4.26 and Linux 2.6.6. Each of them is a revision of one of the stable versions of Linux (2.2, 2.4, 2.6). Table3summarizes the number of system calls targeted by the benchmark experiments carried out along with the number of corresponding parameters and the number of experiments for each OS.

				Windows family					Linux family
		W-NT4 W-2000 W-XP W-NT4S W-2000S W-2003S L-2.2.26 L-2.4.5 L-2.4.26 L-2.6.6
	# System Calls	25	27	26	25	27	27	16	16	16	17
	# Parameters	53	64	64	53	64	64	38	38	38	44
	# Experiments	418	433	424	418	433	433	206	206	206	228

Six versions of Windows

OSs are targeted: Windows NT4 Workstation with SP6, Windows 2000 Professional with SP4, Windows XP Professional with SP1, Windows NT4 Server with SP6, Windows 2000 Server with SP4 and Windows 2003 Server. In the rest of this chapter, Windows 2000 Professional and Windows NT4

Table 3 : Number of system calls, corrupted parameters and experiments for each OS, using PostMark

 3

Table 4 : Number of system calls, corrupted parameters and experiments for each OS, using JVM

 4

Table 5 : Detailed reaction time, using PostMark

 5

	Windows Family
			NT4		2000		XP
		Avg.	S. Dev.	Avg.	S. Dev.	Avg.	S. Dev.
	Treac	148 µs	219 µs	118 µs	289 µs	114 µs	218 µs
	TSEr	45 µs	107 µs	34 µs	61 µs	45 µs	118 µs
	TSXp	40 µs	15 µs	37 µs	15 µs	50 µs	96 µs
	TSNS	234 µs	437 µs	186 µs	375 µs	168 µs	265 µs
		NT4 Server		2000 Server		2003 Server
	Treac	110 µs	221 µs	131 µs	289 µs	102 µs	198 µs
	TSEr	41 µs	66 µs	29 µs	33 µs	25 µs	61 µs
	TSXp	35 µs	15 µs	37 µs	15 µs	48 µs	20 µs
	TSNS	166 µs	280 µs	210 µs	396 µs	156 µs	252 µs
	Linux Family
		2.2.26	2.4.5	2.4.26	2.6.6
	Treac	167 µs	300 µs	466 µs 2276 µs 425 µs 2055 µs	93 µs	12 µs
	TSEr	208 µs	361 µs	92 µs	105 µs	84 µs	6 µs	91 µs	10 µs
	TSXp	88 µs	5 µs	91 µs	8 µs	91 µs	8 µs	106 µs	13 µs
	TSNS	85 µs	5 µs	1545 µs 4332 µs 1405 µs 3912 µs	91 µs	11 µs

Table 6 : Detailed reaction time, using JVM Concerning

 6 the Linux family, the averages presented in this table do not take into account execve, getdents64 and nanosleep system call execution times. As in the case of the use of the PostMark workload, we notice the high values of TSNS corresponding to the two revisions of version 2.4, compared to the two other versions. The very high standard deviation suggests a large variation around the average, which is confirmed in Figure

	Windows Family
			NT4			2000		XP
		Avg.	S. Dev.	Avg.	S. Dev.	Avg.	S. Dev.
	Treac	388 µs	3142 µs	190 µs	451 µs	214 µs	483 µs
	TSEr	298 µs	3006 µs	47 µs	110 µs	51 µs	103 µs
	TSXp	424 µs	3862 µs	84 µs	168 µs	98 µs	209 µs
	TSNS	417 µs	2858 µs	307 µs	588 µs	344 µs	625 µs
	Linux Family
		2.2.26	2.4.5	2.4.26	2.6.6
	Treac	88 µs	85 µs	241 µs 1479 µs 227 µs 1438 µs 88 µs	26 µs
	TSEr	90 µs	101 µs	79 µs	6 µs	84 µs	30 µs	86 µs	8 µs
	TSXp	87 µs	7 µs	85 µs	8 µs	87 µs	8 µs	98 µs	15 µs
	TSNS	84 µs	6 µs	572 µs 2545 µs 523 µs 2545 µs 89 µs	43 µs

Table 7 : Linux and Windows robustness using PostMark and JVM

 7

Table 8 : Faultloads considered

 8

Acknowledgement

We would like to thank all the DBench colleagues who, through the numerous discussions all over the project, helped us in defining the OS benchmark as it is in this paper. In particular, we are grateful to Jean Arlat who contributed to the OS benchmark based on TPC-C Client.


Karama Kanoun

Jean-Claude Laprie

Trend Analysis

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.


est destinée au dépôt

Software reliability and system reliability

Jean-Claude Laprie, Karama Kanoun

To cite this version:

Jean-Claude Laprie, Karama Kanoun. Software reliability and system reliability. M. Lyu. Handbook for Software Reliability Engineering, McGraw-Hill, pp.27-69, 1996, 0-07-039400-8. hal-00761643