Cédric Jeanneret
email: jeanneret@ifi.uzh.ch

Martin Glinz
email: glinz@ifi.uzh.ch

Benoit Baudry
email: benoit.baudry@irisa.fr

Benoit Combemale
email: benoit.combemale@irisa.fr

Impact of Footprinting on Model Quality: An Experimental Evaluation

Keywords: requirements modeling, model footprinting, controlled experiment P-Value 0.91 0.41 0.20 0.63

When modeling requirements, software analysts have to choose the relevant modeling constructs among all those available. If they do not choose the right set, their model may lack some important information or their model may contain many superfluous details. In previous work, we proposed to capture the purpose of a model with a set of model operations such as queries or model transformations. Then, modelers can analyze the footprints of these operations, that is, the set of model elements touched during their execution.

In this paper, we report on two controlled experiments performed with students to evaluate whether footprinting can help them in creating better models. While our studies did not demonstrate statistically significant benefits of footprinting, they reveal the importance of training and tool support for the analysis of footprints.

I. INTRODUCTION

With the advent of Model Driven Engineering (MDE), models play an important role in the development of software. Conceptually, a model is an abstract representation of an original for a given purpose. Ideally, a model exactly represents its domain, that is, the set of all possible statements that would be correct about the original and relevant for the purpose at hand [START_REF] Lindland | Understanding quality in conceptual modeling[END_REF].

A model can differ from its domain in two different ways. In [START_REF] Lindland | Understanding quality in conceptual modeling[END_REF], Lindland et al. define validity as the extent to which a model only contains statements from the domain. A model with superfluous details may be harder to understand and is probably more expensive to create than necessary. On the opposite, a model may lack some relevant details, making it incomplete. When using an incomplete model, an interpreter may draw wrong conclusions about the original.

Thus, the value of a model depends heavily on whether it contains all relevant information and only this information. However, creating models that actually represent their domain is not an easy task. Eliciting information about the original is not enough; the modeler needs to understand the purpose of the model as well. Often, little is known about this purpose and typical modeling assignments only mention the modeling language to be used. When the language contains many constructs, as UML does, this indication offers little help.

Recently, we have invented a technique called footprinting to detect the presence of superfluous elements in models [START_REF] Jeanneret | Estimating footprints of model operations[END_REF]. In a MDE setting, the purpose of a model can be characterized by the set of model operations (e.g., queries, view extractions or model transformations) that the model must enable. A footprint is the set of all model elements that have been used during the execution of these operations. When footprints are highlighted, modelers can easily find the elements that were not used by the set of operations executed on the model, which gives hints about their possible irrelevance. However, the benefits of footprinting have not yet been evaluated empirically.

In this paper, we are interested in the actual effect of footprinting on the quality of models. To investigate this effect, we performed a pair of controlled experiments: one at the University of Zurich, Switzerland, involving undergraduate students enrolled in a basic Software Engineering course, and one at the University of Rennes 1, France, involving graduate students enrolled in a course on MDE techniques.

The remainder of the paper is structured as follows. In the next section, we provide background information about model quality and model footprinting. We describe our experiments in Section III, while their results are presented and analyzed in Section IV. Finally, in Section V, we discuss our findings and we conclude in Section VI.

II. BACKGROUND

A. Model Quality

As models become more and more important for the development of software, many researchers proposed frameworks for evaluating and improving the quality of models. In [START_REF] Lindland | Understanding quality in conceptual modeling[END_REF], Lindland et al. proposed the first systematic approach to identifying quality goals and means to achieve them. In this framework based on semiotics (the study of symbols), models are seen as sets of statements. Models are compared to three other sets of statements: The language is the set of statements expressible in the modeling language. The domain is the set of all possible statements that would be correct and relevant while the audience interpretation is the set of statements that the model users think a model contains.

Their framework defines three types of model quality. Syntactic quality is how well a model corresponds to the language. Semantic quality is how well the model corresponds to the domain. Finally, pragmatic quality is how well a model corresponds to the audience interpretation. We chose the framework of Lindland et al. for our work because it laid the foundation for many other publications. Krogstie et al. extended this framework with additional constructs from the semiotics theory [START_REF] Krogstie | Process models representing knowledge for action: a revised quality framework[END_REF]. Moody et al. validated the framework empirically in [START_REF] Moody | Evaluating the quality of information models: Empirical testing of a conceptual model quality framework[END_REF] while España et al. used the framework to compare two RE methods in [START_REF] España | Evaluating the completeness and granularity of functional requirements specifications: A controlled experiment[END_REF].

Lindland et al. distinguish two semantic quality goals: completeness and validity. A model is complete if it contains all the statements from the domain, while a model is valid if it only contains statements from the domain (and nothing else). A statement can be invalid for two reasons: it is incorrect or it is irrelevant for the purpose at hand. In our work, we distinguish between the two cases. We use the term confinement for the extent to which a model only contains relevant statements [START_REF] Mohagheghi | Definitions and approaches to model quality in model-based software development: A review of literature[END_REF] while correctness designates the extent to which a model only contains correct information.

In typical settings, the domain does not exist explicitly. Semantic quality is then typically assessed subjectively with reviews. In our experiments, the domain is represented by a reference model. This has been done previously by España et al. to compare two RE methods in [START_REF] España | Evaluating the completeness and granularity of functional requirements specifications: A controlled experiment[END_REF]. Furthermore, we not only consider the actual quality of models -measured on the model with respect to the reference model -but also the quality as it is perceived by the modelers. This perception is important, because on a typical modeling task, the domain does not exist explicitly and there is no such thing as a reference model. Thus, modelers consider their modeling task complete when their models contain all statements from what they perceive as the domain.

B. Model Footprinting

To assess the relevance of some model elements, the modelers must know the purpose of their models. In previous work [START_REF] Jeanneret | Modeling the purposes of models[END_REF], we proposed to use goal oriented requirements engineering methods to capture this purpose. Eventually, this purpose must be operationalized with some model operations such as analyses, queries and model transformations. For example, a class diagram could be used to derive a glossary of a domain or a state machine could be used to generate the skeleton of an implementation. The main assumption behind the idea of footprinting is that the purpose of a model can be characterized by the set of model operations that will be performed on this model. These operations can be performed mentally by humans or executed by a computer. In both cases, a model must conform to some structure to support these operations. This structure is typically expressed with an object-oriented metamodel. Such a metamodel defines the types and features -meta-classes, meta-attributes and meta-references -that can be used in a model. For example, the metamodel of Footprinting consists of finding which elements of a model are used by a given model operation during its execution. There are two techniques for footprinting: dynamic and static footprinting [START_REF] Jeanneret | Estimating footprints of model operations[END_REF]. Dynamic footprinting analyzes the execution of a set of operations, while static footprinting estimates the footprint based on a static analysis of the definition of the operations. In [START_REF] Jeanneret | Estimating footprints of model operations[END_REF], we demonstrated that, despite some limitations, static footprints are very precise estimates of dynamic footprints and yet cheap to compute.

In the experiments, we have used static footprinting, as it can be done without tool support. Static footprinting is divided into two phases (see Figure 1). First, the source code of the operations is analyzed. Types and features are collected along their control flow graph. This analysis yields a static metamodel footprint, the set of all constructs relevant for the operations being analyzed. In the second phase, models are filtered through the static metamodel footprint: only elements related to constructs in the static metamodel footprint are kept in the static (model) footprint. This filtering is very simple and can be done manually. Thus, as long as the static metamodel footprint is provided, static footprinting can be done without tool support.

For example, the operation generating a glossary from a class diagram may be implemented as a model-to-text transformation. The source code of this operation may involve classes and generalizations, but probably not ports or dependencies. In addition, the operation will likely read the name of a class, but not whether it is abstract. The static metamodel footprint would then include the following elements:

• Class -name • Generalization
The modelers should only model elements that are instances of the static metamodel footprint. Any other elements will not be used by the operations performed on the model. They may therefore be irrelevant, decreasing the confinement of the model. Furthermore, the modeler should consider all constructs in the static metamodel footprint. If a relevant meta-class or meta-attribute is omitted from the model, some information is probably missing, decreasing the completeness of the model. The filtering step of static footprinting validates whether a model only contains elements related to the static metamodel footprint.

We believe that with the help of footprinting, modelers produce models that are more confined and more complete than without footprinting. Indeed, static metamodel footprints make explicit, what kind of details should be modeled and what kind should not. Besides, we expect footprinting to make modelers more confident in the completeness and confinement of their model. The purpose of these experiments is to confirm or to refute this belief. In the next section, we discuss the planning of our experiments.

III. EXPERIMENT PLANNING

The goal of our experiments is to evaluate the effect of model footprinting on the actual and perceived quality of models in the context of students modeling the requirements of a software system. Following the template defined by Wohlin [START_REF] Wohlin | Experimentation in software engineering: an introduction[END_REF], we present the design of the experiments we conducted towards this goal. In the next section, we present the context and the material of the experiments. Then, in Section III-B, we highlight the variables of our experiments and formulate our hypotheses. In Section III-C, we lay out the design of our experiments and we discuss threats to validity in Section III-D.

A. Context and Material

Our experiments were carried out at two sites: Rennes and Zurich. In Zurich, the context of our experiment is a second year (undergraduate level) Software Engineering (SE) course at the University of Zurich. As a prerequisite for this SE lecture, participants had to follow a lecture on modeling 1 . While this modeling lecture does not aim at teaching UML per se, many UML diagrams are covered during the lecture. Students learn the various constructs and modeling guidelines, and they apply this knowledge on some practical exercises. In Rennes, the experiment took place within a Model Driven Engineering (MDE) lecture, which is visited by two second year classes on the graduate level: Miage (oriented towards business) and GL (oriented towards software engineering). The students from Rennes had to visit a lecture on modeling as prerequisite too 2 . At both sites, the experiment was part of a series of compulsory labs and exercises. Students were told that they would not be graded on performance, but that they were expected to solve their modeling assignments in a professional manner to obtain the points assigned to the lab.

Alternatively, we could have recruited these students with some financial incentives. According to Sjøberg et al. [START_REF] Sjøberg | Conducting realistic experiments in software engineering[END_REF], our experiment would have been easier to organize with this alternative. However, some students could have refused to participate in our experiment, reducing the number of data points and introducing a bias towards motivated students.

Participants were not aware that we were attempting to evaluate the impact of footprinting, but knew that we would evaluate the quality of their models by comparing them with a reference model. The experiments were monitored by the first author on both sites. He was not involved in any of the courses as an assistant, reducing the bias of the experiment. During the experiment, students were not allowed to talk to each other.

The modeling assignments were solved with MagicDraw3 , an UML modeling tool. We chose MagicDraw over other modeling tools because it fully supports UML class diagrams and state machines and it is intuitive to use. Using a tool makes the modeling task more realistic than pen-andpaper only [START_REF] Sjøberg | Conducting realistic experiments in software engineering[END_REF]. It has the additional advantage that it prevents participants from making syntactic mistakes. While the tool is used by the students from Rennes during their curriculum, students from Zurich never used it before the experiment. To overcome this issue, we answered all the questions related to MagicDraw during the experiments. Furthermore, we let students use comments to specify UML elements when they did not find out how to specify them properly with the tools. For example, some students modeled the triggers of state transitions by attaching comments to the transitions. We took these comments into account when assessing the quality of models.

For this experiment, we developed the following material: a modeling assignment with 2 exercises, the reference models and the corresponding static metamodel footprints. This material is included in the appendices. The exercises consist in creating UML models from case descriptions in plain text. The exercises were taken from [START_REF] Satzinger | Systems Analysis and Design in a Changing World[END_REF]. The description for the class diagram is the dental clinic case (on page 272) and the description for the state machine is the shipment case (on page 275). Students were asked to solve each exercise within 45 minutes, so that they had enough time to solve both exercises and fill in the questionnaires during the time allocated for the labs (2 hours). The textbook also provides some solutions, which were used to create the reference models.

To define the static metamodel footprints, we decided not to define (and later analyze) some operations, but rather derive the static metamodel footprint from the reference model directly. That is, we collected all types and features present in the reference model. We decided to do so to avoid a potential threat to validity: defining model operations ourselves might have introduced some bias in the experiments. However, this decision leaves the validation of the main assumption of footprinting -the purpose of a model can be characterized as a set of operations -out of the scope of this paper. This will be investigated in future work. The experiment remains nevertheless valid, as the origin of the static metamodel footprint is of no importance for modelers.

In total, students answered 4 questionnaires. First, they had to participate in a pre-experimental survey, introducing them to the experiment and collecting their modeling experience. After each modeling exercise, students had to fill in a questionnaire to report how they perceived the modeling task, the case description and the quality of their model. They also had to mention how long they took to perform the modeling task. The post-experimental questionnaire explained briefly the goals of the experiment and invited students to provide us with feedback about the experiments.

B. Variables and Hypotheses

In our experiments, we investigate the impact of footprinting on model quality. We measured 4 quality aspects characterizing the semantic quality of a model as defined by Lindland in [START_REF] Lindland | Understanding quality in conceptual modeling[END_REF]: completeness (COM), confinement (CON), correctness (COR) and overall quality (OAQ). Each quality aspect comes in two flavors: actual and perceived quality. Actual quality is the semantic quality of a model with respect to a reference model. We measure this quality by comparing the model to a reference model. Perceived quality is the quality of the model as perceived by its modeler. We measure it with questionnaires. In total, our experiments involve 9 variables: 8 dependent variables and 1 independent variable.

Actual Quality: We quantify actual quality by counting the number of mistakes in the model with respect to a reference model. We distinguish three types of mistakes. A completeness mistake is the absence of some element from the model. For example, in a class diagrams at the analysis stage, the absence of multiplicities or a missing attribute are completeness mistakes. Confinement mistakes are due to the presence of superfluous but correct elements in the model. For example, the presence of interfaces or getter operations are confinement mistakes in an analysis class diagram. All other mistakes are denoted as correctness mistakes. This kind of mistakes includes wrong multiplicities or redundant attributes in a class diagram. ACOM counts the number of completeness mistakes, ACON counts the number of confinement mistakes, ACOR counts the number of correctness mistakes and AOAQ is the total number of mistakes.

Perceived Quality: Perceived quality is measured with a questionnaire. More preciseley, students were asked to what extent they agreed on the following statements using a 5level Likert scale: The purpose of the model was stated informally in all assignments. Still, some participants received additional information for one of the two diagrams they had to create: the static metamodel footprint (FP), that is, the set of modeling constructs relevant for the purpose of the model. This is a nominal variable: either the static metamodel footprint was available or not. FP is the only independent variable of our experiment.

•
Hypotheses: Based on the goal of the experiment, we formulate, for each of the dependent variables, the following null-hypothesis (H 0 ): there is no difference between the quality of models when the modeler knows (or does not know) the static metamodel footprint (FP). The alternative hypothesis (H 1 ) is that footprinting has an impact (positive or negative) on the quality of models.

In this paper, we use Wilcoxon rank-sum tests to test our hypotheses. A Wilcoxon rank-sum test is a non-parametric test to compare the median of two samples. We do not use the T-test because the data does not respect all its assumptions: the counts of mistakes (actual quality) follow Poisson distributions rather than normal distributions and the Likert-scales (perceived quality) only deliver ordinal values. If the non-parametric test is not powerful enough, we fit the counts of mistakes to Poisson distributions and compare their parameter.

C. Experiment Design

This experiment only involves one factor whose effect is interesting to us: whether the static metamodel footprint is given in the task assignment or not. However, a nuisance factor may impact the quality of models: the ability of students in modeling. Therefore, we opt for a completely randomized design in which each participant uses both methods (with and without static metamodel footprint) [START_REF] Wohlin | Experimentation in software engineering: an introduction[END_REF]. Overall, our design is inspired by those of Briand et al. when they investigated the benefits of OCL in UML based development [START_REF] Briand | A controlled experiment on the impact of the object constraint language in UML-based development[END_REF].

To minimize learning effects between the modeling tasks, we considered two different objects: a class diagram and a state diagram. We keep ordering effects under control by using four groups instead of two, covering all possible permutations. These groups are presented in

D. Threats to Validity

External Validity: In this experiments, participants are students, half-way to get their Bachelor's or Master's degree. They may not be representative of software analysts working in the field. However, these students know UML at this stage as much as when they will leave the university and enter their professional career. Besides, footprinting is meant to help modelers who do not know much about how their models are used, which is typically the case for novice modelers.

We intentionally kept the modeling tasks small enough so that the students could achieve them within 45 minutes. This constraint reduces the (undesired) effect of fatigue on the results. However, these assignments are not representative of modeling tasks in an industrial context because of their small size and the presence of a complete case description (participants did not have to elicit requirements). Still, if footprinting has an impact on small models, it will likely have an impact on larger models, too. Elicitation was kept out of the experiment, as it is not within the scope of our investigations.

Internal Validity: Besides the availability of the static metamodel footprint, other factors can impact the quality of models, like the ability of participants and ordering effects (learning and fatigue). We kept ability under control by having each participant use both methods, while ordering effects were kept under control by having four groups solving two different modeling tasks in different order.

We provided footprinting in its most basic form: a static metamodel footprint in textual form. Alternatively, we could have provided an example of models containing all constructs from the static metamodel footprint or a tool displaying the model footprint. We chose the most basic form to reduce the bias towards footprinting. Furthermore, the static metamodel footprint can give some hints about completeness, while the static (model) footprint cannot.

Construct Validity: The perceived quality (PCOM, PCON, PCOR and POAQ) is measured using a questionnaire. We phrased our questions so as to avoid bias. Actual quality (ACOM, ACON, ACOR and AOQ) is measured by counting the number of mistakes made with respect to a reference model, that is, the solution of the modeling assignment. This reference model was written by the first author based on the correct model provided in [START_REF] Satzinger | Systems Analysis and Design in a Changing World[END_REF]. Still, to mitigate the risk that this domain is biased, the domain was validated by other authors. The count of mistakes is not a fully accurate measure of quality, as some mistakes are more important than others. Still, we did not assign weights to them to keep the measure as objective as possible.

Students may have acted so as to please our expectancies. This threat was reduced by (a) not telling the students the exact hypotheses behind our experiments and (b) having the experiments supervised by a researcher who is not involved in the lectures participants were enrolled to (SE for Zurich, MDE for Rennes). In the next section, we present and discuss the results obtained during the experiments.

IV. RESULTS ANALYSIS AND INTERPRETATION

In total, 86 students participated to our experiments: 14 Miage and 23 GL students from Rennes and 49 SE students from Zurich. From them, we gathered 72 complete datasets including both diagrams and answers to surveys. The datasets are spread almost equally in each group (19 in group A, 15 in B, 19 in C, 19 in D). Based on the preexperimental survey, most students consider themselves as experienced with modeling and UML in general. However, the average experience is lower with state machines than with class diagrams. Most students found that both the modeling tasks and the case descriptions were clear and that they had enough time to do the modeling. In average, participants took 39 minutes for the class diagram and 32 minutes for the state machine. Footprinting had no impact on the time used for modeling. In the remainder of this section, we first analyze the impact of footprinting on actual quality and then its impact on perceived quality.

A. Actual Model Quality

To assess the actual quality of models, we compared the models to our reference models. We used more than one reference model, as the information from a textual description may be modeled correctly in different ways. Besides, we ignored differences in names, as long as the names were meaningful. Some other deviations may have been acceptable (e.g., the presence of business operations in a class diagram), but we marked them all to avoid introducing some bias in the results. In the class diagrams, we identified a total of 72 possible mistakes. 31 mistakes are related to confinement, 17 to completeness and 24 to correctness. For the state machines, we have listed 52 mistakes. 16 mistakes are confinement issues, 16 are completeness problems and 20 are correctness mistakes.

The actual quality of models is presented as boxplots in Figure 2. Figure 2a shows the number of mistakes made in class diagrams, while Figure 2b displays the number of mistakes made in the state machines. The results are grouped by the availability of the static metamodel footprint (the treatment) and the various quality aspects (completeness, confinement, correctness and overall quality). The y-axis represents the number of mistakes made. The higher the number of mistakes, the lower the actual quality of a model. In a boxplot, the boxes have lines at the first quartile, median, and third quartile. The notches around the medians represent their 95% confidence interval, which is useful for comparing medians.

Table II displays the median number of mistakes made. For each kind of diagrams and for each quality attribute, we compare the median number of mistakes made by students who knew the static metamodel footprint with the median number of mistakes made by students who did not. For this comparison, we use a Wilcoxon rank-sum test whose p-value is displayed in Table II. We cannot reject any nullhypotheses, as no p-value is above the usual significance level α = 5%. Thus, we use a parametric model to further investigate whether footprinting had a statistically significant impact on the actual model quality.

The numbers of mistakes follow Poisson probability distribution laws. Table III displays the average number of mistakes made by students in their models. The average is an unbiased estimator of the λ parameter of the Poisson law. Overall, students made 7.86 mistakes in their class diagrams and 5.13 mistakes in their state machines. Table III also provides the 95% confidence intervals for λ. The difference between the quality with and without footprinting are not statistically significant, as all confidence intervals for λ overlap.

Interestingly, the results obtained for the class diagram contradict those obtained for the state machine. For class Footprinting diagrams, footprinting improves all quality aspects except completeness. Indeed students with footprinting makes fewer mistakes in total (AOAQ) than without footprinting: fewer confinement mistakes (ACON), fewer correctness mistakes (ACOR) but more completeness mistakes (ACOM). In contrast, footprinting only improves completeness for state machines (ACOM), while degrading all other qualities (ACON, ACOR and AOAQ).

B. Perceived Model Quality

The ratings of perceived quality are plotted as boxplots in Figure 3. Given the ordinal nature of Likert scales, we use the median to characterize the perceived model quality (see Table IV). For class diagrams, footprinting had almost no impact on the perceived quality of models. For state machines, footprinting reduces the perceived confinement (PCON) of models, but improves the perceived correctness (PCOR) and the perceived overall quality (POAQ). However, the p-values of the Wilcoxon rank-sum tests are all above the α = 5% level required to reject a null hypothesis. Thus, the differences between the medians are not statistically significant.

In general, the perceived quality is coherent with the actual quality. To assess this coherence, we group the models according to their perceived quality and we compute the average actual quality for each group. Table V displays these averages. For example, 19 students assessed the overall quality of their model with a 2. The average number of mistakes for these students is 9.63. In comparison, 28 students assessed the overall quality of their class diagram with a 4 and made, in a average, 6.46 mistakes. In most cases, the higher the perceived quality is, the lower the number of mistakes is, and, thus, the higher the actual quality is. We observe similar results for models made with and without static metamodel footprints. Thus, in Table V, we only show the results for all models, no matter whether the static metamodel footprint was available for their construction.

V. DISCUSSION

We have hypothesized that footprinting would improve the quality of models and increase the confidence of the modelers in the quality of their models. After all, footprinting provides additional hints that should have helped the participants in their modeling tasks. However, our experiments did not demonstrate any statistically significant effect of footprinting on model quality. Unfortunately, our post experimental survey did not include any question to further explain this issue. Thus, more research is needed to investigate whether this is due to the design of our experiments. In this section, we propose and discuss possible explanations, providing some direction for future research.

The modeling assignements may have been too simple to demonstrate the benefits of footprinting. Indeed, the case descriptions do not include many irrelevant details with respect to the given modeling purpose. In the postexperimental surveys, most participants disagreed that both case descriptions were too detailed. Thus, the modelers had no difficulty to figure out what was to be modeled, no matter whether or not they knew the static metamodel footprint. The results may have been different if we had used a large case description with many superfluous details. In such a case, the static metamodel footprint may help for deciding which details should be included in the model and which details should be left out.

Participants may have overlooked the static metamodel footprint or may not know how to exploit this information. Indeed, some kinds of mistakes would have been avoided if the static metamodel footprint had been used properly. The number of students committing these mistakes supports this explanation. For example, 30 students forgot to model the triggers on transitions in the state machines: 18 students had the static metamodel footprint, 12 had not. For the class diagrams, 11 students modeled realization relationships, which are superfluous for the purpose at hand: 8 of them had the static metamodel footprints, 3 had not. This also suggests that our measure for actual quality is not responsible for the results.

Future research should clarify to what extent the form of footprinting reduces its impact on model quality. We chose to provide the static metamodel footprint as text, directly integrated in the modeling task without highlighting it. Alternatively, we could have extended the modeling tool to provide the students with some feedback about the model footprint, displaying warning messages about superfluous or missing elements. We could also have presented an example of a model containing all the constructs in the static metamodel footprint. In these forms, the participants can more easily use the information provided by footprinting than in its basic form and they can less easily overlook it.

In our experiments, we did not introduce footprinting and we did not train participants to use it. Thus, participants may not be experienced enough in metamodeling or in the UML metamodel to properly exploit the static metamodel footprint. While this may be the case for the students in Zurich, these subjects were taught to the students from Rennes in the MDE lecture. Yet, there is no significant difference in the results from both sites. Besides, most students considered themselves as experienced in modeling, in UML and in class diagrams during the pre-experimental survey. Still, footprinting may require some training before it can actually deliver some benefits to the modelers.

VI. CONCLUSION AND FUTURE WORK

Creating requirement models is not an easy task. In addition to eliciting information about the original, modelers need to understand the purpose of their model. Often, this purpose is kept implicit and the only indication is a modeling language. In a MDE setting, where the model is often used to feed some model operations (like queries or transformations), footprinting can be used to assess and improve the quality of models. Still, the benefits of footprinting on model quality were not yet empirically investigated.

To investigate the benefits of footprinting, we conducted a pair of controlled experiments involving students from two universities, Rennes and Zurich. Participants had to model a class diagram and a state machine. Some participants used footprinting for the class diagram, while others used it for the state machine. We evaluated both the actual quality of models -by counting the number of mistakes with respect to a reference model -and the quality perceived by the modeler -by using a questionnaire.

Our results are inconclusive: the effect of footprinting on quality is not statistically significant and the results in the class diagram contradict those in the state machine. We believe that these results can be accounted by the way we presented footprinting to the participants: a static metamodel footprint in text form. Participants may have overlooked it or may not have used it properly. We would have obtained different results if we had trained the participants to footprinting or if we had provided them with a modeling tool displaying feedback on the confinement and the completeness of their models.

Further research is needed for establishing the impact of footprinting on model quality. Furthermore, footprinting is not only meant for creating better models, it can also be used to better understand model operations. Thus, investigating the impact of footprinting on the comprehension of an operation is another direction for demonstrating the benefits of footprinting empirically. Finally, one could evaluate to what extent a set of operations is a good characterization of a model's purpose.

APPENDIX A. DENTAL CLINIC Instructions

The following section describes a system to manage patient records in a dental clinic. Your task is to produce an analysis model of this system with a class diagram based on the description. Your model will be used to generate the skeleton of a glossary. We will compare your model to a reference model: Make sure to include every relevant piece of information, but not more!

Static Metamodel Footprint

Note: The static metamodel footprint has not been handed to all participants.

In this model, we are only interested in the following items:

• Classes name of the class

• Attributes name of the attribute

• Associations

• Association Ends multiplicity of the association end

• Generalizations

Case Description

A clinic with three dentists and several dental hygienists needed a system to help administer patient records. This system does not keep any medical records. It only processes patient administration.

Each patient has a record with his/her name, date of birth, gender, date of first visit, and date of last visit. Patient records are grouped together under a household. A household has attributes such as name of head of household, address, and telephone number. Each household is also associated with an insurance carrier record. The insurance carrier record contains name of insurance company, address, billing contact person, and telephone number.

In the clinic, each dental staff person also has a record that tracks who works with a patient (dentist, dental hygienist, x-ray technician). Because the system focuses on patient administration records, only minimal information is kept about each dental staff person, such as name, address, and Information is maintained about each office visit, such as date, insurance copay amount (amount paid by the patient), paid code, and amount actually paid. Each visit is for a single patient, but, of course, a patient will have many office visits in the system. During each visit, more than one dental staff person may be involved in the visit by doing a procedure. For example, the x-ray technician, dentist, and dental hygienist may all be involved in a single visit. In fact, some dentists are specialists in such things as crown work, and even multiple dentists may be involved with a patient. Detailed information is kept about procedures performed by a staff person during a visit. This information includes type of procedure, description, tooth involved, the copay amount, the total charge, the amount paid, and the amount insurance denied. Finally, the system also keeps track of invoices. There are two types of invoices: invoices to insurance companies and invoices to heads of household. Both types of invoices are fairly similar, listing each visit, the procedures involved, the patient copay amount, and the total due. Obviously, the totals for the insurance company are different from the patient amounts owed. Even though an invoice is a report (printed out), it also maintains some information such as date sent, total amount, amount already paid, amount due and also the total received, date received, and total denied. (Insurance companies do not always pay all they are billed.)

Reference Model

A reference model is given in Figure 4.

APPENDIX B. SHIPMENT

Instructions

The following section describes the behavior of a shipment by Union Parcel Shipments. Your task is to document this behavior with a state machine. Your model will be used to generate an implementation based on the State design pattern. We will compare your model to a reference model: Make sure to include every relevant piece of information, but not more!

Static Metamodel Footprint

Note: The static metamodel footprint has not been handed to all participants.

In this model, we are only interested in the following items:

• States name of the state

• Transitions trigger of the transition (as signal event)

• Signal Events name of the event

• Initial States • Final States

Case Description

A shipment is first recognized after it has been picked up from a customer. After it is in the system, it is considered to be active and in transit. Every time it goes through a checkpoint, such as arrival at an intermediate destination, it is scanned, and a record is created indicating the time and place of the checkpoint scan. The status changes when it is placed on the delivery truck. It is still active, but now it is also considered to have a status of delivery pending. Of course, after it is delivered, the status changes again.

From time to time, a shipment has a destination that is outside the area served by Union. In those cases, Union has working relationships with other courier services. After a package is handed off to another courier, it is noted as being handed over. In those instances, a tracking number for the new courier is recorded (if it is provided). Union also asks the new courier to provide a status change notice after the package has been delivered.

Unfortunately, from time to time a package gets lost. In that case, it remains in an active state for two weeks but is also marked as misplaced. If after two weeks the package has not been found, it is considered lost. At that point, the customer can initiate lost procedures to recover any damages.

Reference Model

A reference model is given in Figure 5.

  My model is a good model. (POAQ) • My model contains all relevant information. (PCOM) • My model only contains relevant information. (PCON) • My model only contains valid information. (PCOR) Method: All participants used the same modeling languages (UML class diagrams and UML state machines).

Figure 2 :

 2 Figure 2: Actual quality of models

Figure 3 :

 3 Figure 3: Perceived quality of models

1 Figure 4 :

 14 Figure 4: Reference model for the dental clinic domain

Figure 5 :

 5 Figure 5: Reference model for the behavior of a shipment

Table I .

 I For example, participants in group A first model the class diagram without the static metamodel footprint. They then proceed to the state diagram, this time with the static metamodel footprint.

Table I :

 I Experiment design

			Group A	Group B	Group C	Group D
	Tasks	1st 2nd	Class Diagram With Footprint State Machine Without Footprint	Class Diagram Without Footprint State Machine With Footprint	State Machine With Footprint Class Diagram Without Footprint	State Machine Without Footprint Class Diagram With Footprint

Table II :

 II Median number of mistakes

		ACOM	ACON	ACOR	AOAQ
	No Footprinting	2.50	3.50	2.00	8.00
	Footprinting	2.50	3.00	1.00	7.00

Table III :

 III Confidence interval of λ for the counts of mistakes

			ACOM	ACON	ACOR	AOAQ
	Class Diagram	Altogether No Footprinting Footprinting	2.53	3.79	1.74	8.06
	State Machine	Altogether No Footprinting

Table IV :

 IV Median rating of perceived quality

				PCOM	PCON	PCOR	POAQ
			No Footprinting	4.00	4.00	4.00	3.00
	Class	Diagram	Footprinting P-Value	4.00 0.59	4.00 0.22	4.00 0.38	3.00 0.33
			Reject H0?	No	No	No	No
			No Footprinting	3.00	4.00	3.00	2.00
	State	Machine	Footprinting P-Value	3.00 0.73	3.00 0.28	4.00 0.57	3.00 0.23
			Reject H0?	No	No	No	No

Table V :

 V Coherence between perceived and actual model quality

					Actual Quality
				COM	CON	COR	OAQ
	Perceived Quality	State Machine Class Diagram	1 2 3 4 5 1 2 3 4 5	5.50 (2) 3.82 (11) 2.70 (20) 2.18 (34) 1.40 (5) 3.40 (5) 2.33 (21) 1.23 (22) 0.88 (24)	2.67 (3) 3.75 (16) 3.62 (47) 3.67 (6) 1.67 (3) 1.00 (3) 1.41 (27) 1.67 (33) 2.67 (6)	1.50 (2) 2.00 (2) 1.65 (20) 1.68 (44) 1.00 (4) 2.25 (4) 1.90 (10) 1.75 (24) 2.03 (32) 1.50 (2)	9.63 (19) 8.08 (25) 6.46 (28) 7.00 (8) 5.71 (28) 4.13 (23) 4.25 (12) 7.00 (1)

This lecture is taught by Prof. Martin Glinz. The description and the material of the module can be found at http://bit.ly/P8RrNZ (in German)

This lecture is taught by Dr. Noël Plouzeau. The description of the module can be found at http://bit.ly/OdIKBU (in French)

http://www.nomagic.com/products/magicdraw.html

ACKNOWLEDGMENT

Our work is partially funded by the Swiss National Science Foundation under the project 200021 134543 / 1. We are grateful to Irina Todoran and Clément Guy who helped on the organisation of the experiments and also the students from Rennes and Zurich who participated in the experiments.


Elena ; Bozzetta
email: elena.bozzetta@izsto.it

Marzia Pezzolato
email: marzia.pezzolato@izsto.it

Cristiana Maurella
email: cristiana.maurella@izsto.it

Katia Varello
email: katia.varello@izsto.it

Benedetta Guia

Rosa Richelmi
email: guia.richelmi@izsto.it

Carolina Draisci
email: carolina.ferranti@iss.it

Antonio Ferranti

Maria D'angelo

Caramelli

G B Richelmi

R Draisci
email: rosa.draisci@iss.it

C Ferranti

A D' Angelo

M Caramelli
email: maria.caramelli@izsto.it9

Dvm Phd

S C Istopatologia

Test Rapidi

Development of an enhanced histopathological approach to detect low-dose

Keywords: Drug residues -hormones, Residues, Veterinary drug residues, Veterinary drug residues -anabolic steroids illicit treatment, dexamethasone, calves, thymus, histology 47 48 2005). Furthermore, glucocorticoids may be responsible for Cushing

In the attempt to devise an accurate biological method that could detect illicit drug treatment in food-producing animals, we characterized microscopic morphologic alterations of the thymus in veal calves administered low-dose dexamethasone versus control animals. For this purpose, 122 male calves were farmed for 6 months in controlled condition: 81 animals were orally administered dexamethasone (0.4 mg/day) for 20 days during the sixth month and the remaining 41 were kept as control. Urine samples were collected systematically during the treatment period, the suspension period and at the slaughterhouse. All animals were slaughtered 10 a day starting from 10 days after the last dexamethasone administration and the thymus was sampled for histological examination. The difference between the two animal groups was evaluated by means of a non parametric test of hypothesis.

No residues were detected in the urines collected since the third day after the last administration, whereas morphometrical analysis of the thoracic thymus revealed a significant decrease in the cortex: medulla ratio in the treated animals (p<0.0005). We can conclude that this histological approach offers encouraging prospects as a screening method to overcome current limitations in controlling growth promoter abuse. (Gardini et al., 2006;Toffolatti et al., 2006;Reiter et al., 2007;[START_REF] Odore | Development of a metabolomic approach based on liquid chromatography-high resolution 289 mass spectrometry to screen for clenbuterol abuse in calves[END_REF] .

77

78

Several studies on young animals have found that illegally administered glucocorticoids 79 promote a physiological reduction of the thymus identifiable as gross and microscopic lesions 80 (Groot et al., 1998;Schilt et al., 1998;Biolatti et al., 2005;Cannizzo et al., 2008) [START_REF] Courtheyn | Recent developments in the use and abuse of 292 growth promoters[END_REF]Biolatti 85 et al., 2005;Carletti et al., 2007), with the inevitable consequence of increased risk for 86 consumer health (Groot et al., 1998).

88

This situation further stresses the importance of identifying chemical-induced thymic atrophy 89 and differentiating it from physiological thymic involution. Due to the potential difficulties of 90 differentiating those latter changes from chemical-related effects, histopathological 91 examination should be carried out by comparing treated with control animals (Elmor, 2006).

92

The aim of this study was to determine whether morphological changes in the thymus could

151

The API 3000 triple quadrupole mass spectrometer equipped with an atmospheric pressure 152 chemical ionization (APCI) -heated nebulizer (HN) source was set in negative ionization 153 mode with a source temperature of 400°C and a needle current of -3 µA; ultra pure nitrogen 154 was used as curtain and collision gas, while ultra pure air was used as nebulizer and 155 auxiliary gas.

  simple biological indicators that can aid in laboratory analysis are needed.

  of chemical growth promoters in food-producing animals constitutes a 50 potential risk for human health. Increasingly widespread is the administration of synthetic 51 corticosteroids that can mimic, block or interfere with hormone actions (Vazquez et al., called omics techniques (transcriptomic, proteomic and metabolomic), based on the 76 simultaneous detection of biomarkers predictive of administration of specific substances

935

  constitute a reliable treatment parameter following administration of low-dose glucocorticoid 94 for growth promotion in veal calves. To do this, we devised an accurate histopathology-95 based biological method that may prove useful in detecting glucocorticoid compound abuse 96 by the meat producing industry. 122 male veal calves, aged between 15 and 35 days, were bought by local 101 breeders and randomly divided in two groups homogenous for body weight and age and 102 farmed in two boxes in the same condition for 6 months. Each box had its own crib, multiple 103 drinking troughs, and a dedicated automated milk feeder system. To protect the animals 104 against infections, all were vaccinated against IBR, Para influenza (PI3), BRSV and BVDV 105 (CATTLEMASTER® 4 Pfizer Animal Health; New York, USA). Clinical controls were carried 106 out daily by a veterinarian and treatments for occurring infections were performed without 107 using hormonal active substances. All animals had free access to fresh water and were fed standard milk replacer with an 109 automatic milk feeder until 4 months of age, then 0.5 kg of corn was added to the diet twice a 110 day. Before administration, all feeds, milk replacer and corn were analyzed with an ELISA 111 test to exclude the presence of hormonal active substances. 112 During the sixth month, 81 randomly selected animals received dexamethasone 21 disodium 113 phosphate (0.4 mg/day per os) mixed with the milk for 20 days while the remaining 41 114 animals were used as controls and admistered milk and placebo mixed too. The study was 115 designed to achieve a statistical power of 80% with Confidential Interval 95%; an expected 116 frequency of disease in the treated group of 27 % and a frequency of 5% in the control group 117 were considered. On this basis a simple random and representative sample of 40 controls 118 and 80 treated animals was evaluated to be consistent with a detection of an Odds Ratio of 119 7.120The trial was conducted in a blinded manner.121The animals were slaughtered 10 a day in an EC certified slaughterhouse starting from 10 122 days after the last drug administration. All experiments were carried out according to 123 European Economic Community Council Directive 86/609, recognised and adopted by the 124 Italian Government (D.L. 27/01/1992 no. 116). 125 Chemical analysis 126 Urine samples of 10 treated animals and 10 controls were collected when spontaneous 127 urination occurred, the day before the first administration, 10 days later, and the last day of 128 treatment; urine was also collected after the last treatments, 1, 2, 5 and 10 days later, and at by solid phase extraction (SPE) using an Oasis hydrophile lipophile 137 balance (HLB) cartridge ( 60 mg, 3 mL) (Waters, Milford, Ma, USA). After elution with 138 methanol, the solvent was removed at 40°C under nitrogen stream and the residue was 139 dissolved in 100 µl of methanol and injected into the liquid chromatography-mass 140 spectrometry/mass spectrometry (LC-MS/MS) system. 141 142 Liquid chromatography-mass spectrometry/mass spectrometry analysis 143 Analyses were carried out using a liquid chromatography system (Perkin Elmer Series 200 144 Micro Pump with a PE Series 200 autosampler, PerkinElmer, Waltham, MA, USA). The 145 chromatographic separations were obtained under gradient conditions at room temperature 146 using a reversed phase HPLC column (250 mm × 4.60 mm I.D., 4 mm POLAR-RP 80 A) 147 Synergi C18 (Phenomenex, Torrance, CA, USA) with a C18 guard column (4 mm × 2 mm 148 I.D.) (SecurityGuard, Phenomenex). The mobile phase was composed of 1% acetic acid and 149 acetronitrile and the flow rate was 0.8 mL min -1 .

  CCα) and detection capability (CCβ) for dexamethasone, calculated in 158 accordance with Commission Decision 2002/657/EC [16], were 0.3 ng mL -1 and 0.4 ng mL -1 , the central area of the thoracic thymus of each animal was sampled, fixed 165 in 4% buffered formaldehyde at room temperature for 2 days, routinely processed, 166 embedded in paraffin wax, sectioned at 3-5 µm and stained with hematoxylin and eosin (HE).

167

  Routine histology by light microscopy was performed on HE-stained paraffin sections. blindly performed. The presence of fat, as a possible 171 indirect indicator of the degree of gland atrophy, was evaluated and a score from 1 to 3 (mild, 172 moderate, severe) was attributed to the pattern. The patterns of the optical and dyeing 173 parameters were evaluated based on the histopathologist's experience rather than by 174 as the null hypothesis that there would be no difference between the 192 two groups; to verify the presence of a significant difference in the degree of thymic atrophy 193 between the two groups, a test for linear trend was performed and the difference between valuable feature to distinguish between beclomethasone treated and untreated 230 veal calves. Later Elmore(Elmor, 2006) included this parameter among criteria to distinguish 231 between chemical-induced and age or stress-related thymic atrophy. Other possible causes of 232 thymus ahrophy, as autoimmune phenomenon and immunodeficiency syndromes, have been233shown to affect cellular density with symmetrical reduction of thymic cortical and medullary and lead to muscle atrophy. Once often combined with beta-agonists 246 and/or anabolic steroids, more recently corticosteroids are administered alone, because even 247 low doses of glucocorticoids have been found to improve feed intake, increase live weight 248 gain, reduce feed conversion ratio, reduce nitrogen retention and increase water retention and 249 fat content(Istasse et al., 1989). Yet precisely because of their use at low dosage, their 250 detection in matrices of biological origin is far from evident.

  Fig.1: Treated animal: Thymus, HE 4x

  Box plots representing the significant difference in the cortex: medulla ratio between treated and control animals. 90x65mm (300 x 300 DPI)

Table 1

 1 Dexamethasone concentration in urine samples of treated and control bovines 339

Table 2 :

 2 The risk of gland atrophy is expressed as odds ratio (OR) and relative 95% 340 Confidential Interval: there was no increase in infiltrating fat in treated vs. control animals.

Table 1

 1 Dexamethasone concentration in urine samples of treated and control bovines.

			Food Additives and Contaminants		Page 18 of 18
	DEXAMETHASONE CONCENTRATION IN URINE SAMPLES (ng mL -1 )
	TREATED BOVINES
	Sampling day	0 th	10 th	20 th	21 st	22 nd	25 th	30 th 30 th
									slaughterhouse
	Average	n.d. 2.0	2.1	1.2	1.0	0.4	n.d. n.d.
	Concentration interval --	0.5-9.1 0.3-5.9 0.4-3.1 n.d.-2.5 n.d.-2.5 --	--
	>CCα (0.3 ng mL -1 )	0/7 10/10	10/10	10/10	9/10	5/10	0/10 0/10
	CONTROL BOVINES
	Sampling day	0 th	10 th	20 th	21 st	22 nd	25 th	30 th 30 th
									slaughterhouse
	Average	n.d. n.d.	n.d.	n.d.	n.d.	n.d.	n.d. n.d.
	Concentration interval --	--	--	--	--	--	--	--
	>CCα (0.3 ng mL -1 )	0/8 0/9	0/10	010	0/10	0/10	0/8	0/9

http://mc.manuscriptcentral.com/tfac Email: fac@tandf.co.uk

Table 2 :

 2 The risk of gland atrophy is expressed as odds ratio (OR) and relative 95% Confidential Interval: there was no increase in infiltrating fat in treated vs control animals.

	Page 19 of 18
	Fat Score	Odds Ratio		95% CI
	1	1	-	-
	2	1.2	0.1	20.4
	3	1.8	0.1	30.3
	chi square for trend=0.16,		p=0.7

http://mc.manuscriptcentral.com/tfac Email: fac@tandf.co.uk

Food Additives and Contaminants

http://mc.manuscriptcentral.com/tfac Email: fac@tandf.co.ukFood Additives and Contaminants

device (Leica DMD108 Digital microimaging device for clinical diagnostics labs) to evaluate

Table2shows the trend and the chi square value for trend.

et al., 2005), can induce morphologic changes in the thymus, resulting in a significant

reduction in the cortex: medulla ratio. In 1998 Schilt(Schilt et al., 1998) first reported this

M., Carletti, M., Biolatti, P. G.,Barbarino, G. (2005) Effects of low-dose dexamethasone

on thymus morphology and immunological parameters in veal calves. J. Vet. Med A.

Physiol. Pathol. Clin. Med., 52,[202][203][204][205][206][207][208]

biomarkers of exposure in veal calves illegally treated with dexamethasone. Toxicol. In

Elmore, S. A. (2006) Enhanced histopathology of the thymus.Toxicol. Pathol., 34,[656][657][658][659][660][661][662][663][664][665]

Gardini, G., Del Boccio, P., Colombatto, S., Testore, G., Corpillo, D., Di Ilio, C., Urbani, A.,

Nebbia, C. (2006) Proteomic investigation in the detection of the illicit treatment of calves

with growth-promoting agents. Proteomics 6, 2813-2822.

Groot, M. J., Schilt, R., Ossenkoppele, J.S., Berende, P.L.,Haasnoot, W. (1998)

Istasse, L., De Haan, V.,Van Eanaeme, C., Buts, B., Baldwin, P., Gielen, M., Demeyer, D.,

Bienfait, J. M. (1989) Effect of dexamethasone injections on performance in a pair of

monozygotic cattle twins J. Anim. Physiol. A:Anim. Nutr., 62,[150][151][152][153][154][155][156][157][158]

Jubb,Kennedy and Palmer's (2007) Pathology of domestic animals. Canada edited by M

Pharmacol. Ther., 29,[91][92][93][94][95][96][97]

Reiter, M., Walf, V. M., Christians, A., Pfaffl, M. W., Meyer, H. H. (2007) Modification of

mRNA expression after treatment with anabolic agents and the usefulness for gene

expression-biomarkers. Anal. Chim.Acta, 586,[START_REF] Odore | Development of a metabolomic approach based on liquid chromatography-high resolution 289 mass spectrometry to screen for clenbuterol abuse in calves[END_REF][74][75][76][77][78][79][80][81]

Schilt, R., Groot, M. J., Merende, P. L., Ramazza, V., Ossenkoppele, J. S., Haasnoot, W.,

European Communities L 224, 1-8.

Toffolatti, L., Rosa Gastaldo, L., Patarnello, T.,Romualdi, C., Merlanti, R., Montesissa, C.,

Poppi, L., Castagnaro, M.,Bargelloni, L. (2006) Expression analysis of androgen-

Acknowledgments

269

This research was founded by the Italian Ministry of Health (RF-IZP-2006-364645)

three increasing degrees of atrophy was expressed as odds ratios (OR). Dexamethasone residues (> CCα) were detected in all the urine samples collected from the 199 treated bovines at the 10 th and 20 th day of treatment and the day after the last administration 200 (21 st day). At the 22 nd and 25 th day of sampling, dexamethasone residues were found in 90%

201

and 50% of the analyzed samples, respectively (Table 1). No residues were detected in the 202 samples collected the day before the treatment, 10 days later, or in the samples collected 203 from the urinary bladder at the slaughterhouse. These findings showed that the drug 204 disappeared shortly after treatment, precluding a long-term survey.

205

No dexamethasone residues were detected in the control animals; this precluded any further 206 statistical analysis.

213

The amount of fat infiltrating the thymic tissue was not found to be significantly associated 214 with the steroidal treatment.

Microscopic examination at low magnification of the thymus glands showed a striking 218 tinctorial change: the thymus of the controls was darker than that of the treated animals. At a 219 higher magnification, a decrease in the cortical area in the treated animals was visible.

220

Qualitative overall assessment indicated that the cortical: medullary ratio was reversed in the 221 treated animals (Fig. 1) in comparison with the normal 2:1 ratio (Fig. 2).

222

The two-sample Wilcoxon rank-sum (Mann-Whitney) test highlighted a significant decrease 223 in the cortex: medulla ratio between the treated animals and the controls (p=0.0005) (Graph 224 1).

225

Our findings demonstrate that low dosages of dexamethasone administered as a growth 226 promoter in veal calves, according to a protocol often illegally adopted in farm practice (Biolatti We wish to thank Dr. Antonio Longo for editing assistance.

271

Brambilla, G., Fiori, M., Pierdominici, E., Antonucci, G., Giorni, P., Ramazza, V., Zucchi, M.

278

(1998) A possible correlation between the blood leukocyte formula and the use of