Georgios Kanakis

Djamel Eddine Khelladi

Stefan Fischer

Michael Tröls

Alexander Egyed

An Empirical Study on the Impact of Inconsistency Feedback during Model and Code Co-changing

Keywords: Model Driven Engineering, Consistency Feedback, Change Propagation, Models, Code

Model and code co-changing is about the coordinated modification of models and code during evolution. Intermittent inconsistencies are a common occurrence during co-changing. A partial co-change is the period in which the developer changed, say, the model but has not yet propagated the change to the code. Inconsistency feedback can be provided to developers for helping them to complete partial co-changes. However, there is no evidence whether such inconsistency feedback is useful to developers. To investigate this problem, we conducted a controlled experiment with 36 subjects who were required to complete ten partially completed change tasks between models and code of two non-trivial systems. The tasks were of different levels of complexity depending on how many model diagrams they affected. All subjects had to work on all change tasks but sometimes with and sometimes without inconsistency feedback. We then measured differences between task effort and correctness. We found that when subjects were given inconsistency feedback during tasks, they were 268% more likely to complete the co-change correctly compared to when they were not given inconsistency feedback. We also found that when subjects were not given inconsistency feedback, they nearly always failed in completing co-change tasks with high complexity where the partially completed changes were spread across different diagrams in the model. These findings suggest that inconsistency feedback (i.e. detection and repair) should form an integral part of co-changing, regardless of whether the code or the model changes first. Furthermore, these findings suggest that merely having access to changes (as with the given partially completed changes) is insufficient for effective co-changing.

Introduction

Model-Driven Engineering (MDE) has shown to be effective in the development and maintenance of large scale and embedded systems [START_REF] Box | Statistics for Experimenters[END_REF][START_REF] Basili | Building knowledge through families of experiments[END_REF]. Models are used in all development stages, from specifying the customers' requirements, design, all the way to source code. The benefits range from increased productivity to reduced time to market [DCF + 16]. These benefits, however, hinge on the assumption that model and code remain consistent over time which is often not the case. Changes in models and code cause inconsistencies and subsequent errors if these inconsistencies are not recognized in a timely manner [START_REF] Egyed | Automatically detecting and tracking inconsistencies in software design models[END_REF]. Any conclusions or automation based on them cannot be trusted for as long as these inconsistencies remain. Inconsistencies are better detected with today's consistency checking mechanisms [START_REF] Krishnan | Measuring process consistency: implications for reducing software defects[END_REF]. However, literature tends to treat inconsistencies as errors -typically as errors within models (and their representations). Indeed, there is very little discussion about inconsistencies between model and code due to evolutionary changes.

That is, model and code inconsistencies are often the result of evolutionary changes: changes where the model might have been evolved in response to changing requirements or defects, but the code has not been updated to reflect those changes. Here, inconsistencies imply that either model, code or both are out of date. Consequently, inconsistencies between model and code are cues about the incomplete propagation of changes from model to code or code to model. These inconsistencies occur intermittently -after, say, the engineer changed the model but before the engineer propagated the change to the code. The questions we are interested in here are: do engineers benefit from inconsistency feedback during model and code evolution? Or more concretely: does this inconsistency feedback benefit effort (i.e., save time) or does it improve quality (i.e., result in more complete change propagations)? Or both? The goal of this paper is to explore these questions. To examine the usefulness of inconsistency feedback during model and code co-change, we performed a controlled experiment with 36 subjects -all bachelor level students at the Johannes Kepler University but with average professional work experience of almost 2 years (many students do work part-time). The subjects were confronted with model or source code changes and they were then asked to complete those changes (i.e., to repair the inconsistencies that had been caused by those changes). The key distinguishing factor was that subjects were sometimes given inconsistency feedback and other times not. This way we could measure whether the availability of inconsistency feedback was beneficial to the subjects while they repaired the inconsistencies. Specifically, we measured their effort and their correct/complete propagation.

To ensure wider applicability, we asked each subject to work on ten change request tasks covering two, non-trivial software systems. Five tasks were about changes to an open source monopoly game called "Matador" and the other five tasks were about a proprietary calendar application called "Calendarium". Each task could come in two possible change directions: with the model-to-code change direction, subjects were presented with model changes and were asked to update the code (i.e., propagate changes from model to code). With the code-to-model change direction, subjects were asked to update the model in response to code changes (i.e., propagate changes from code to model). The change direction, task order, and availability of inconsistency feedback was randomized for each subject. Hence, every subject was asked to perform some change propagations with and some without inconsistency feedback; some from model to code and some from code to model; and all tasks in a different order.

Out of the possible 360 tasks (if all 36 subjects had managed to complete all ten change requests), we obtained 281 completed solutions -7.9/10 completed tasks per subject on average. Our findings showed that subjects working with inconsistency feedback solved 268% more correct tasks than those without inconsistency feedback (59 correct tasks with consistency feedback as opposed to 22 tasks without feedback). We found that the correctness improved regardless of the type of project or the direction of the change. Hence, this benefit is solely the result of the availability of inconsistency feedback. Inconsistency feedback thus helps engineers to correctly propagate changes between model and code. Surprisingly, the findings seemed to show little difference in effort for subjects with and without inconsistency feedback. A deeper analysis revealed that, when deprived of inconsistency feedback, subjects mostly failed to solve certain tasks entirely. We found that this applied particularly to tasks with higher complexity where the changes affected multiple diagrams of a model (e.g., engineers had to understand a class and a sequence diagram change together to understand how to co-change the code).

Our results imply that inconsistency feedback is an essential complement for change propagation between model and code. Merely understanding model changes is not sufficient to understand how to co-change the code and vice versa. Given the strong correctness benefit, we argue that the use of inconsistency feedback should be common practice not just for error detection but also be change propagations/co-evolution given that there is almost no cost involved in using consistency checking mechanisms (e.g., Nentwich et al [START_REF] Nentwich | Flexible consistency checking[END_REF], Egyed [START_REF] Egyed | Automatically detecting and tracking inconsistencies in software design models[END_REF], Musuvathi et al [MPC + 02]) and given that many commonly usable consistency rules are available [START_REF] Damiano Torre | Uml consistency rules: A systematic mapping study[END_REF]. We believe that there are likely subsequent benefits that stem from the quicker availability of inconsistency feedback for more correct models or code which we did not measure since it was out of the scope of this paper. Given that we chose two reasonably complex projects, ten diversely complex change request tasks, two task directions, and different kinds of UML diagrams, we believe that our findings have significant value to the software engineering community. The rest of the paper is structured as follows. Section 2 discusses related work in the area of consistency checking and similar experiments. Section 3 presents the experimental set-up in-depth and the variables that were taken into consideration. Section 4 reports the results and the findings of the experiment. In Section 5, we discuss the threats to validity and how threats were mitigated or avoided. Finally, Section 6 concludes this paper and discusses possible areas of future research.

Related Work

To the best of our knowledge, no empirical study ever investigated the benefits of inconsistency feedback for engineers between code and models. However, there is significant related work on consistency checking mechanisms and consistency rules. Krishnan and Kellner [START_REF] Krishnan | Measuring process consistency: implications for reducing software defects[END_REF] reported the relation between software defects and the lack of software process consistency in an empirical study. The study was based on 45 products from leading software vendors which adopted various practices of the Capability Maturity Model (CMM). Their focus was on the number of produced defects in a product when the vendors do not consistently follow the processes described in the CMM. The authors found that consistent adoption of CMM practices and personal capability of the team reduce the number of product defects in a developed system.

Mäder and Egyed [START_REF] Mäder | Assessing the effect of requirements traceability for software maintenance[END_REF][START_REF] Mäder | Do developers benefit from requirements traceability when evolving and maintaining a software system[END_REF] conducted a similar experiment to measure if traceability benefits software developers when they evolve and maintain a software system. [START_REF] Mäder | Do developers benefit from requirements traceability when evolving and maintaining a software system[END_REF] extended the work presented in [START_REF] Mäder | Assessing the effect of requirements traceability for software maintenance[END_REF] and presented the effect on performance that the subjects had with the use of traces during their maintenance and development tasks. They found that effort and quality improved if subjects had available traceability. While traceability links are not the same as consistency rules, they do both imply dependencies -albeit in different ways. Hence, it can be argued that any manner which helps developers understanding relationships say between model and code, is useful.

Spanoudakis and Zisman [START_REF] Spanoudakis | Inconsistency management in software engineering: Survey and open research issues[END_REF] discovered that different stakeholders produced inconsistencies in overlapping models in their survey. They presented their findings as parts of a process for identifying and resolving inconsistencies. They also presented processes for identifying overlapping models and, how to diagnose inconsistencies on these overlapping models. Finally, they described the handling, tracking and management of inconsistencies for these overlapping models.

Usman et al. [START_REF] Usman | A survey of consistency checking techniques for uml models[END_REF] presented consistency checking techniques for UML models in their survey. They found that nearly all consistency checking techniques are based on rules and monitor these rules on the UML models. They analyzed each tool with their defined parameters e.g. Nature of consistency checking, UML diagrams, UML version etc. and they classified the techniques according to their intermediate representation.

These studies have shown the potential of consistency checking and fixing mechanisms but they did not show the effect for engineers when correcting inconsistencies between model and code. This is the objective of our empirical study to explore any benefit of such mechanisms. Thus, the nature of our study differs from the previous work described in this section.

Method

This section details the method used in our controlled experiment with student subjects and allows its reproducibility. The subjects had to manually complete change requests between model and code based on consistency rules provided to them. For some of these change requests, we provided inconsistency feedback. For others we did not. In total, subjects had to complete ten such change requests with different degrees of complexity. The task complexity was based on the number of diagram types a change request affected -simple tasks affected one type of diagram, complex tasks affected two or more types of diagrams. The premise was always the same: the subjects were given an already changed model or code together with a change description; and they were asked to propagate the changes to the unchanged code or model. This is where the consistency information came into play. Inconsistency feedback becomes relevant after the model or code changes. The experiment thus explored if the availability of inconsistency feedback made a difference in propagating the initial changes.

We define inconsistencies as situations where engineering artifacts are contradictory. Since, this experiment focused on model and code, inconsistencies imply that model and code are out of synchronization. Our working assumption was that the initial change to the model or code was correct and the inconsistencies was the result of not yet having propagated the change to the code or model respectively.

Inconsistencies are defined through consistency rules -which describe a condition that must hold [START_REF] Usman | A survey of consistency checking techniques for uml models[END_REF]. Since our experiment focused on models and code (specifically UML class, state and sequence diagrams as well as Java source code), we considered consistency rules between model and code applied in this experiment, Table 1 shows these six rules. For example, for a sequence diagram the order of calls among messages must correspond to the sequence of calls in the code.

A change request came in two variations: either the code was changed by us and the subjects were asked to resolve inconsistencies with the model; or the model was changed by us and hence subjects had to resolve inconsistencies with the code. Regardless of task, project or change direction, the consistency rules were always the same.

To control the experiment, a laboratory was used. The allocated time slot for the experiment was three hours. Since the laboratory could not accommodate all subjects at once, we performed the experiment three times in a row with roughly 12 subjects each. The data were analyzed as one set because each setup was identical and this experiment should not be considered a family of experiments [START_REF] Basili | Building knowledge through families of experiments[END_REF] but rather as one experiment.

The three groups' experiment parameters were kept constant. A procedure was set up that followed a predefined schedule. Instructions were given to subjects orally. A demo change request was done together with the subjects to familiarize them to the tasks. An initial session of 20 minutes was reserved for this. Subjects could ask questions. Since, we considered each subject's individual effort and correctness as part of the experimental analysis, subjects were not allowed to interact with one another during the experiment.

Subjects

The subjects performed the experiment as a supplementary part of a software engineering course. We asked them to volunteer for this experiment and their participation was not a mandatory part of their course. We did not exclude any volunteering student from the experiment. We had 36 subjects, all bachelor-level computer science students at our university: 31 were male and five were female. Their mean age was 23.7 years with the youngest subject having been 20 years old and the oldest subject having been 38 years old. Their professional programming experience ranged from 0 years (no experience) to 17 years. The average professional experience for the subjects was two years. As computer science students, all had extensive academic programming experiences; all were familiar with Java and UML. The subjects had an average Java experience of three years and about one year of UML modeling experience. The results from these subjects can be generalized for the target group of junior engineers.

Independent Variables

The experiment aimed to measure the effort and correctness of subjects while they resolved the inconsistencies stemming from the change requests. Every subject received the same change request tasks -half the tasks with inconsistency feedback and the other half without inconsistency feedback. Every subject thus had tasks with and without feedback. Furthermore, every subject had to work on both change directions -half the tasks focusing on model-to-code changes and the other half focusing on code-to-model changes. This setup avoided accidental groups of "good" subjects or "bad" subjects. We prepared ten tasks for two different projects to ensure that our findings were not limited to a specific project or a specific task. Each project had five tasks. Each task involved types of UML diagrams. Moreover, the tasks were of different complexity. Two tasks affected one UML diagram type; two tasks affected two UML diagram types and one task affected all three UML diagram types. Each change request task has a model-to-code and code-to-model direction with or without inconsistency feedback.

Hence, the experiment s independent variables were: the project, the change direction, the (availability of) inconsistency feedback and the complexity of the tasks. The independent variables are described in detail next.

Projects

We used two different projects to ensure that the results are applicable beyond a specific project. The first project was an open source monopoly game called "Matador" and the second project was a proprietary calendar application called "Calendarium". Both projects were implemented in Java and they were selected because of the availability of source code and different UML diagrams types.

The game project Matador is a Monopoly-like game with some alterations from the original Monopoly game. It is quite small with <6KLOC across 37 Java classes. Altogether with six diagrams and 661 model elements were available. The calendar application Calendarium is larger with 21KLOC across 150 Java classes. It has 12 UML diagrams and 2843 model elements. Nevertheless, the size of these project was ideal for a three hours experiment while being diverse and complex enough. Note that the sizes of the projects, as seen in Table 2, vary among tasks because of the changes we made to trigger the experiments. The data set is archived in the FigShare platform1 to be used for reproducibility purposes.

Change Direction

In model driven engineering, programmers work side-by-side with modelers. A modeler could thus initiate a task by changing the model which the programmer then needs to implement in the code. Or a programmer could perform the requested changes first which the modeler will then document in the model. Consequently, inconsistencies may arise from either model changes or code changes in both directions. Regardless, these inconsistencies need to be resolved. Thus, for each of the ten tasks, we created two change directions: a task could either be initiated by model changes that need to be propagated to code; or by code changes that need to be propagated to the model. Each subject was randomly assigned a change direction for each task. Every subject thus had five model to code and five code to model change requests.

Inconsistency Feedback

The experiment examined the impact of having knowledge about inconsistencies between model and code. Thus, another independent variable was the availability of such knowledge. We considered consistency rules that cut between model and code. Consistency rules imposed by the specific languages for their artifacts are not violated by the created tasks and they are respected by the participants. Table 1 presented the rules that must hold between the model and code. Table 3 presents example messages for each consistency rule that subjects had to consider as part of the inconsistency feedback.

Complexity of Task

We choose ten change request tasks with increasing complexity. We define the task complexity to be related to the number of diagrams a task is affecting. Multi diagram tasks are more complex than one diagram tasks since multiple elements must be consistent not only between model and code but also across the affected diagrams. On the code side, multi diagram tasks usually affect multiple classes and methods in them in order to achieve consistency between the model and the code.

Dependent Variables

The experiment measured the performance of subjects resolving inconsistencies when inconsistency feedback was provided in contrast to when it was not provided. To measure the performance, we observe two variables: 1) the time (effort) subjects required to perform each change request task and 2) the correctness of each provided solution. The time variable describes the duration a subject required to complete a task. The time was measured in minutes and it was calculated by subjects recording the start and end times for every task they worked on.

The correctness variable describes the correctness of the given solution with respect to resolved existing inconsistencies. Correctness is a categorical variable with three discrete values, incorrect, partially correct and correct. Each solution required multiple changes to be performed for the solution to be complete. We split the changes into three groups of related changes (e.g, method renaming was one group, class relationships was another group, etc.). Subsequently, we assessed how many of these groups the subjects had completed. Tasks for which subjects did not complete any group were considered incorrect. Tasks for which one or two groups were completed were considered partially correct. Finally, tasks for which all three groups of changes were completed were considered as correct. Correctness was assessed by three people, two authors and an external person.

Experiment Variations

The experiment was designed with two projects in mind that every subject would have worked on, each with five change request tasks. Each task has a direction (model change to code or code change to model). A change direction was either supplemented with inconsistency feedback or it was not. Hence, there were 2x2x5 = 20 possible settings how the experiment could unfold (following the factorial design [START_REF] Box | Statistics for Experimenters[END_REF]). While the tasks were randomly assigned to subjects, we ensured that all possible task variations were included. However, every setting had the tasks of each project being grouped together. Thus, a subject could start with Matador or Calendarium project and had to complete its tasks before the subject could move to the next project.

Experiment Tool

The experiment was conducted using the Eclipse Integrated Development Environment (IDE)2 in the Java development edition. We installed the UML Designer plug-in3 for the UML model of the projects. We did not use any other tools for the experiment since the Eclipsed IDE with the modeling plug-in could serve both developer roles (Modeler and Programmer). The use of a standard IDE was preferred over the development of a specialized tool due to the support for both types of work artifacts for the experiment and the familiarity this tool had among all subjects.

Procedure and Material

Before the start of the experiment, we provided informational material to subjects. The material contained a questionnaire about the demographic information of the subjects. We also provided a demo task to illustrate an example of the problem and the solution (another task which was not used in the experiment but served for training purposes only). Finally, the subjects received the list of ten tasks they had to complete. The experiment did not require the subjects to be aware of the functionality of the projects. We spent the initial 20 minutes of the experiment for a briefing. The briefing included the subjects filling in the required demographic information and then proceeding to explaining the purpose and scope of the experiment. This briefing also included, a presentation about the Eclipse IDE and its modeling perspective for the UML model of the task. During the presentation of Eclipse IDE, we also discussed the demo task to provide a concrete example.

Main Experiment Part Following the briefing, the subjects started the main phase of the experiment. In this phase, each subject had to solve the ten tasks in the assigned order. Initial test runs (conducted with other students beforehand) showed that the tasks were solvable in roughly three hours. Therefore, we considered three hours working time to be sufficient for the experiment. For each task, subjects had to record the start and end times on their own task sheets. The changes had to be implemented using the Eclipse IDE. Both the task sheets and the workspaces of the Eclipse IDEs (this is where the model and code is stored) were collected at the end of the experiment to measure time and correctness.

After the experiment The final step was to check the answers provided by the subjects for correctness and to record the data in an excel sheet for further statistical analysis. In this step, we filled the column with the correctness value after checking one by one the answers of the subjects. Correctness was checked by three people independently. Differences were discussed and resolved to derive a single answer for each task: Correct, partially correct, or incorrect. The recorded time was also stored in the same excel sheet for the analysis phase. Unfinished tasks were rejected from the analysis process since they could not be measured in respect to time and correctness.

Analysis and Results

This section analyses the results we gathered from the performed experiment. The research questions provide the motivation for the following analysis of the results.

Research Questions

In this experiment we are investigating the effect of inconsistency feedback provided to engineers resolving inconsistencies. We will answer the following research questions.

1. What is the effect of inconsistency feedback on correctness when engineers are presented with inconsistent models and code changes?

2. What is the effect of inconsistency feedback on time (effort)?

3. Does the complexity of the tasks affect time and correctness?

Overall Analysis

For the analysis of the collected data, we used the R statistical analysis environment [R D08]. We choose to analyze the dependent variable time with the ANOVA [DCF + 16] method for the analysis of variance. ANOVA is based on the assumptions of normal distribution and homogeneity of variance. To test this, we performed the Shapiro-Wilk's [START_REF] Samuel | An analysis of variance test for normality (complete samples)[END_REF] test for data normality in the continuous depended variable time ( W=0.99408, p-value=0.4749, α = 0.05). The test provides the null hypothesis that the data are normal and the p-value confirmed it. Since, the data was normally distributed, we performed the Bartlett's homogeneity test [START_REF] George | Statistical methods, 8thedn[END_REF](K-squared = 5.3777, p-value = 0.1461) which provided no evidence for lack of the variance homogeneity. We have considered the following formula for the ANOVA analysis for the time depended variable

T ime ∼ F eedback * Change Direction * P roject + T ask Complexity (1)
For the ANOVA analysis, we observe six different columns, the degree of freedom (Df) which is the number of freedom of a variable, the sum of squares (Sum Sq) is the sum of squares of the deviations of values from the expected value, the mean squares (Mean Sq) is the division of Sum Sq by the Df, the F value which is the ration between two mean squares that forms the basis for a hypothesis test, the p-value the probability to obtain the statistic F value if the null hypothesis is true and the Significance(Signif) where it shows the effect of a independent variable to the outcome of a dependent variable.

In Table 4, we see that the inconsistency feedback independent variable has an effect on the time dependent variable in correlation with project and direction independent variables. This means that the time is affected by the provided feedback both when the project changes and when the direction is different. We need to examine in the detailed analysis to find out in what way time is affected. In this ANOVA analysis, we also see a variation in time between the subjects resolving tasks in the different projects. We expected that due to the difference in size and nature of the two projects. We also performed ANOVA on the second depended variable correctness. The formula used for the ANOVA is

Correctness ∼ F eedback * Change Direction * P roject * T ask Complexity (2)
In Table 5, we observe that the variance of correctness is mostly affected by the inconsistency feedback independent variable. This is interesting since multiple other variables are changing but they do not seem to have a large effect in the correctness depended variable. The only ones that have some effect into the dependent variables are the direction and complexity. Complexity was expected to have some effect since our subjects have different level of experience, providing an interest point towards the effect of direction to correctness. After the analysis of the variance for the two dependent variables, we present the overall time and correctness results of the subjects participated in the experiment. In total, we had 360 tasks: 180 with inconsistency feedback and 180 without inconsistency feedback. The subjects completed 141 tasks with inconsistency feedback and 140 without feedback. In order to answer our first research question, we present the analysis in the correctness variable in subsection 4.3 where we examine overall correctness and correctness per independent variable.

Research Question 1

Revisiting our first question, we are interested in identifying the effect of inconsistency feedback for subjects correctly resolving change tasks. Table 6 presents the correctness of the subjects in fixing inconsistencies at the requested tasks. We observe that subjects with inconsistency feedback have solved almost three time more correct tasks than without inconsistency feedback. This overall 268% increase is seen by having 59 correct tasks for subjects with inconsistency feedback and only 22 correct tasks for subjects without inconsistency feedback. The partially correct tasks remained at the same levels for subjects with and without inconsistency feedback.

Table 6 showed overall correctness. In the following, we will further examine the correctness with respect to the different projects and the change request direction. We expect from the variance analysis above that we would not have any significant difference from the overall analysis for the independent variables Projects and Direction seen in Table 5. Table 7 verifies the findings from the ANOVA analysis about the correctness improvement due to inconsistency feedback. We see that in both projects, inconsistency feedback improved task correctness more than 200% with partially correct task remaining in the same levels for both groups. It is interesting to observe that the correctness with inconsistency feedback is better for the larger project "Calendarium" than compared to the smaller project "Matador". This suggests that the importance of inconsistency feedback improves with project size but more data is needed to confirm this.

Finally, as can be seen in Table 8, the different change directions do not affect the correctness of inconsistency feedback regardless of direction. The benefit is respectively 285% and 237% for both model to code inconsistencies and code to model inconsistencies. Thus, we conclude that both designers and programmers can benefit equally from inconsistency feedback in keeping model and code consistent.

Research Question 2

Next, we are discussing the effect of inconsistency feedback on the effort that engineers need in identifying and resolving inconsistencies. We measure the effort with the time subjects needed for completing their tasks.

In Table 9, we see that the mean overall time that subjects required to complete the tasks without inconsistency feedback is in fact smaller than those with inconsistency feedback. This is the case for all above scenarios for tasks (all task, correct tasks, correct and partially correct tasks). In the case of the correct task, we also observe that the medium time is almost identical. We will explore this phenomenon with the time variable further down in subsection 4.5 where we present a more detailed analysis on the effort required by the subjects to complete the tasks with and without inconsistency feedback. Table 10 presents the mean time of the subjects completing task correct or partially correct between the two projects. We observe that in both projects the mean time of the subjects is slightly higher for those having inconsistency feedback than those without inconsistency feedback. We expected a variation between the subjects with and without inconsistency feedback based on the variance analysis of the time dependent variable when correlated to the inconsistency feedback.

According to the ANOVA analysis, we expect a variation for the time measurements between the different directions of the tasks. In Table 11, we observe that the model to code change request tasks have benefited in time from the inconsistency feedback which was expected since a model change usually is required to be implemented with a specific way to the code. Interestingly, the code to model direction is far from being considered benefited by the inconsistency feedback since there is large time difference between subjects with inconsistency feedback and subjects without it.

In order to understand what is happening with the time measurements, we are going to limit the comparison groups to only correct tasks to see if the same results are present.

We again observe in Table 12 that subjects without feedback took less time than the subjects with inconsistency feedback. This is interesting because we expected before the experiment that the subjects with the inconsistency feedback will require less effort to complete the tasks. Thus, we need to examine deeper into the correct tasks to reveal the causes of such diversion from our expectation. From Table 10 and Table 11, we see that the time was not favorably affected by inconsistency feedback. However, not all tasks have the same complexity. Therefore, we investigated more carefully how time and correctness are influenced by the tasks' complexity. The findings are presented into the subsection below where we present the effect of complexity to the effort of subjects completing tasks and what kind of tasks were able to complete with and without inconsistency feedback. Figure 1 presents the number of correct complex tasks based on the different UML diagram types when providing inconsistency feedback or not. There are four types of tasks that affect multiple diagrams. The tasks for all three diagrams and for a class and sequence diagrams are present in both projects. The class*statechart task is present only in the "Callendarium" project while the sequence*statechart is present only in the "Matador" project. This explains their lower number in Figure 1 in contrast to the other two combinations. We choose to have the class*sequence task interleaving between the two projects because there are the most commonly used diagrams in UML modeling. The class*sequence*statechart task is also present in both projects since it is the most complex task. Figure 1 shows that all different complex tasks benefit from inconsistency feedback. In particular, the correct class*sequence tasks and class*sequence*statechart were respectively five and three times higher when inconsistency feedback was provided.

We performed the same analysis for the simple task to see which UML diagram type is more affected by the inconsistency feedback. We have class diagram tasks present only in the "Callendarium" project while we have statechart diagram tasks present only in the "Matador" project. Sequence diagram tasks are present in both projects. Figure 2 shows the number of corrects simple tasks per diagram type. We have twice as many sequence diagram tasks compared to the other two diagram types, because we included this task type in both projects. Another interesting observation is that sequence and statechart diagrams are favored around two times when inconsistency feedback is applied while the class diagram type tasks are improved by three times. This subsequently leads to class diagrams being the most favored among simple tasks when inconsistency feedback is present. Figure 2 shows that all different simple tasks benefit as well from inconsistency feedback.

Threats to Validity

In this section, we discuss the threats to validity ( [FM10], [START_REF] Healy | Comprehensive criteria to judge validity and reliability of qualitative research within the realism paradigm[END_REF]) of this study and explain how we mitigated these dangers in this experiment.

External Validity

A possible external validity threat is the use of students as subjects in the experiment. Recent studies [START_REF] Salman | Are students representatives of professionals in software engineering experiments[END_REF][START_REF] Svahnberg | Using students as subjects-an empirical evaluation[END_REF] have shown that students can be valid subjects for experiments and that students are well representative when it comes to developing tasks. To reduce the effect of this threat, we included students with a variety of professional and academic experience in programming and modeling. We had subjects with minimum knowledge of programming and modeling and subjects which are considered experienced programmers.

From the subjects used in the experiment, one may conclude that our generalization group is junior engineers. This limitation is present in the use of students as subjects but we expect that consistency feedback will be beneficial to senior engineers working on larger projects. Thus, we expect to have similar findings for also more experienced engineers but reproducing the experiment with practitioners is required to make conclusions.

Another possible threat is the chosen projects. We chose to have two projects which were both implemented in Java but we expect that this does not affect the results in any other programming languages since consistency checking does not focuses on programming language specific aspects but rather on the existence of key modeling aspects that should also be found in the code.

Our study was performed on projects that were medium and large for the scope and the three hours time subjects were asked to participate. Both projects had models that consisted of three type of diagrams: class, sequence and statechart diagrams. The projects were applications in different nature, a game and a calendar, thus, we can avoid the possibility of the results to be applicable only to the specific nature of a project. The choice to avoid larger more complex (> 1M LOC) projects was deliberately taken because in complex, real-world systems, engineers often have indepth knowledge of the system they are designing and maintaining. Thus, they are likely to know where to propagate changes when inconsistencies arise. By choosing smaller projects, we aimed at mitigating the subjects' unfamiliarity and difficulty to understand the projects.

However, the fact that we found strong benefit in correctly resolving inconsistencies in projects of this size, implies that more complex projects will be equally (or better) affected by the use of consistency feedback to resolve inconsistencies. We also believe that in more complex projects, we can gain effort as well since the complexity of the inconsistencies will be higher than in the projects used in our experiment.

Finally, we choose to use a real life IDE for the experiment in order to imitate real working environments and condition for the experiment and the subjects.

Internal Validity

To avoid variability in knowledge between the subjects, we made sure not to provide any information about the experiment prior to the date of the experiment. All material was made available to the subjects during the beginning of the experiment and they were kept at a minimum. We asked the subjects after the briefings if they had any questions but apart from some organizational questions, they claimed to be ready for the experiment. The subjects had no prior knowledge of the projects. They could also not know about the tasks which were created by us for the purpose of this experiment.

To avoid the Hawthorne effect, we used subjects from a course that the authors were not tutor or lecturers so that the subjects will not feel examined from their teacher. We also during the briefing and invitation to participation that their performance in the experiment will not be assessed within their course. We confirm them that their performance was only relevant for metrics within our research group and for only research purposes.

The design of the study is crossover study, a variant of within-subject design in which all participants are exposed sequentially to all variances of the tasks. This design minimizes the number of participants required to identify statistically significant differences between the results [START_REF] Jones | Design and analysis of crossover trials[END_REF]. This design has the learning effect as a threat. This threat was mitigated by assigning the tasks to the subject in different order. Thus, one subject could get a task as first whereas another subject would get it as last. All tasks were assigned in all different places.

The time that subjects could use to solve the tasks was limited to three hours in total without limitation per task. The time limit was chosen by preliminary test runs (with different subjects). Most of the 36 subjects have completed all of their task, which suggests that the assigned time was sufficient for the average subject. We also did not allow the students to communicate with each other and we provided different set up for each one of them.

We had created the initial changes and inconsistencies, allowing us to control the complexity of each task based on the affected UML diagram types. To reduce the threat of meaningless change requests we created real life inconsistencies from Table 1 between code and models. The goal was to investigate the effect of consistency feedback on the resolved tasks.

Finally, to avoid that completed tasks were incorrectly categorized, we performed the correctness check independently by three different people. In case of diverse opinion among the three, a discussion session was held for concluding the correctness of the specific task.

Construct Validity

The experiment aimed to evaluate the effect of consistency feedback in resolving inconsistencies between collaborating software modelers and software developers. The problem is caused when the selected dependent variables do not relate to the target of the experiment. Inconsistencies usually require the developer's time to be resolved and not all of them are surely resolved correctly. Thus, we chose to assess the effect by measuring the effort and correctness of the subjects resolving the introduced inconsistencies. We measured the effort with the time subjects required to complete each task. Faster time in resolving inconsistencies implies less effort by the subject whereas correctness expects the subjects to resolve all inconsistencies between model and code based on the defined consistency rules.

Conclusion Validity

Possible threats to the statistical conclusion validity are low statistical power, low effect size and violation of the assumptions for the statistical procedure. To avoid these threats, we worked with 36 subjects, created equally sized groups with randomized task assignment. Before the data analysis, we verified the normality assumption using the Shapiro-Wilk test and we used the Bartlett's homogeneity test to verify homogeneity over the time data. Later, we performed the ANOVA analysis of time and correctness variables. However, for a finer grain analysis, such as for the effort (time) of subjects for complex tasks, we need to reproduce the experiment and gather more data points. This is future work.

Conclusion

In this paper, we investigated the usefulness of inconsistency feedback during the co-change of models and code. We conducted a controlled experiment with 36 subjects that were asked to resolve inconsistencies between model and code. We designed ten change requests tasks of different complexity for two diverse in nature projects. Each task has two directions of co-change one from model to code and one from code to model. Every direction has two variations with consistency feedback provided or not.

For each project, we had a model with three types of UML diagrams (class, sequence, statechart diagrams) and its Java source code.

We found that consistency feedback provides a significant effect in correctly resolving inconsistencies. Subjects with consistency feedback resolved correctly 268% more tasks than those without consistency feedback (59 with consistency feedback versus 22 without). This effect was similarly observed for any direction and any project. Effort was not significantly affected in the overall analysis, leading us to explore deeper the causes for it. Thus, we examined the cause of the time not being affected by consistency feedback focusing on the complexity of the tasks. We found that subject with consistency feedback have resolved correctly three time more complex tasks (31 over 10). Interestingly, with consistency feedback subjects resolved more complex tasks than simple tasks (31 vs 28). Whereas without consistency feedback subjects resolved more simple tasks than complex tasks (12 vs 10). The mean time for simple tasks was significantly lower when consistency feedback was not provided. This resulted in the overall mean time being relatively lower for subjects without consistency feedback. However, the mean time for complex tasks was lower when consistency feedback was provided.

Our findings provide evidence that the consistency feedback, when present, can benefit developers and designers in correctly resolving inconsistencies. In particular when correcting more complex tasks. As future work, we plan to reproduce our experiment to gain more evidence especially to further investigate the effect of consistency feedback on time. At the second experiment, we will add an additional post-study questionnaire to see the level of trust from subjects to the consistency feedback provided.

  Four change requests were designed to affect only one kind of diagram, e.g., class diagram only or statechart diagram only. Four more change requests were designed to affect two types of diagrams, e.g., class and sequence diagrams or sequence and statechart diagrams. The last two change request tasks were designed to affect all types of diagrams, e.g., class, sequence and statechart diagrams. The tasks covered actual change requests that can occur in a project development phase where change can affect multiple type of model diagrams and code places. Every task is designed to be totally independent from the others task of the project.

Figure 1 -

 1 Figure 1 -Correct Complex Task Analysis

Table 1 -

 1 Consistency rules used in the experiment from Reidl et all.[START_REF] Riedl-Ehrenleitner | Towards model-and-code consistency checking[END_REF]

	Diagram Type	Consistency rules
	Class Diagram Consistency Rules	CR1: The structural features of the class diagram to be represented in the code (vice versa) CR2: The code class references and inheritance
		must be present in the class diagram (vice versa)
	Sequence Diagram Consistency Rules	CR3: Order of calls in the Sequence Diagram must correspond to the code (vice versa) CR4: Methods' calls from a code class must
		be represented to the lifeline of that class (vice versa)
	State chart Diagram Consistency Rules	CR5: The transition between states are methods with condition and must be represented in the code (vice versa) CR6: The code execution state space must follow
		the state chart diagram's state sequence (vice versa)

Table 2 -

 2 Summary of Projects in Model and Code Size

	Project	Lines of Code	Number of Classes	Different Diagrams	Diagrams	Model Elements
	Matador	5.674	37	Class, Sequence, Statechart	6	661
	Calendarium	21.016	150	Class, Sequence, Statechart	12	2.843

Table 3 -

 3 Example of inconsistency feedback per rule provided to the subjects

	Inconsistency Example message
	Violation	Methods getTransaction() and getConnections are missing
	of CR1	in the class ConnectionManager in "dblayer class diagram"
	Violation	Missing relationship to Interface Controller from classes
	of CR2	Game Controller, Board Controller, Player Controller ...
	Violation of CR3	Messages lock(), commit() and processSql() in "DBLayer sequence diagram" are called in the wrong order as it is the code in the method deleteEntityObject()
	Violation	Messages lock(), commit() and processSql() in "DBLayer
	of CR4	sequence diagram" are called from the wrong lifeline
	Violation	Transitions in the "PersistentBroker State Machine diagram"
	of CR5	are not declared as methods in the class PersistenceBroker
	Violation	State generated by the payDept() not present
	of CR6	in the statechart diagram "Matador_Statecdiagram"

Table 4 -

 4 ANOVA analysis of the effort depended variable Time

	Variables	Df	Sum Sq Mean Sq F value Pr (>F) Signif.
	Feedback	1	161.6	161.62	4.8151	0.029199	*
	Direction	1	1	0.8	0.024	0.87679
	Project	1	174.1	174.09	5.1864	0.023671	*
	Complexity	9	169	18.7	0.581	81181
	Feedback:Direction	1	329.6	329.61	9.8198	0.001948	**
	Feedback:Project	1	293	293.1	9.093	0.0029	**
	Residuals	195	6127.3	31.422

Table 5 -

 5 ANOVA analysis of the Correctness dependent variable

			Df	Sum Sq Mean Sq F value Pr (>F) Signif.
	Feedback		1	13.69	13.695	25.137	1.19E-06	***
	Direction		1	3.59	3.587	6.584	0.011	*
	Project		1	0.89	0.892	1.638	0.2021
	Complexity		9	11.97	1.33	2.441	0.0118	*
	Feedback:Direction		1	0.01	0.01	0.019	0.8912
	Feedback:Project		1	0	0.003	0.006	0.938
	Direction:Project		1	0.25	0.246	0.452	0.5024
	Feedback:Complexity		8	4.23	0.528	0.97		0.461
	Direction:Complexity		8	8.72	1.089	2		0.0483	*
	Feedback:Direction:Project		1	0.64	641	1.176	0.2795
	Feedback:Direction:Complexity	8	1.39	0.174	1.176	0.9579
	Residuals		197	107.33	0.545
		Table 6 -Overall Correctness
	Correctness	Inconsistency Feedback	No Inconsistency Feedback	Incr
	Correct		59(41.84%)	22(15.71%)		268%
	Partially Correct		49(34.75%)	50(35.71%)		0%
	Incorrect		33(23.4%)	68(48.57%)	-51,5%

Table 7 -

 7 Correctness per Project

	Matador	Inconsistency Feedback	No Inconsistency Feedback	Incr
	Correct	34(44.74%)	15(20%)	226,7%
	Partially Correct	18(23.68%)	24(32%)	-25%
	Incorrect	24(31.58%)	36(48%)	-33,4%
	Callendarium	Inconsistency Feedback	No Inconsistency Feedback	Incr
	Correct	24(36.92%)	7(10.77%)	342%
	Partially Correct	26(40%)	26(40%)	0%
	Incorrect	15(23.08%)	32(49.23%)	-46,9%

Table 8 -

 8 Correctness per Direction

	Model to Code	Inconsistency Feedback	No Inconsistency Feedback	Incr
	Correct	40(47.06%)	14(23.33%)	285,7%
	Partially Correct		23(38.33%)	130%
	Incorrect	15(17.65%)	23(38.33%)	-65%
	Code to Model	Inconsistency Feedback	No Inconsistency Feedback	Incr
	Correct	19(33.93%)	8(10.00%)	237,5%
	Partially Correct	19(33.93%)	27(33.75%)	-70,4%
	Incorrect	18(32.14%)	45(56.25%)	-40%

Table 9 -

 9 Overall Time Measurements

	Mean Time (min)	Inconsistency Feedback	No Inconsistency Feedback
	All Tasks	21.86	19.11
	Correct Tasks	22.90	22.64
	Correct & Partial Correct	22.94	20.46

Table 10 -

 10 Mean Time per Project for Correct and Partial Correct

	Mean Time(min)	Inconsistency Feedback	No Inconsistency Feedback	Diff
	Matador	24.34	20.87	3.47
	Callendarium	21.9	19.97	1.93

Table 13 -

 13 Correctness for the different Complexities of Tasks

	Correct Task	Inconsistency	No Inconsistency
	Analysis	Feedback	Feedback
	Class	9	3
	Sequence	12	6
	State	7	3
	Total	28	12
	Class*Sequence	10	2
	Class*State	3	2
	Sequence*State	5	2
	Class*Sequence*State	13	4
	Total	31	10

Table 14 -

 14 Mean Time for the different Complexities of Tasks

	Correct Tasks Mean Time	Inconsistency Feedback	No Inconsistency Feedback	Diff
	Class	23.1	18	-5.1
	Sequence	17.5	12.6	-4.9
	State	25.1	28.33	3.23
	Total Mean Time	23.71	13.17
	Class*Sequence	22.3	68	45.7
	Class*State	23	23.5	0.5
	Sequence*State	23.2	15.5	-7.7
	Class*Sequence*State	21.3	20	-1.3
	Total Mean Time	22.1	28.30

Journal of Object Technology, vol. 18, no. 2, 2019

https://figshare.com/s/cb1a69751b68de5cc6a7 Journal of Object Technology, vol. 18, no.

2, 2019

Eclipse IDE website: https://eclipse.org/

Plug-ins website https://marketplace.eclipse.org/content/uml-designer Journal of Object Technology, vol. 18, no. 2, 2019

Acknowledgments

The research leading to these results was funded by the Land of Upper Austria -Joint International PhD Program in Informatics under the grant number Wi-2014-202477/6.

Funded by the JKU Linz Institute of Technology (LIT) the state of Upper Austria, grant no. LIT-2016-2-SEE-019 and Pro2Future, a COMET K1-Center of the Austrian Research Promotion Agency (FFG), grant no. 854184. Received funding from Rennes Métropole under grant AIS no. 190270.

Research Question 3

Initially, we examined the effort subjects used based on the complexity of the tasks in order to investigate their effect on the mean time for the subjects with inconsistency feedback. In our experiment, we included tasks that are of different complexity. For each project, we have two tasks that are simple and affect only one kind of diagrams, two that affect two kinds of diagrams and one task that affects all three kinds of diagrams in the model. We grouped these tasks into simple tasks (i.e. one diagram type affected) and complex tasks (i.e. two or three diagram types affected).

We also observed in Table 9 that the time subjects took to complete tasks are relative better for those without feedback. However, this does not unveil the effect of the tasks' complexity on the overall mean time for correct tasks.

In Table 13, we present the correct tasks divided by their different complexity with and without inconsistency feedback. We see that the number of complex tasks being resolved correctly are higher (31) with consistency feedback than without inconsistency feedback (10). We also observe that when feedback was provided, the number of complex correct tasks is higher than the simple ones (31 vs 28). Whereas without inconsistency feedback the number of complex tasks is lower than the simple tasks (10 vs 12). Table 14 similarly shows that the mean time for tasks divided by their different complexity with and without inconsistency feedback. It shows that for simple tasks when inconsistency feedback is not provided, their mean time is significantly lower than when inconsistency feedback is provided. This explains the overall mean time being relatively better for the subjects without inconsistency feedback since the simple tasks were the most resolved tasks in that case. Herein, the overall mean time without inconsistency feedback is mainly affected by the time subjects required for simple tasks.

Furthermore, in Table 14 we see that the mean time for correct complex tasks is lower for subjects with inconsistency feedback. This suggests that subjects benefited by inconsistency feedback when resolving complex tasks. However, when inconsistency feedback was not provided, the number of correct complex tasks per affected types of diagrams is statistically insufficient (e.g., two tasks for class*sequence) to draw conclusions.

About the authors

Georgios Kanakis is PhD student at the Johannes Kepler University, Institute of Software Systems Engineering. His research topic is Cloud Computer Collaboration and Change Propagation. Research interests also include software engineering, Model-Driven Engineering. Employed currently at the MIC custom solution software company in Linz, Austria. Contact him at georgios.kanakis@mic-cust.com Djamel Eddine Khelladi is a CNRS researcher in the IRISA reasearch lab in the DI-VERSE team, Université Rennes 1. Before that he was a Postdoctoral researcher in the Institute for Software Systems Engineering (ISSE) at the Johannes Kepler University Linz. He hold a PhD in the Laboratoire d'Infromatique de Paris 6 (LIP6) at the university of Piere et Marie Curie (UPMC). Research interests include Software engineering, Model-Driven Engineering, DSL Evolution, Evolution impacts, Software Processes, Process Adaptation, Formal Process Verification, and Agile Methods. Contact him at djamel-eddine.khelladi@irisa.fr,djamel_eddine.khelladi@jku.at Stefan Fischer Dr. Stefan Fischer is a post-doctoral researcher at the Johannes Kepler University Linz. He received his Ph.D. in Computer Science from the Johannes Kepler University Linz. His main research interests are test automation and software analysis in variability intensive software systems. Contact him at stefan.fischer@jku.at Michael Tröls Michael Tröls earned his master's degree in Computer Science at the Johannes Kepler University in Linz, Austria. He is currently working there as a PhD student at the Institute for Software Systems Engineering (ISSE). The focus of his work is on consistency checking within collaborative engineering environments. Contact him at michael.troels@jku.at Alexander Egyed Alexander Egyed heads the Institute for Software Systems Engineering (ISSE) at the Johannes Kepler University, Austria. He received his Doctorate from the University of Southern California, USA and previously worked many years in industry before joining academia. Dr. Egyed was recognized among the Top 10 scholars in software engineering and his work has received numerous awards. Contact him at alexander.egyed@jku.at


F Khattar

F Dornaika

B Larroque

F Luthon

3D Object-Camera and 3D Face-Camera Pose Estimation For Quadcopter Control : Application To Remote Labs

We present the implementation of two visual pose estimation algorithms (object-camera and face-camera) with a control system for a low cost quadcopter for an application in a remote electronic laboratory. The objective is threefold: (i) to allow the drone to inspect instruments in the remote lab, (ii) to localize a teacher and center his face in the image for student-teacher remote communication, (iii) and to return back home and land on a platform for automatic recharge of the batteries. The object-camera localization system is composed of two complementary visual approaches: (i) a visual SLAM (Simultaneous Localization And Mapping) system, and (ii) a homography-based localization system. We extend the application scenarios of the SLAM system by allowing close range inspection of a planar instrument and autonomous landing. The face-camera localization system is based on 3D modeling of the face, and a state of the art 2D facial point detector. Experiments conducted in a remote laboratory workspace are presented. They prove the robustness of the proposed object-camera visual pose system compared to the SLAM system, the performance of the face-camera visual servoing and pose estimation system in terms of real-time, robustness and accuracy.

Introduction

Remote labs constitute an interesting and novel way of doing labs. Anywhere and at anytime the student can access the lab equipment and do his labwork. This new way of distance learning can be used to increase the motivation of nowadays students [START_REF] Luthon | Use of gaming and computer vision to drive student motivation in remote learning lab activities[END_REF]. Quadcopters equipped with a camera can be used in these laboratories in order to mimic the student behavior in traditional lab and increase motivation for learning. It can be an interesting way to make the lab experience immersive and ludic. Specifically, in remote electronics laboratories it can fly and move in 3D space to inspect electrical instruments, consequently sending direct visual feedback of the results of an experiment to the student. Furthermore it can also search for a teacher in the lab and move towards him, centering his face in the image to allow remote student-teacher interaction. In this way the student can achieve his lab-work and can also ask questions to the teacher in case he needs to. To achieve this double objective two systems are needed. First, a localization system that can estimate the position and orientation of the quadcopter in 3D space with respect to an object of interest (front panel of an electrical instrument or face of the teacher in our case). Second, a control system that sends appropriate commands to the quadcopter in order to reach a reference relative or absolute 3D position. Many localization systems and sensors can be used in order to estimate quadcopter position and orientation in 3D space. For outdoor environments, GPS sensors are the best solution to localize a quadcopter. For indoor environments, artificial markers can be placed in the scene to facilitate the task of localization [START_REF] Eberli | Vision based position control for MAVs using one single circular landmark[END_REF]. These markers can also be reflective and detected by an external localization system that gives accurate position estimate. However, the challenge in these applications is to use only available on-board sensors. Stereo rig cameras [START_REF] Schauwecker | On-board dual-stereo-vision for the navigation of an autonomous MAV[END_REF] and RGBD cameras [START_REF] Flores | A vision and GPS-based real-time trajectory planning for a MAV in unknown and low-sunlight environments[END_REF] have been investigated. They allow for absolute pose estimate however this comes with an additional weight and power consumption. In this work we use the Parrot AR Drone 2.0 [5], a low cost quadcopter equipped with two monocular cameras facing forwards and downwards, in addition to pressure, ultrasound and inertial sensors. Using the monocular cameras available on-board constitutes a good trade-off between weight and information recovery from the environment (3D localization, environment recognition, etc.). However, a monocular camera alone cannot give absolute scale pose estimate due to the well known scale ambiguity rising from the perspective projection of 3D world into 2D images. Despite this fact, combining the visual information with some prior knowledge of 3D world or other sensors, that can give partial but absolute pose estimate, can overcome this issue and allow for absolute 3D pose estimate. The well known SLAM algorithm PTAM (Parallel tracking and mapping) [START_REF] Klein | Parallel tracking and mapping for small AR workspaces[END_REF] combined with inertial and ultrasound sensors readings in order to get the absolute 3D pose estimate is used in [START_REF] Engel | Scale-aware navigation of a lowcost quadrocopter with a monocular camera[END_REF]. All the available data is processed in a Kalman filter allowing for information fusion and delay compensation. Here, starting from [START_REF] Engel | Scale-aware navigation of a lowcost quadrocopter with a monocular camera[END_REF] we propose the object-camera pose system and use it for 3D pose estimation when the quadcopter is exploring the 3D world to search for an instrument or for a teacher. However, since the visual SLAM relies on corresponding points detected in the flow of images, it will fail to give 3D pose estimate if the quadcopter is asked to inspect an object of interest, since these points will disappear when the object of interest occupies the majority of the image. To overcome this limitation, this system is extended by using the object of interest as a landmark. In this way two localization modules are available: a visual SLAM module (for localization w.r.t. an arbitrary world coordinate system) and a localization module that relies on detecting and localizing a planar object with respect to the quadcopter and to the arbitrary world coordinate system (needed for controlling the remote lab activities and sending visual feedback to the student). The first module is suitable while exploring the environment whereas the latter is suitable when the drone is in a short distance range from the object of interest or when it needs to land on the electrical recharge platform. To be able to control the drone to center the face of a teacher in the image, the position of the drone with respect to the face of the teacher must be estimated. For this purpose, we use a deformable 3D model of the face and fit it to the image. The paper is organised as follows: In the second section we present an overview of the remote electronic lab LaboREM. In section 3 we explain how detection and 3D localization can be carried out for the purpose of remote instrument inspection. In section 4 we present the face-camera pose estimation approach. In section 5, the 3D pose visual servoing of the quadcopter is explained. In the final section, we present our experiments and results.

Review of the remote electronics lab

LaboREM is a remote laboratory in electronics developed for first year undergraduate students in engineering. The learning objective of LaboREM is to enable students to wire and test remotely electronic circuits, make measurements and characterize each circuit by its time or frequency response. The electronic circuits consist of operational amplifiers, active filters and oscillators. Its design is based on a classic client-server architecture [START_REF] Luthon | LaboREM a remote laboratory for game-like training in electronics[END_REF]. The student calls for a lab session by simple URL addressing. A first-in first-out strategy is adopted to give access to the remote lab to one client (student) at a time.The remote lab application is developed using NI-LabVIEW software and the easy-to-use RFP protocol to pilot the remote devices. The hardware setup includes: (i) a robotic arm that mimics the student's hand for placing electronic components equipped with magnets on an electronic breadboard, (ii) measurement instruments and data acquisition system (DAQ), (iii) a webcam with zoom control that mimics the student's eye in order that the student doesn't feel so far away from what is actually happening in the lab, (iv) a quadcopter (AR quadcopter 2.0) with the role of flying in the lab for exploring the environment, inspecting electrical instruments and interacting with a teacher in order to increase student immersion and motivation.

3 Object-Camera pose estimation 3.1 Quadcopter-Object relative 3D pose : A real-time and marker free solution

Planar objects are a well defined type of objects that are widely available in human made environments. Incorporating the information that the object of interest is planar is of great benefit for object-camera pose estimation. The homography matrix is a matrix that relates 3D points lying on a plane to their 2D projections. Given this transform one can directly calculate the rotation and translation matrix as done in [START_REF] Medioni | Emerging topics in computer vision[END_REF]. In order to estimate the homography that maps any plane into another plane by means of perspective projections several methods can be used. These methods are usually classified into local (feature-based) and global (featureless) methods. Given a template image of the planar object, local methods extract local keypoints and attribute a descriptor to each of them both in the template image and the current image. After this step, keypoints (at least four keypoints) in both images are matched according to a similarity metric performed on the descriptors. Given the point correspondences, the homography matrix is estimated using robust methods like RANSAC (Random Sample Consensus) in order to deal with the presence of outlier correspondences. Local methods can work well with no prior information on the homography parameters. However, in some cases the robust computation may be computationally expensive and do not work in real time. A survey about keypoint detectors and descriptors can be found in [START_REF] Krig | Interest point detector and feature descriptor survey[END_REF]. On the other hand, the global methods use all the information in the image and attempt to find the homography matrix that best aligns the template patch to the test image. This process however gives rise to NLM (non linear minimization) problems that can be solved using iterative algorithms like gradient descent or LM (Levenberg-Marquardt). Thus, a good initialization is necessary to guarantee the convergence of those algorithms. Different similarity functions exist to measure the degree to which two patches are aligned, the most used ones being the sum of squared distance (SSD) and the enhanced correlation coefficient (ECC) [START_REF] Georgios | Parametric image alignment using enhanced correlation coefficient maximization[END_REF]. In practice the first one uses a brightness model in order to cope with variation of additive and multiplicative change of illumination [START_REF] Dornaika | Registering conventional images with low resolution panoramic images[END_REF] whereas the latter is by definition insensitive to those illumination changes. These methods have the advantage that they can run in real time and give good results if a coarse estimate of the homography parameters is known. Thus the two families of methods are complementary. The first one is robust with no priors needed but computationally expensive, while the second is fast and works well if a prior is available. Here, both approaches are used in order to estimate the 3D pose of the quadcopter with respect to the object of interest. The first approach is used for detecting the object of interest as well as for recovering from a tracking loss. The second is used in the tracking process. The approach is divided into two steps: detection and tracking [START_REF] Fawzi | Quadcopter control using onboard monocular camera for enriching remote laboratory facilities[END_REF].

In the detection step, template matching on a pyramid of the image is used to search for the desired object. If the normalized correlation coefficient is greater than α the object is declared detected. Once the detection is done, a homography transformation is computed by using the bounding box of the detected object to determine the object-camera relative pose. A command is then sent to the drone in order to move it closer to the object. Template matching is used to allow successful detection of the object despite its distance from the camera and its size, as keypoints detector fails to detect and put in correspondences keypoints if the object of interest doesn't occupy a certain amount of image pixels. However once the distance between the camera of the drone and the instrument is less than a threshold λ, the SIFT descriptor [START_REF] David G Lowe | Object recognition from local scale-invariant features[END_REF] is used to allow more robustness to orientation changes. The object is declared detected if the number of matched keypoints is greater than N . Once the object is detected, the tracking stage begins. As a rough estimation of the homography matrix is available from the detection stage, it is used as an initial solution for the next frame and the ECC algorithm is applied to estimate the homography in this frame. The homography estimation is propagated in this way from a frame to the next one, and used as a prior for the ECC algorithm. However, sometimes the ECC algorithm will fail to converge due to several reasons. For example, communication problems between the quadcopter and the computer makes the last estimated homography not close enough to the real solution of the current frame, which prevents algorithm convergence. Besides, the image quality can be degraded by motion blur or decoding/encoding problems. In this work, tracking loss is declared if the ECC algorithm is unable to converge or if it converges to a clearly unrealistic estimation. At each frame, we compute the 3D pose of the quadcopter with respect to the planar object. By monitoring the estimated traveled distance between two consecutive frames and comparing it to a threshold D, we can detect a loss of tracking. Another threshold β is also imposed on the difference of each angle of orientation (yaw, roll, pitch). If the tracking fails, we resort back to the local method (SIFT) if the quadcopter-object distance is relatively small, or to the template matching method in the other case, to reinitialize the ECC tracker as shown in Figure 1. This pose estimate is fused with inertial measurements sent by the drone in a Kalman filter framework in order to smooth this estimate, and to provide robustness when the visual tracker fails.

The Kalman filter is used also to compensate for time delays as done in [START_REF] Engel | Scale-aware navigation of a lowcost quadrocopter with a monocular camera[END_REF]. The homography estimation process is shown in Figure 1 while the feedback control loop is shown in Figure 2.

Face-Camera pose estimation

Human-drone interaction is an interesting way of controlling drones. In [START_REF] Nagi | Human control of uavs using face pose estimates and hand gestures[END_REF] authors use face pose and hand gestures in order to allow human-drone interaction. Their face pose estimation process is based on the Viola & Jones face detector [START_REF] Viola | Robust real-time face detection[END_REF]. They compute a face score vector by applying frontal and side face detector on the flipped and the original image. Using this face score and a machine learning technique they estimate the yaw angle of the face pose. The distance from the face is estimated by the size of the face bounding Hand gestures are used to give order to the drone to move to an orientation while maintaining the distance from the face. In [START_REF] Monajjemi | Uav, come to me: End-to-end, multi-scale situated hri with an uninstrumented human and a distant uav[END_REF], authors also used hand gestures and face localization for drone-human interaction. Their approach is unique for the fact that it allows the drone to approach a human that is 20 meters away, by detecting periodic hand gestures. The drone then approach the target by tracking its appearance. Once at a short distance, the drone centers the face of the subject and detects hand gestures in order to take a picture. However, the orientation of the face is not estimated and the user has to be facing the camera in order to take a frontal photo. In this work, we adopt a 3D approach that models the human face in 3D and subsequently uses full perspective projection in order to recover the 3D face pose parameters. By using this modeling and matching it with image specific data related to the face, all the 6 pose parameters are inferred. The 3D modeling is based on the CANDIDE deformable 3D face model.

CANDIDE 3D model

CANDIDE is a parameterized 3D face model specifically developed for modelbased coding of human faces. CANDIDE is controlled by 3 sets of parameters: global, shape and animation parameters. The global parameters correspond to the pose of the face with respect to the camera. There exist 6 global parameters: 3 Euler angles for the rotation and 3 for the translation (t x , t y , t z ). The shape parameters adjust facial features position in order to fit to different subjects (eye width, distance between the eyes, face height etc). The animation parameters adjust facial features position in order to display facial expressions and animations (smile, lowering of eyebrows). The 3D generic model is given by the 3D coordinates of its vertices P i , i = 1, n. where n is the number of vertices. This way, the shape, up to a global scale, can be fully described by a 3n-vector g, the concatenation of the 3D coordinates of all vertices:

g = G + Sτ s + Aτ a (1)
G is the standard shape of the model, the columns of S and A are the shape and animation units, τ s ∈ R m and τ a ∈ R k , are the shape and animation control vectors, respectively.

Inferring pose parameters

In order to determine the pose from the 3D model, we have to fit this model to the face data available in the image. Fitting the model means determining its different parameters: pose, shape and animation parameters. In this work only pose and shape parameters are of interest for us, however recovering the animation parameters can be an interesting way to allow human-drone interaction based on facial expressions. Different approaches attempt to adapt the model in different ways. However, the majority of them follow a step by step approach, starting by estimating the shape parameters τ s in order to adapt the 3D model to different face anatomy and then estimating the pose and animation parameters.

From the face image, many face related data can be used to fit the 3D model. In [START_REF] Dornaika | On appearance based face and facial action tracking[END_REF], the authors use the gray scale appearance of the image to adapt the 3D model after estimating its shape parameter off-line. In our work, we make use of the advancement in facial landmark detection and use these landmarks to adapt the model and recover the 3D face pose from a set of 3D-to-2D correspondences.

The shape parameters are estimated using a frontal picture of the subject following the method described in [START_REF] Unzueta | Efficient deformable 3d face model fitting to monocular images[END_REF]. We use the facial point detector in [START_REF] Kazemi | One millisecond face alignment with an ensemble of regression trees[END_REF], that can detect 68 2D landmarks on a face in one millisecond by a pre-trained ERT (Ensemble of Regression Trees), given that a face image patch is available. However, since the algorithm needs a region of interest that contains a face, the total time for its execution from the detection of the face to the detection of the landmarks is more than one millisecond due to the computationally expensive face detection step. One way to reduce this time and make the process working at more than 30 fps (frame per second) is to perform a search for the face around the last detected bounding box of the face instead of looking for the face in the whole image. We make use of only 46 points from the 68 points given by the landmark detector. The points were chosen to be semantic and mostly rigid thus eliminating points along face contour. Once the 2D landmarks are detected in the image we use state of the art pose estimation algorithms that are based on 3D-2D point correspondences to recover the pose. This problem is known in the literature as PnP (Perspective n Point). Many algorithms attempt to solve this problem. The P3P algorithm [START_REF] Gao | Complete solution classification for the perspective-three-point problem[END_REF] (perspective 3 point) can estimate the pose using only 3 point correspondences. Other algorithms like EPNP [START_REF] Lepetit | EPNP: An accurate o (n) solution to the pnp problem[END_REF] (Efficient Perspective N Point) can handle any number of points. Another approach is to use non-linear minimization techniques to recover the pose that best minimizes the distance between the projected 3D points and the 2D points. However, this method requires an initial guess of the pose parameters in order to converge to the global minimum. This initial guess can be made available using the estimated pose from the previous frame or using any closed-form solution like EPNP, P3P, etc. in case it is not available. We propose to use the Levenberg-Marquardt technique as it gives good results and fast execution time.

The face-camera pose estimation process is shown in Figure 3. The pose used to control the drone is computed by fusing the visual pose from the 3D model with inertial and ultrasound measurements in a Kalman filter as done in [START_REF] Engel | Scale-aware navigation of a lowcost quadrocopter with a monocular camera[END_REF].

Visual control of the quadcopter

In order to control the 3D position and orientation of the quadcopter for the purpose of instrument inspection, a closed-loop control is used taking as a feedback the pose derived from the SLAM or the homography. The control loop is shown in Figure 2. If the objective is to control the drone to maintain a relative position from the face, the algorithm explained in section 4 is used as a feedback sensor for the control loop. The controlled degrees of freedom associated with the quadcopter are the 3D translation vector and the yaw angle. Each degree is controlled by a closed loop control system with a traditional PID controller.

Experiments

Before presenting the different scenarios for evaluation, we begin with an evaluation of the performance of the Face-Camera pose estimation. We compare the accuracy of different techniques and their execution time on a database for pose estimation (UPNA head pose database) [START_REF] Ariz | A novel 2d/3d database with automatic face annotation for head tracking and pose estimation[END_REF]. The UPNA database contains 120 videos corresponding to 10 different subjects, 12 videos each, in which the subject changes its head pose by following guided and free movement. The ground-truth relative 3D face motion is known for all frames in all videos. We conclude that all the techniques converge to the same optimal solution if followed by a non-linear minimization method. As shown in Table 1 the method that yields the best pose estimate and excecution time is the Levenberg-Marquardt method that tracks the 3D face from a frame to the next one based on an initial estimate. In order to evaluate the proposed implementation of 3D pose estimation and 3D pose-based servoing, we design three different scenarios. These scenarios are the following: behavior of the system in response to perturbations when asked to inspect an object, autonomous visual inspection of planar object, and drone-face visual servoing.

First scenario: The first experiment aims to test the quality of the homography based visual feedback control system. To this end, we control the quadcopter in such a way that the reference 3D pose of its on-board camera is fronto-parallel to the planar object with a translation vector allowing a centered view. The pose used for the visual feedback control is the homography based pose. Since the servoing objective is to maintain a rigid link between the quadcopter and the object of interest, any motion induced to the object will force the quadcopter to compensate for it. We can induce such motion by a walking person that carries the object or by giving manual kicks to the quadcopter. The quadcopter then follows the object, centering it in the image. Figure 4 shows the response of the system facing manual perturbations applied to the drone (5 perturbations to the x position, 2 for y, 2 for z and 3 for the angle yaw). The objective is to see the behavior of the system when the drone is pushed away from the reference pose causing the visual tracking to fail. Despite the loss of tracking (red curves in Figure 4) due to the fast kicks applied to the drone, it is always able to return to a position that allows the tracking to restart. This is done by controlling the drone based on the pose estimation procured by the Kalman filter [START_REF] Engel | Scale-aware navigation of a lowcost quadrocopter with a monocular camera[END_REF] that fuses the inertial and ultrasound measurements to have an estimate of the current position of the drone. Two videos of this scenario are available online [24], [25].

Second scenario: In remote lab context, an interesting scenario is the following: the remote student will send a command to the quadcopter to go and inspect an electrical device. After receiving this command, the server tells the quadcopter to carry out the following tasks: it should first take off, initialize the SLAM algorithm (by following a vertical path in order to change the height and correctly estimate the scale of the SLAM map [START_REF] Engel | Scale-aware navigation of a lowcost quadrocopter with a monocular camera[END_REF]), and initiate a search procedure for the required instrument. After object detection, the drone moves towards the instrument and the feedback loop uses the 3D pose based on the homography for control. In this way the quadcopter is able to fly to inspect an electrical instrument maintaining its position with respect to the instrument. After the mission is over the drone is sent back home by using the pose derived from the SLAM. The position of the landing platform is estimated by using the homography-based pose estimation applied on the video of the bottom camera of the AR Drone 2.0. The drone hover then at a certain altitude above the landing platform preparing for landing. A quantitative comparison between the approach proposed here for pose estimation when inspecting an object at close range with the SLAM algorithm used in [START_REF] Engel | Scale-aware navigation of a lowcost quadrocopter with a monocular camera[END_REF] is shown in Figure 5. It shows the superiority of our approach and the drift of the SLAM approach due to the reasons explained in the introduction. Figure 6 shows the first detection and the tracking of the object of interest and its robustness in spite of problems of tracking failure. The green cross represents the center of the image while the red cross represents the projection of the center of the object on the image. The objective is to control the position of the drone so that the two crosses are as close as possible. Videos of this scenario are available in [26], [27].

Third scenario: In this scenario we test the performance of the face-camera visual servoing system explained in section 4. The drone has to fly, detect a face, align its line of sight with that of the subject face, and centering the face in the image while maintaining a fixed distance with it. In this experiment the user is in motion in order to induce perturbation to the control system. The drone has to correct for the user displacement and the out-of-plane orientation of his face. Figure 7 shows the experiment seen from the camera of the drone and from an external camera. A video of the experiment is available on [28].

Conclusion

This paper presents the implementation of a visual servoing system of a quadcopter in a remote lab environment to increase student immersion in the lab and hence his motivation. The objective is to allow remote instrument inspection and remote human-teacher communication. The proposed localization system for the first objective is proven to outperforms the SLAM system in [START_REF] Engel | Scale-aware navigation of a lowcost quadrocopter with a monocular camera[END_REF] through qualitative and quantitative experiments, allowing the quadcopter to inspect an object and return to its base autonomously. The approach uses only the on-board sensors available on the low cost drone. The localization system for face-camera servoing is based on 3D modelling of the face and a state-of-the art 2D facial point detector. The approach controls all 4 degrees of freedom (3 degrees for translation as well as the orientation of the face). It is shown robust, accurate and working at frame rate through qualitative and quantitative experiments.

Fig. 1 . 2 .

 12 Fig. 1. Homography estimation diagram. Fig. 2. Feedback control loop.

Fig. 3 .

 3 Fig. 3. Face-camera pose estimation.

Fig. 4 .Fig. 5 .

 45 Fig. 4. Quadcopter control and estimated pose facing perturbations. For each control loop: In green the estimation from the homography when the tracking is good, in red estimation based solely on navigation data when the tracking fails and in light blue the reference 3D pose. Vertical arrows indicate the time each perturbation was applied.

Fig. 6 .

 6 Fig. 6. Detection and tracking of the instrument.

Fig. 7 .

 7 Fig. 7. Third scenario experiment: Face tracking.

Table 1 .

 1 Average pose errors and computation time for different face pose estimation methods. tx, ty, tz are in millimeters, roll, yaw, pitch in degrees, time in milliseconds.

	Method	tx	ty	tz	roll yaw pitch time
	EPNP	11,84 7,11 12,67 0,55 3,74 2,39 0.117
	Ransac P3P	12,50 7,79 18,34 1,56 6,52 6,11 0.898
	EPNP + NLM	11,51 7,23 13,38 0,56 2,28 1,45 0.363
	P3P Ransac + NLM 11,51 7,23 13,38 0,56 2,28 1,45 1.138
	NLM	11,51 7,23 13,38 0,56 2,28 1,45 0.234


Ikram Bouras

Rosa Figueiredo

Michael Poss

Fen Zhou

Exact Algorithms for Fixed Charge Network Design Problem with User Optimal Flow

Keywords: Network design, Bi-level programming, Cutting plane, Branch-and-Cut

Introduction

The Fixed Charge Network Design Problem (FCNDP) consists in selecting a subset of edges from a given network, in such a way that a set of commodities can be transported from its origins to its destinations. The objective is to minimize the sum of fixed costs (depending on the selected edges) and variable costs (depending on the flow of commodities on the edges). There are several variants of FCNDP in the literature, each of which considers a particular objective function or takes into account specific additional constraints.

Herein we are interested in a specific variant of the FCNDP, namely Fixed Charge Network Design Problem with User Optimal Flow (FCNDP-UOF). This problem combines the FCNDP with multiple shortest path problems.

Problem description

We consider a transport network modeled by an undirected graph G(V, E), where V represents the set of facilities and E represents the connections between them, which are uncapacitated. There is a set of K commodities to be delivered in the transport network G(V, E), where each edge e = (i, j) is associated with several parameters : a length c ij , a fixed cost f e and a variable cost g k ij for each commodity k. Our objective is to design a subnetwork with minimum total cost to transport K commodities such that each commodity k ∈ K with a flow φ k can be delivered through the shortest path from its origin o(k) to its destination d(k) in this subnetwork. The FCNDP-UOF has been formulated as a bi-level integer programming problem [2], and as an ILP formulations through replacing the second level by its optimality conditions in [2] and [3]. A cutting plane algorithm based on a set of valid inequalities is proposed in [1].

Proposed methods

We propose in this work a new Binary Integer Programming model (BIP) for the FCNDP-UOF. To this end, we replace the objective function of the second level by a valid inequality (1c) eliminating infeasible solutions. Our BIP formulation is given as follows :

min e∈E f e y e + k∈K (i,j)∈E φ k g k ij x k ij (1a) s.t. j∈δ + (i) x k ij - j∈δ -(i) x k ji = b k i ∀i ∈ V, ∀k ∈ K (1b) (i,j)∈E x k ij ≤ (i,j)∈P c ij + (|P | - e∈P y e )M ∀P : path for k ∈ K (1c) x k ij + x k ji ≤ y e , x k ij ∈ {0, 1} , y e ∈ {0, 1} ∀e ∈ E, ∀k ∈ K (1d)
Where (1b) represents the flow conservation constraint, and M is a large number. Our first contribution is to implement several cutting plane algorithms. These algorithms solve the initial problem defined by Eq. ( 2) so that we can obtain a subnetwork y * , and a set of paths x * for all origin-destination pairs. min

   e∈E f e y e + k∈K (i,j)∈E φ k g k ij x k ij , s.t. (1b), (1d)    (2)
Then, they check if x * is the shortest path in y * . If it is the case, then (x * , y * ) is optimal for the FCNDP-UOF. Otherwise, a valid inequality (1c) is added to problem (2) for the commodities with infeasible paths. We repeat the procedure until an optimal solution is obtained.

The above approach can be expensive in terms of number of iterations to get the optimum. Thus, our second strategy is to combine the two methods together to form a "branch-andcut" algorithm, known to be an efficient way to handle such constraints. The latter algorithm solves the problem (2) by a branch-and-bound procedure, and adds the inequalities to each integral node violating the shortest path constraint. Our second contribution lies in assessing the difficulty of the instances used in the simulations. To this end, we categorize the instances by calculating the angle α between the objective function vectors of the first and the second levels. We have coded and compared the performance of six algorithms listed as follows :

B&B1, B&B2 : one-level formulations from [2] and [3], respectively ; CP1, B&C1 : The proposed cutting plane and branch-and-cut algorithms ; CP2, B&C2 : Cutting plane and branch-and-cut algorithms using valid inequalities in [1].

Conclusions and future work

For the FCNDP-UOF, we proposed a new BIP formulation, a cutting plane and a branchand-cut algorithms to solve it. Numerical results demonstrate that the cutting plane algorithms outperform the other algorithms. The instances with an angle of 40 • ≤ α ≤ 50 • are the most difficult. For future work, we intend to work on exact approaches for the general case of the FCNDP-UOF, where the capacity constraint on edges is considered.

Références

  Numerical tests are performed using random and real data sets. Tab. 1 compares the number of instances solved to optimality in 3600s by the six methods. The table shows that CP1 and CP2 have a comparable performance and they solve more instances optimally than the others.

	TAB. 1 -Number of instances solved to optimality
		Total B&B1 B&B2 CP1 CP2 B&C1 B&C2
	0 • ≤ α ≤ 10 •	135	100	109 110 110	110	109
	40 • ≤ α ≤ 50 •	135	88	89	95	95	94	92
	80 • ≤ α ≤ 90 •	135	88	97 105 105	102	102
	Ravenna data	7	6	3	7	7	4	4
	Sum	412	283	301 317 317	310	307

  [1] Fatma Gzara. A cutting plane approach for bilevel hazardous material transport network design. Operations Research Letters, 41(1) :40-46, 2013. [2] Bahar Y. Kara and Vedat Verter. Designing a road network for hazardous materials transportation. Transportation Science, 38(2) :188-196, 2004. [3] A. Mauttone, M. Labbé, and R. M. V. Figueiredo. A tabu search approach to solve a network design problem with user-optimal flows. In VI ALIO/EURO Workshop on Applied Combinatorial Optimization, 2008.