M C Baas

G Verboom

C Paintaud

H J Desvignes

H J Guchelaar

Gelderblom
email: a.j.gelderblom@lumc.nl

L L Krens

J M Baas

M C Verboom

G Paintaud

C Desvignes

H Gelderblom

Pharmacokinetics and safety of cetuximab in a patient with renal dysfunction

Keywords: cetuximab, renal dysfunction, pharmacokinetics

Pharmacokinetics and safety of cetuximab in a patient

with renal dysfunction.

Introduction

Cetuximab is a monoclonal antibody, targeting epidermal growth factor receptor (EGFR) and registered for the treatment of colorectal and head and neck cancer. During its development, the drug has been investigated in patients with adequate renal and hepatic function only and a dose of 250 mg/m 2 every week, after an initial loading dose of 400 mg/m 2 , is defined in the summary of product characteristics.

No specific dose recommendations are given for patients with renal impairment (http://packageinserts.bms.com/pi/pi_erbitux.pdf ). The elimination of antibodies occurs via both nonspecific intracellular catabolism, following fluid-phase endocytosis, and receptor-mediated elimination after binding to their target antigen. Part of cetuximab clearance is therefore explained by binding to EGFR. Clearance of the EGFR antibody cetuximab seems independent of the liver and kidney function [Do you have references?]. In addition, there are 4 case reports [START_REF] Thariat | Cetuximab pharmacokinetics in end-stage kidney disease under hemodialysis[END_REF][START_REF] Inauen | Feasibility of using cetuximab and bevacizumab in a patient with colorectal cancer and terminal renal failure[END_REF][START_REF] Aldoss | Cetuximab in hemodialysis: a case report[END_REF][START_REF] Fontana | Colorectal cancer patient on maintenance dialysis successfully treated with cetuximab[END_REF] of haemodialysis patients who could safely be treated with standard doses of cetuximab. The aim of this study was to determine the pharmacokinetics of conventional dose cetuximab in patients with impaired renal function and to compare it to published data obtained in populations of cetuximab treated patients with normal renal function.

Method

Case presentation

We treated a 68 year old metastatic osteosarcoma patient with impaired renal function due to prior chemotherapy on compassionate use basis with 400 mg/m 2 cetuximab. This treatment was based on preclinical data on cetuximab activity in osteosarcoma [START_REF] Pahl | Anti-EGFR antibody cetuximab enhances the cytolytic activity of natural killer cells toward osteosarcoma[END_REF] and the lack of other treatment options.

The medical ethical committee approved the treatment and the pharmacokinetic analysis and the patient gave informed consent. The starting dose was 740 mg, preceded by the recommended 2 mg of the antihistamine clemastine, to avoid an allergic reaction to cetuximab.

Sample collection

Cetuximab was infused over two hours. Serum samples were collected 2, 3.5, 4.5, 44 and 168 hours after the end of the infusion in line with a previous pharmacokinetic study [START_REF] Azzopardi | Cetuximab pharmacokinetics influences progression-free survival of metastatic colorectal cancer patients[END_REF]. Cetuximab serum concentrations were measured using a validated immunoassay [8]. Limit of detection was 0.012 mg/L and lower limit of quantitation (LLOQ) was 0.75 mg/L.

Pharmacokinetic parameters

Pharmacokinetic parameters after the first dose, including dose normalised AUC from time zero to day 7 (AUC0-7), clearance (Cl), elimination half-life (t1/2) were estimated by trapezoidal non compartmental methods using MW/PHARM 3.5 (Mediware, Groningen, The Netherlands). Results

Results

The maximum concentration (Cmax), measured at the end of the 2 hour infusion, in this patient was 297 µg/mL, the minimum concentration (Cmin or through level) was 34.4 µg/mL. The reported serum concentration profile, shown in figure 1, was used to calculate the AUC, Cl and t ½. In table 1,an overview of the pharmacokinetics of cetuximab in study populations with normal renal function and in this case is shown [2;9;10]. In this table the pharmacokinetics after a single dose of 400 mg/m 2 are depicted, and are used for comparison.

In this patient the calculated AUC after the first dose was 20,280 µg*day/mL. The half-life after a single dose of cetuximab was 53.2 hours with a calculated clearance of 32.6 mL/h. The Cmax was approximately 30% higher compared to the Cmax in the studies of Tan et al. and Fracasso et al. [START_REF] Pahl | Anti-EGFR antibody cetuximab enhances the cytolytic activity of natural killer cells toward osteosarcoma[END_REF][START_REF] Azzopardi | Cetuximab pharmacokinetics influences progression-free survival of metastatic colorectal cancer patients[END_REF] The other parameters Cl, AUC, T1/2 and V are comparable as reported in those 2 studies. The half-life of cetuximab in our patient is 30% shorter than that calculated in the studies.

During the first course of cetuximab, the patient experienced adverse effects: reversible grade 2 hallucinations and fatigue. After careful considerations these symptoms were deemed to be most likely caused by the 2 mg clemastine. Due to these side effects the patient refused further treatment with cetuximab. Little is known about the pharmacokinetics of clemastine, nonetheless, normally no dose reductions are advised in patient with renal impairment.

Cetuximab-related side effects such as skin toxicity and diarrhoea did not occur in this patient during the first course.

Discussion

This case report shows that the pharmacokinetics of cetuximab in a patient with renal failure is similar to that with adequate renal function. Different studies investigated the pharmacokinetics of cetuximab in population with adequate renal function. For the comparison, studies with single doses were used. Most of the studies reported cetuximab pharmacokinetic parameters at steady state after a loading dose of 400 mg/m 2 followed by weekly doses of 250 mg/m 2 . Using a loading dose, steady state is usually reached within three weeks. Since this patient discontinued therapy after one single infusion, we can only compare this single dose to similar administrations available in the literature.

Our study shows some difference between the estimated parameters in our patient and patients with normal renal function. Due to inter patient variability, nonetheless, overall the kinetic profile is in line with the population with adequate renal function. Cetuximab pharmacokinetics are not studied in patients with impaired renal function, four case reports [START_REF] Thariat | Cetuximab pharmacokinetics in end-stage kidney disease under hemodialysis[END_REF][START_REF] Inauen | Feasibility of using cetuximab and bevacizumab in a patient with colorectal cancer and terminal renal failure[END_REF][START_REF] Aldoss | Cetuximab in hemodialysis: a case report[END_REF][START_REF] Fontana | Colorectal cancer patient on maintenance dialysis successfully treated with cetuximab[END_REF] studied the kinetics of cetuximab in patients undergoing haemodialysis. These four case studies showed that cetuximab in a patient with haemodialysis may be safely used. The estimated pharmacokinetic parameters were comparable to cetuximab patients with normal kidney function. Our study shows, for the first time, that pharmacokinetic parameters were also not altered in a patient with decreased renal function without haemodialysis.

Many drugs that enter the market are studied in a patient population with normal renal function and no dose recommendations are made for patients with impaired organ function and formal organ impairment studies are lacking. This is also the case for cetuximab. Other than from common sense and four case reports in haemodialysis patients there was no information available to guide our decision on how to use cetuximab in this patient. As renal impairment is a common problem in head and neck cancer patients due to prior cisplatin chemotherapy and in colorectal cancer patients due to the high incidence of the disease our finding of no clinically relevant alteration of cetuximab pharmacokinetics in our patient with impaired renal function has clinical importance for dose guidance in future patients.

  were compared to historical data on cetuximab in patients with normal renal function as reported by Tan et al.[9] and Fracasso et al.[10].

Acknowledgements

Measurement of cetuximab serum concentrations were carried out within the CePiBAc platform of Tours University Hospital. CePiBAc is cofinanced by the European Union. Europe is committed to the region Centre with the European Regional Development Fund.

Paintaud G: An enzyme-linked immunosorbent assay for therapeutic drug monitoring of cetuximab. Ther Drug Monit 2009;31:597-601. 9

Tan AR, Moore DF, Hidalgo


Paul Issartel
email: paul.issartel@limsi.fr

Florimond Gu Éniat
email: florimond.gueniat@limsi.fr

Mehdi Ammi
email: mehdi.ammi@limsi.fr

Slicing Techniques for Handheld Augmented Reality

Keywords: Augmented reality, 3D interaction, tangible user interface, scientific visualization H.5.1 [Information Interfaces And Presentation]: Multimedia Information Systems-Artificial, augmented, and virtual realities

Manipulating slice planes is an important task for exploring volumetric datasets. Since this task is inherently 3D, it is difficult to accomplish with standard 2D input devices. Alternative interaction techniques have been proposed for direct and natural 3D manipulation of slice planes. However, they also require bulky and dedicated hardware, making them inconvenient for everyday work. To address this issue, we adapted two of these techniques for use in a portable and self-contained handheld AR environment. The first is based on a tangible slicing tool, and the other is based on a spatially aware display. In this paper, we describe our design choices and the technical challenges encountered in this implementation. We then present the results, both objective and subjective, from an evaluation of the two slicing techniques. Our study provides new insight into the usability of these techniques in a handheld AR setting.

INTRODUCTION

Exploration of volumetric datasets is important in many scientific fields, such as physics, medical science or architecture. During exploration, scientists often use slice planes to visualize the internal structures of complex volumes. The orientation and positioning of a slice plane within a volume is inherently a 3D interaction task. Currently, this manipulation is generally done with standard desktop computers. However, interaction techniques are limited by the mouse, which does not provide sufficient degrees of freedom for 3D interaction. This leads to suboptimal and unnatural mappings. To overcome this issue, alternative approaches have been proposed. They support direct 3D manipulation of slice planes through realworld objects, called "tangible objects". The object is tracked in the 3D space, and its movements are directly mapped to the slice plane position and orientation.

These approaches can be divided into two groups. In the first group, the tracked object is used as an tangible slicing tool, and the result of slicing is shown on an external display. In the PassProps [START_REF] Hinckley | Passive realworld interface props for neurosurgical visualization[END_REF] interface, the slicing tool is a rectangular plate. De Guzman et al. [START_REF] De Guzman | EEWWW!!: tangible interfaces for navigating into the human body[END_REF] proposed a fork-shaped tool, which could be either attached to an adjustable arm or freely moved. Qi et al. [START_REF] Qi | Tangible user interfaces for 3D clipping plane interaction with volumetric data: a case study[END_REF] and Mulder et al. [START_REF] Mulder | The personal space station: Bringing interaction within reach[END_REF] used a pen, where the slice plane was orthogonal to its axis. In each case, however, the system assumes a fixed display (computer screen or workbench) and fixed tracking equipment (magnetic or optical). None of these interfaces are thus portable.

In the second group, the tracked object is a spatially aware display, thus the slice can be shown on the slicing tool itself. There is no need for any other input device. Examples of such interfaces include the mechanical device proposed by Bertelsen et al. [START_REF] Bertelsen | Volume visualization using a spatially aware mobile display device[END_REF], the PaperLens [START_REF] Spindler | PaperLens: advanced magic lens interaction above the tabletop[END_REF] and the WYSIWYF approach proposed by Song et al. [START_REF] Song | WYSIWYF: exploring and annotating volume data with a tangible handheld device[END_REF]. This configuration seems particularly suited to make the system fully portable. Yet, the aforementioned works are all constrained in this regard. The display in [START_REF] Bertelsen | Volume visualization using a spatially aware mobile display device[END_REF] is attached to a mechanical arm for tracking purposes. The display in the PaperLens interface is a passive sheet of paper, and thus requires a top projector to visualize the resulting slice. The WYSIWYF interface uses a handheld device with embedded sensors, however this is only one part of the whole system, which still depends on a fixed wall display.

We believe that a portable and self-contained system would be more convenient for end users. A portable system does not force users to go to a dedicated place. It can be brought inside the workplace, or even at home. Unlike fixed installations, it is also conceivable to deploy a portable system to all target users. Indeed, handheld devices have become a popular platform for designing portable and self-contained interactive systems, due to their ubiquity. When combined with augmented reality, they become a good platform for 3D interaction.

In this paper, we describe the adaptation of the two tangible slicing approaches identified above ("tangible slicing tool" and "spatially aware display") for a handheld AR interface. We discuss the design choices and technical challenges encountered during the adaptation. Finally, we present an evaluation of the two techniques in this new environment.

PORTABLE AUGMENTED REALITY PLATFORM

One of our main requirements was to use common, off-the-shelf hardware. The interface was implemented on an Android tablet: a Toshiba AT270. It has a 7.7" screen and weighs 332 g. This form factor was chosen to be easy to hold in one hand, while still having a relatively large screen. The tablet has a rear camera, which is required for handheld augmented reality.

Since the system was to be portable and self-contained 1 , we could only rely on the tablet's own sensors. However, in order to support 3D interaction in augmented reality, we also had to make the volume dataset appear to be outside the screen (i.e. move independently). Therefore, we needed a frame of reference outside the tablet. In our interface, we added an additional tangible object for this purpose: the reference object (Figure 2). It provides an anchor point for the volume dataset. The dataset is shown at this location, and thus appears to be outside the tablet, in a 3D space. The reference object is tracked -relative to the tablet -using ARToolKit 2 markers. Unlike markerless or environment tracking, which depend on a sufficient amount of visual features, an object with fiducial markers can be tracked reliably in any environment (such as the user's home or workplace). An alternative could be non-optical tracking, such as magnetic or inertial sensors, but this would require adding new sensors to the tablet and/or the reference object. Another benefit of a tangible reference object is to allow direct ma-nipulation of the dataset: the user can translate and rotate the volume by manipulating the object.

The volume dataset is loaded and processed using the VTK library 3 . It is then rendered with OpenGL ES 2.0, and overlaid on the reference object. In this configuration, the visual and interaction spaces become co-located. As shown by Ware et al. [START_REF] Ware | Rotating virtual objects with real handles[END_REF], this can significantly facilitate 3D manipulation.

SLICING TECHNIQUES

We implemented the two main slicing approaches described in the introduction. For the "tangible slicing tool" method, we used a tangible stylus. For the "spatially aware display" method, we used the tablet itself as a slicing tool.

Stylus-based slicing

We designed a pen-shaped tangible object, called the stylus, to be used as a slicing tool. In this mode, a virtual plane is attached to the stylus tip (which is tracked with markers). This plane defines a slice through the data volume. The result is shown in the augmented reality view, on the tablet screen (Figure 1). When the stylus is moved, the slice is updated in real-time, giving the impression of effectively slicing the virtual data with a physical tool.

Figure 1: A virtual slice plane is attached to the stylus, which is manipulated with the fingers to slice through the dataset.

The elongated shape of the stylus "suggests" to use it like a pen: it has a handle, and the tip is an effector. This means the stylus benefits from the dexterity previously acquired from the use of pens. When the stylus handle is held with the fingers, a few controlled movements are enough to change the position and the orientation of the slice plane in the data volume. This type of grasp is known as a "precision grasp" [START_REF] Subramanian | Tangible interfaces for volume navigation[END_REF]. As a precision tool, the stylus is expected to be more suited for accurate adjustment of the slice plane. It might however be less efficient for large movements within the dataset.

We chose to align the virtual plane with the stylus axis, instead of making it orthogonal as in previous works (e.g. [START_REF] Qi | Tangible user interfaces for 3D clipping plane interaction with volumetric data: a case study[END_REF]). This choice is justified by the constraints of handheld AR on a self-contained device. Since the camera used for tracking has to be located on the back of the tablet, users need to bring the stylus behind the tablet for it to be detected. The usable workspace is further limited by the user's arm length (because both the tablet and the stylus are held at the same time) and the camera's field of view. If the slice plane was orthogonal to the stylus axis, the users's wrist or arm would be blocked by the tablet. Therefore, we chose to align it with the stylus. Whether this change affects performance remains to be tested experimentally.

Since most handheld devices do not yet support 3D textures, slicing cannot be implemented directly in OpenGL. Instead, we used the VTK library to render the slice section into a texture, based on the geometrical transformation between the stylus tip and the reference object (i.e. the data volume). This texture is then applied to a quad -a rectangular polygon -aligned with the stylus, in the augmented reality view. 3 http://www.vtk.org/

Tablet-based slicing

In the second slicing technique, the tablet itself is used for slicing. It has a planar shape, and its position and orientation relative to the data volume (the reference object) are known from tracking. Thus the tablet is actually a spatially aware tangible plane, directly controlled by the user in the 3D space. This tangible plane defines a slice through the data volume. Any slice plane can be obtained by positioning the tablet and the reference object relative to each other. The slice section is displayed on the tablet screen (Figure 2), and updated in real-time during manipulation.

However, the tablet and the reference object are both physical, tangible objects, so they cannot interpenetrate. A strict interpretation of this metaphor would make some slice planes inaccessible. We worked around this issue by shifting the virtual slice plane a few dozen centimeters behind the tablet. This offset is called the slicing distance. Unfortunately, this makes it more difficult to rotate the slice plane within the data volume, since the center of rotation remains located on the tablet. To overcome this limitation, users can rotate the data volume instead, by manipulating the reference object.

Figure 2: The slice plane is located at a fixed distance behind the tablet surface. The user moves the tablet to manipulate the slice.

Compared to the stylus, the tablet is larger and relatively heavier. It will be held firmly by the fingers, possibly with both hands. This type of grasp is known as a "power grasp". Manipulating an object held in such a grasp involves large muscle groups, such as the elbow and shoulder, since fingers cannot be used. These muscles allow larger movements, but the manipulation is less accurate. Because of that, and because of the rotation issue described above, this interaction mode is expected to be more suited for quickly skimming through the dataset (i.e. large translational movements) than for precise manipulation of the slice plane.

The slice section is computed according to the relative transformation between the tablet and the reference object, and rendered into a texture using VTK. Since the slice plane is always aligned with the tablet, this texture is then applied to a fullscreen quad. To avoid occlusion artifacts with other virtual objects, the fullscreen quad is artificially given a depth value the depth buffer (equal to the slicing distance).

EVALUATION

In order to evaluate the two techniques (tablet-based and stylusbased), we compared them with each other in an exploration task. The goal was to highlight differences in performance and usability in this handheld AR configuration.

To assess their usefulness relative to more common techniques, we also compared them with a baseline. Currently, most scientists use either command-line or WIMP-style visualization tools to manipulate slice planes. Thus, a 2D mouse technique was deemed to be a good baseline. This technique was carefully reimplemented on a tablet, so as to be functionally equivalent to the slice plane widget found in ParaView 4 , a popular visualization software on desktop computers. When the right mouse button was held down, mouse movements resulted in a rotation ("arcball" rotation) of the slice plane within the 3D volume. When the left button was held down, mouse movements resulted in a translation of the slice plane along its normal vector. It should be noted that this technique cannot support simultaneous translations and rotations, because of the limited degrees of freedom provided by a 2D mouse (only 2 DOF). Since this is an inherent limitation of WIMP interaction, we believe the mouse technique described here is representative of current software.

The entire experiment was conducted on a single tablet. For the baseline mouse technique, we connected a wireless ambidextrous mouse to the tablet.

Participants

Nine unpaid participants (1 female, 8 male) from 23 to 29 years old (mean=25.0, SD=1.7) were recruited from the laboratory. One reported to be left-handed.

Experimental design

A CT-scan of a human head was loaded into the software and attached to the reference object (Figure 3a). Inside the head, 3 spherical targets ( 1 cm) were added. Their color was strongly contrasted with the rest of the data, to make them stand out among other cranial features (Figure 3b). Targets could not be seen from outside the volume, and were only visible on slice planes. Slice planes were controlled by the technique to be evaluated.

Participants were instructed to find the three targets within the data volume, and to make a slice plane intersecting all three spherical targets. This state had to be maintained for 0.8 seconds before the task was automatically validated. There was a 5 minute time limit on each trial. The three techniques were evaluated one after the other. To compensate for learning effects, the technique order was permuted between participants. We generated 18 predefined tasks (triplets of target positions). They were screened to exclude degenerate cases, such as overlapping targets, positions being too close to each other or to the volume bounds. All participants were presented the same 18 tasks, in random order. Each block of 6 consecutive tasks was accomplished with a different technique. Each technique block was preceded by 2 training tasks. Participants were allowed to take a short break between each block.

During the experiment, participants were seated in front of a table, on which the reference object was (initially) placed on a small cardboard box. This box allowed to raise the object slightly above the table surface, increasing the usable 3D workspace. Participants were permitted -and encouraged -to manipulate the reference object freely, since it is an integral part of the interface. Although being seated seems to forgo the advantages of portability, we argue that this setting is representative of a realistic use case. Even though the system itself is portable, in many cases the user does not have the physical space, or willingness, to move around the table. We still emphasize the importance of portability for the system itself, but in this experiment the user remained seated during the interaction. Therefore, manipulation of the reference object was often required to reach certain slice planes.

The main quantitative measure was the task completion time, but the entire slice plane trajectory was also logged. With 9 participants, 3 techniques and 6 tasks per technique, 162 trials were logged in this experiment. At the end of the experiment, participants completed a questionnaire. They were asked to grade each technique, on 6-point Likert scales, according to the perceived fatigue and their overall preference. They were also asked to summarize, in plain text, the positive and negative points of each technique.

Results and discussion

In this section, we present the preliminary results of this experiment. We intend to recruit more subjects to increase the reliability of these results.

Objective measures

The completion times were not normally distributed (Shapiro-Wilk W=0.79, p<0.01), but the data conformed to a lognormal distribution (Kolmogorov-Smirnov D=0.05, p=0.76). We thus applied a log transformation to the time data. Statistical analysis was performed on transformed data, but time values are reported untransformed. Variances were homogeneous in the transformed values (Levene F=0.55, df=2, p=0.57), meeting the prerequisites for parametric tests. The completion times are shown in Figure 4. The mean time was 56.5 seconds with the stylus, 84.6 seconds with the tablet, and 78.7 seconds with the mouse. A few trials had to be interrupted at the 5 minute time limit: 2 with the stylus, 1 with the tablet, and 3 with the mouse. An ANOVA revealed a significant difference among the mean completion times (F(2,16)=4.52, p<0.05). Posthoc t-tests, with Holm correction, revealed that the mean completion time with the stylus was significantly shorter than with the tablet (p<0.01) and the mouse (p<0.05). There was however no significant difference between tablet and mouse (p>0.05).

From these preliminary results, the stylus appears to be significantly more efficient than the baseline technique for this task. This confirms our assumption that a direct 3D positioning technique can be more efficient than a mouse-based technique for slice plane manipulation. The stylus completion times were also strongly skewed toward the left (γ 1 =2.16). This means that the stylus was much more efficient under some conditions than in others. In our interpretation, supported by observations, this difference was due to clutching. Whenever the target slice plane could be reached with small movements of the fingers, wrist or arm, the task was completed fairly quickly. When the target slice plane could not be reached because of anatomical constraints, participants had to interrupt the manipulation to change position (clutching), which caused longer completion times.

The tablet mode did not outperform the mouse for this task. This could indicate that tablet-based slicing was not suited for the experimental task: locating and reaching a given slice plane. To understand why, we analyzed the trajectories of the slice planes. Figure 5 shows the mean translation speed of the slice cross section. The speed was measured at the cross section centroid. Since the slice plane itself is manipulated very differently between the three techniques, we chose to analyze the trajectory of the useful part of the slice plane: the cross section. The mean speed was 1.2 cm/s with the stylus, 2.6 cm/s with the tablet, and 0.65 cm/s with the mouse. An ANOVA revealed a significant difference among the mean translation speeds (F(2,16)=38.77, p<0.01). Post-hoc t-tests, with Holm correction, confirmed that the three techniques were significantly different (p<0.01) in terms of translation speed. As shown in Figure 5, the cross section of the slice plane thus moved significantly faster in tablet mode. We then analyzed the expansion of this trajectory. Expansion was computed as the mean distance between all points of the trajectory and their isobarycenter -that is, the standard deviation of the trajectory points. It was 7.5 cm (SD=3.1 cm) with the stylus, 12.5 cm (SD=2.7 cm) with the tablet, and 6.9 cm (SD=3.7 cm) with the mouse. Statistical analysis revealed that the trajectory expansion with the tablet was significantly larger than with the other techniques (p<0.01 for both comparisons). Therefore, the cross section covered a significantly larger space in tablet mode.

Even though participants were performing the same task with all techniques, the cross section of the slice plane moved faster within the dataset in tablet mode, covered a larger space. This suggests that tablet-based slicing is more suited for exploration than the other techniques. However, participants did not complete the task faster with the tablet. Since the above results suggest this technique was faster for the initial exploration phase (i.e. locating the targets), we can assume it was much less efficient for the final alignment phase. This is consistent with our expectations. As explained before, the tablet is held with a firm grasp, or "power grasp", which involves large muscle groups. These muscles are suited for large and fast movements, rather than small and precise ones -for which the stylus is better. Furthermore, the slicing distance causes the slice plane to be shifted away from the tablet. While it does not affect translations, it causes the center of rotation to be located on the tablet itself, rather than on the slice plane. This restricts the range of possible orientations relative to the data, assuming users are seated. To reach further angles, users were forced to interrupt their manipulation to rotate the data volume instead, which decreased task performance. In contrast, the stylus mode was much less affected by this issue, since the distance between the stylus handle and tip is very small.

Subjective measures

All three techniques were almost equally rated in terms of fatigue. The mean ratings were: stylus=4.0, tablet=3.9, mouse=3.6 (on a 6point Likert scale, 1=not tiring, 6=tiring). Differences were not statistically significant. It seems that the technique itself did not have a major effect on user fatigue. Indeed, some participants commented that the tablet had to be held in the hand all the time, and that this factor alone was more tiring than any of the three techniques.

The overall preference ratings were more contrasted: stylus=5.0, tablet=2.7, mouse=4.0 (1=dislike, 6=like). The lower rating for the tablet mode was likely due to its inadequacy for the experimental task, as suggested by the objective measures. The difference between stylus and mouse was not statistically significant, which means that participants had diverging opinions. However, the preference rating for the mouse may have been biased by the familiarity with the device, and the stylus may have been rated lower due to occasional tracking issues. We expect that a larger number of participants will make the difference significant.

Participants were also asked to list in plain text the positive and negative points of each technique. Interestingly, many participants used similar words or expressions to describe the same point, while a few others had very specific opinions. The baseline mouse technique was found to be "accurate" by three participants, "stable" by three others. On the other hand, five participants said it was "unintuitive for controlling the slice plane, especially rotations". Five participants found the stylus "intuitive", and three others said it was "convenient for rotating the slice plane". On the negative side, three of them said it was "difficult to keep in the field of view" (behind the tablet), and three others complained about "tracking issues". One participant found the stylus to be "too sensitive", which may be related either to tracking issues, or to the novelty of tangible interaction. Regarding the tablet, three participants appreciated the fact that it can be "held with two hands". However, three participants found it "difficult and/or inaccurate for positioning the slice plane". Another one complained that the tablet had to be held "too close to the eyes". We indeed noticed that it was an issue for some participants, presumably because of the default slicing distance. Therefore, we recommend that implementations of this technique let users customize this distance.

CONCLUSION AND FUTURE WORK

This paper presents the adaptation of two slicing techniques for handheld AR: using a tangible slicing tool, and using a spatially aware display. Our implementation and user study provided new insight into the usability of these techniques in handheld AR. The results suggest that the tangible slicing tool is efficient for manipulating the slice plane, and the spatially aware display is better for exploration. On the other hand, the tangible slicing tool suffers from ergonomic issues, such as alignment with the handheld device and wrist constraints, while the spatially aware display is less convenient when the user remains seated. Future work should focus on improving the tangible slicing tool as well as the camera's field of view. The evaluation could also be repeated in a different configuration, where the user is moving and the data volume remains fixed. Finally, the two techniques could be combined to get the best out of each metaphor.

Figure 3 :

 3 Figure 3: Experimental task. (a) The head dataset. (b) Using tabletbased slicing to locate the targets (two targets, the red circles, are visible in the slice section).

Figure 4 :

 4 Figure 4: Task completion time (s)

Figure 5 :

 5 Figure 5: Mean translation speed of the slice cross section (cm/s)

The associated tangible objects are considered part of the system.

http://www.paraview.org/


P F Odgaard

C Aubrunp

Y Majanne

Fault Tolerant Power Grids

Keywords: Power Grids, Wind Turbines, Fault Tolerant Control, Fault Detection and Isolation

This special issue contains article on fault detection and isolation and fault tolerant control methods applied to different aspects of modern power grids, both for accommodating faults in the power grid, and for accommodation of faults in power generating units.

In the recent years a high focus has been drawn to making power grids smarter for a number of reasons. The main reason is to ensure grid stability and power quality in power grid, which has changed its nature from being dominated by a few large generating units to a grid dominated by many smaller generating units, with an increasing number of renewable generating sources like wind turbines and photo electric units. This has lead to a high research level in (Smart) Power Grids in general and as well in control of these grids. In the process of ensuring grid stability and quality it is important to detect, isolate and accommodate faults occurring in the physical grid or in the generating units.

The focus of this special issue on fault tolerant control in power grids is to draw research interest into different aspects of fault detection and isolation as well as fault tolerant control of power grids and power generating units. These methods plays an important role in the general handling of faults which could in the worst case result in stability problems in the grid.

This special issue contains a broad spectrum of contributions dealing with faults in wind turbine to a number of different aspects of fault detection and isolation and fault tolerant control in the actual power grid. These works apply a number of different methods. One trend can, however, be seen. Most solutions proposed for fault detection, isolation and accommodation in power grids are distributed solutions, which are beneficial as the grids are dominated in a higher and higher degree of smaller geographically distributed units.

Simani and Castaldi, [START_REF] Simani | Active actuator fault tolerant control of a wind turbine benchmark model[END_REF], present a fault tolerant control solution for a wind turbine connected to a power grid. They focus on accommodation of faults in hydraulic pitch actuator, by using adaptive filters designed using a nonlinear geometric approach. The proposed approach is evaluated on known wind turbine fault tolerant control benchmark model.

Segundo Sevilla, et. al., [START_REF] Sevilla | Fault-tolerant control design to enhance damping of inter-area[END_REF], present both passive and active fault tolerant control approaches for inter area damping in a power grid. A combination of local and remote sensors are used in the design. The schemes are tested on both a linear and nonlinear models of the Nordic equivalent power grid.

Juelsgaard, et. al., [START_REF] Juelsgaard | Fault tolerant distributed portfolio optimization in smart grids[END_REF], present a fault tolerant for portfolio balancing control approach. It consists of a distributed power optimization scheme for grid balancing, which provides tolerance to the grid in the way that it accommodates changes in the portfolio either due to faults or reasons. It also P.F. ODGAARD AND C. AUBRUN AND Y. MAJANNE includes controllable users in the distributed optimization. The work is evaluated with a numerical example.

Huang and Wu, [START_REF] Huang | Minimum cost upgrade of pmu networks for synchrophasor availability[END_REF], deal with the problem of inclusion of additional Phase Measuring Units (PMU) into the power grid control structure, and as well communication with these PMUs. The design is made tolerant towards random communication faults between the PMUs and the grid controller. The scheme is tested on the IEEE 118-bus test model. Yadykin, et. al., [START_REF] Yadykin | Stability analysis of large-scale dynamical systems by sub-gramian approach[END_REF], analyze power grid stability using finite and infinite Gramians to solve the differential and algebraic Lyapunov equations in the time and frequency domains which are used to model the power grid. The proposed scheme is applied to the Kundur four machine two area system model.

Tedesco and Casavola, [START_REF] Tedesco | Fault-tolerant distributed load/frequency supervisory strategies for networked multi-area microgrids[END_REF], presents distributed supervisory control strategy for load and frequency set point distribution for the power generators in a power grid. This solution accommodates unexpected changes in the power balance in the grid, by changing the set points for the active generating units. The scheme is evaluated with a model of a four area power grid.

Zhang, et. al., [START_REF] Zhang | Distributed sensor fault detection and isolation for multimachine power systems[END_REF], suggest a distributed fault detection and isolation method for detecting and isolating islanding faults in smart (power) grids. It is a distributed approach in which each power generating unit in the grid detects and isolates the faults, based on local measurements. It is evaluated on simulation study of a two machine power grid. Wu,et. al.,[START_REF] Wu | A novel islanding fault detection for distributed generation systems[END_REF], propose an active fault detection and isolation scheme for islanding faults. It is a distributed scheme, in which each generating unit in the grid introduces the active detection signals and detects and isolates the faults in the grid, which is obtained using a Set-Membership filter. The scheme is evaluated with a simulation study. Knuppel,et. al.,[START_REF] Knuppel | Fault diagnosis for electrical distribution systems using structural analysis[END_REF], present a structural approach for fault detection and isolation in a power grid. The scheme is designed to adjust itself to changes in the grid, such that the most efficient analytical redundancy relations are used. The scheme is evaluated by using PowerFactory simulation tools.


Tegawendé F Bissyandé
email: tegawende.bissyande@uni.lu

David Lo

Lingxiao Jiang
email: lxjiang@smu.edu.sg

Laurent Réveillère
email: reveille@labri.fr

Jacques Klein

Yves Le Traon
email: yves.letraon@uni.lu

Got Issues? Who Cares About It? A Large Scale Investigation of Issue Trackers from GitHub

Feedback from software users constitutes a vital part in the evolution of software projects. By filing issue reports, users help identify and fix bugs, document software code, and enhance the software via feature requests. Many studies have explored issue reports, proposed approaches to enable the submission of higher-quality reports, and presented techniques to sort, categorize and leverage issues for software engineering needs.

Who, however, cares about filing issues? What kind of issues are reported in issue trackers? What kind of correlation exist between issue reporting and the success of software projects? In this study, we address the need for answering such questions by performing an empirical study on a hundred thousands of open source projects. After filtering relevant trackers, the study used about 20,000 projects. We investigate and answer various research questions on the popularity and impact of issue trackers.

I. INTRODUCTION

Developers and users often find issues with software systems and are encouraged to report them in the available issue trackers that are set up by development teams. There exists a variety of issue tracking systems. Popular systems include Bugzilla, Jira, etc. Other development platforms such as Google Code, GitHub or Freecode have in-house implementations of issue tracking systems. Developers thus have ample opportunities to use issue tracking systems in their development and maintenance process.

Many research studies have investigated ways to improve the user experience in issue tracking systems [START_REF] Bettenburg | What makes a good bug report?[END_REF], [START_REF] Hooimeijer | Modeling bug report quality[END_REF], while others have proposed techniques for helping developers manage and address issue reports [START_REF] Antoniol | Is it a bug or an enhancement?: a text-based approach to classify change requests[END_REF], [START_REF] Sun | Towards more accurate retrieval of duplicate bug reports[END_REF]- [START_REF] Tian | Improved duplicate bug report identification[END_REF]. Also, a number of approaches have linked issue reports to changes in source code [START_REF] Bird | Linkster: enabling efficient manual inspection and annotation of mined data[END_REF], [START_REF] Wu | Relink: recovering links between bugs and changes[END_REF]. Despite the numerous studies that propose techniques for improving the issue reporting process, for enhancing the management of issues, and for leveraging issues in other software engineering tools, there have been limited surveys that investigate the actual adoption of issue trackers in software development and the impact of their adoption to project success.

Our goal in this paper is to fill a gap in the research relevant to software issues by investigating the adoption of issue trackers in software projects. Such a study could make practitionners aware of what to expect when they setup issue trackers for their projects, and would make researchers aware of the scope and potential bias of issue trackers data. In this work, we consider software projects where issue trackers are systematically set up and where there is little barrier to their usage. We also investigate the impact of the usage of issue trackers on the ultimate goal of project success. Project success can be characterized in diverse dimensions: monetary returns, number of downloads, or popularity. This study focuses on open source projects, and we only consider popularity as a measure of project success.

Our study exploits data from GitHub, a super-repository of software projects containing millions of projects. GitHub is free for open source projects and implements an in-house issue tracking system where users can file issues and tag them into self-defined categories. The issue tracking system is easy to use and is systematically provided to all projects hosted in GitHub. We collect a hundred thousands of projects from GitHub and investigate the adoption of issue trackers as well as the impact of issue tracking on software success.

Our contributions are as follows:

1) We perform a large scale study on thousands of software projects.To the best of our knowledge, it is the largest study conducted on issue trackers. 2) We investigate the adoption of issue trackers in terms of the projects that utilize them, the people that report issues, and the kind of issues that are usually reported. 3) We discuss our findings and share some insights on the correlations between the quantities of issue reports and various characteristics of software projects. 4) We also compare our findings with a previous study [27] on Firefox development where the authors have found that the size of the user community influenced the time-to-fix rate of bugs. We find that in general this correlation is small.

The structure of this paper is as follows. In Section II, we introduce GitHub and its issue tracker. Next, we elaborate our empirical study methodology in Section III. Section IV presents the findings of our empirical study; it highlights a number of research questions and their answers. We discuss threats to validity in Section V. Section VI describes related work. We conclude and mention future work in Section VII.

II. PRELIMINARIES & PROBLEM DEFINITION

In this section, we discuss the importance of issue tracking systems in software development, and briefly describe GitHub, the source platform of the projects in our dataset.

A. Issue trackers

Software development generally produces programs with two caveats: (1) they are often incomplete with respect to certain features, and (2) they are usually buggy. Developers testing the programs and end-users using the programs may then need to report issues when they believe that the product is not performing as it should or that it could be better at what it does. The valuable feedback from the user community is actually widely accepted as vital information for any software development [START_REF] Bettenburg | What makes a good bug report?[END_REF]. In practice, an issue report is a request for improving a software system, fixing a bug, adding new features, or enhancing documentation. Project development teams may also use issues to flag "TODO" tasks.

Issue tracking systems, also known as issue trackers, are software products that aim at facilitating the management of issues in a software development project, by providing a featurerich interface to ease issue reporting by the user community. It also enables a better follow-up of user's concerns by project developers. Studies have shown that issue trackers can be beneficial to both open and closed source projects [START_REF] Grammel | Attracting the community's many eyes: an exploration of user involvement in issue tracking[END_REF], when they are publicly available for use.

There is a plethora of issue trackers used in software development projects. A number of them are more commonly referred to as bug trackers as they focus more on bug reports. Many development teams, including those at the Apache Software Foundation or the Linux kernel community, rely on standalone products such as Bugzilla1 , and JIRA2 to track defects in their code, plan fixes, etc. Project hosting platforms, on the other hand, often resort to in-house implementations of issue trackers that are designed specifically to include a set of features in line with their hosting capabilities. GitHub belongs to this second category and associates with each hosted project an issue tracker that provides the usual facilities in issue tracking, such as filing issue tickets, tagging them accordingly to the nature of the issue, and labeling them as the state of resolution evolves. A project development team on GitHub can choose to enable or disable the in-house issue tracker. A team can also choose to use GitHub as a mirror for their project, while hosting their main development and maintenance activities somewhere else.

B. GitHub

The GitHub project hosting platform was launched in 2008 and has since grown to be one of the premier sites where over 3,000,000 projects are hosted and managed. The success of GitHub is largely attributed to the concept of social coding which has created a developer-friendly environment, allowing developers to network, collaborate and promote their projects. It enables forking, which allows to create copies of repositories, and watching, which allows developers to register to the events in a given project. It also supports following, which allows developers to subscribe to the activites of one another.

GitHub provides an extensive set of REST [START_REF] Fielding | Architectural styles and the design of network-based software architectures[END_REF] APIs3 that we can use to retrieve information on many project repositories whose contents are publicly accessible.

The projects hosted by GitHub offer the kind of diversity that is appealing for large empirical studies. Indeed, projects in GitHub produce software from various application domains, from web applets, gaming software, to operating systems. These projects are conducted by teams of developers of varying sizes-between 1 and several thousands, and include source code from a myriad of programming languages.

While our study focuses on the issue trackers managed by the different projects, there is a need to correlate statistical findings on issues with different characteristics of the projects. Thus, we are required to (1) collect a huge amount of data from GitHub and (2) infer important information, such as a project's number of lines of code, that is not directly available through the API, making a large study with GitHub challenging.

Other project hosting platforms, including SourceForge, GoogleCode and Freecode, also provide issue trackers to hosted projects. In this paper we focus on GitHub, but we plan to investigate those issues trackers in future work, to compare the findings, and draw new insights.

C. This study

A large body of literature has discussed the importance of issue trackers in the context of software development [START_REF] Ko | How power users help and hinder open bug reporting[END_REF], [START_REF] Singh | Users of open source software -how do they get help?[END_REF], [27]. Other studies have also investigated user involvement in issue tracking based on issues reported in a few software projects [START_REF] Grammel | Attracting the community's many eyes: an exploration of user involvement in issue tracking[END_REF]. A few other techniques, such as duplicate bug retrieval techniques [START_REF] Sun | Towards more accurate retrieval of duplicate bug reports[END_REF], [START_REF] Sun | A discriminative model approach for accurate duplicate bug report retrieval[END_REF] and techniques to link code changes with issue reports [START_REF] Bird | Linkster: enabling efficient manual inspection and annotation of mined data[END_REF], [START_REF] Wu | Relink: recovering links between bugs and changes[END_REF], have been proposed with the assumption of the prevalence of issues.

In this study, we consider a hundred thousands open source projects to investigate the popularity of issue trackers to provide insights on how issue trackers can be leveraged for improving software development.

III. METHODOLOGY

For our empirical study, we consider the first 100,000 projects returned by the GitHub API following a search request on available project repositories. These projects were retrieved with GitHub API v2 which returned a random list of projects 4 .

We collect our dataset based on information from those projects. The obtained dataset is furthermore curated to distinguish projects that can introduce bias in our analysis from others: e.g., some projects are hosted in GitHub but maintain their issue tracking system outside of GitHub. Last but not least, we statistically characterize the popularity and usage of issue trackers in several dimensions, relying on well-known metrics to assess the statistical significance of our findings.

A. Collecting the dataset

For each project in our large set of projects from GitHub, we collect data related to a number of important software artefacts that commonly occur in development processes and are leveraged in this study.

a) Project information: The GitHub social coding site provides a number of features that enable developers to monitor and track activities in a project repository. In this study, we focus on two concepts, namely watchers and forks, that can be used as metrics for evaluating the popularity and thus the success of a project. The "watchers" metric gives an indication of the amount of attention that is given to a project by the developer community. Developers who subscribe as watchers to a project typically use, report bugs and incidentally promote the project in their developer social network. Similarly, "Forks" is a useful metric for measuring the active involvement of the developer community in the growth of a project's code base and the improvement of its quality. Indeed, developers often create copies (i.e., fork) of project code bases for continuing the development in parallel, while merging their improvements from time to time with the mainline repository. Though these metrics are not absolute, they provide good insights on the popularity of a project. b) Issue reports: For each project, we also collect all (open and closed) issues reported through its tracker. For each issue, we collect information related to the reporter's identity, the labels used by the reporter to categorize the issue, etc. c) Lines of code: We consider the number of lines of code (LOC) in each project. Because GitHub uses the git5 software configuration management system (SCM) to store software revisions, we download the actual git repository for the project, and rely on the SLOCCount6 utility to compute the actual lines of code in the latest revision of the project, ignoring code comments and blank lines. d) Developer contributions: Finally, we consider for our study the development team for each project. Because git records code contributors' names and email addresses with each revision in the repository, we are able to accurately identify all contributors to a project code base, whether they are registered to GitHub or not. Indeed, the git SCM distinguishes between revision authors, who are the end-contributors to the code base, and committers, who have access to the mainline repository and can forward contributions from revision authors.

B. Curating the dataset

GitHub is a hosting site that is used by different project teams to get exposure in the developer community. Thus longlived projects whose active development has started before the era of GitHub keep managing their projects on their own development sites while mirroring their source code repository in GitHub. Examples of such projects include the numerous projects developed under the auspices of the Apache Software Foundation and the Linux kernel mainline tree. These projects disable the issue tracker provided by GitHub to avoid confusion on which tracker is maintained by the project development team. Thus, based on this information we remove from our study the 3,801 projects (out of the 100,000) for which the issue trackers were disabled.

We further investigate the list of projects, which include popular projects such as the Linux kernel or the Ruby on Rails framework, to ensure that most of the projects are non toy projects of substantial sizes. To this end we count the total lines of source code (LOC) in each project. Over 70% of the projects contain more than 1,000 LOC. Around 35% of the projects include more than 5,000 LOC, while more than 20% contain more than 10,000 LOC. Finally, over 600 projects contain more than 1,000,000 LOC. This distribution suggests that a significant number of the projects in the dataset are of substantial sizes. To further ensure that most of the projects are not toy projects typically created by GitHub new users, we have filtered the data to projects with at least 1,000 LOC.

C. Research questions

Research and practice of software development is performed under various assumptions about issue tracking that have not been validated through extensive empirical studies. Thus several hypotheses can be made, and accepted, that we will attempt to (in)validate in this work: H1 -Open source projects receive large numbers, e.g., over 1000, of issue reports.

H2 -During a software project lifecyle, developers write code for which a distinct community of users report issues. Thus we expect less than 50% of issue reporters to be among project developers, and vice versa.

H3 -Collaborative coding environment has a positive impact on issue reporting.

H4 -The more people are interested in a project and submit issue reports, the quicker project developers handle them.

Given the extent of our dataset, we investigate in this work the following research questions to explore and assess the relationship between various aspects of software projects and the activities in their issue trackers:

RQ1. What is the proportion of projects that receive issue reports and how can projects be differentiated in that respect?

For this research question, we study the overall adoption of issue trackers and investigate the characteristics of projects that receive issue reports.

RQ2. How many issues are tracked in projects whose trackers are used?

With this question, we further investigate the popularity of issue trackers by quantifying the number of issues reported in the projects.

RQ3. What is the number of occurences of category tags in issue reports? What are the most frequently appearing categories?

In this question, we investigate the kinds of issues that are reported, and we discuss the prevalence of the different categories.

RQ4. Who enter issues into issue tracking systems? How many of them are project team members?

We study the user-community that submits issue reports and investigate the proportion of reporters that contribute to the code base.

RQ5. What is the relationship between utilization of issue tracking system and project success (i.e., number of forks and watchers)?

We would like to establish and assess the correlation between the success of a project and the number of issue reports that it receives.

RQ6. Does the size of user communities impact the timeto-fix rates of bugs? In this research question, we investigate on a larger dataset of projects a result of previous studies based on a unique development project, namely Firefox.

D. Statistical measurements

Our empirical study is based on a sample of projects. Though, to the best of our knowledge, no related study involving issue trackers has ever exploited that many projects, there is a need to ensure that, statistically, our findings are significant. To this end, we resort to common metrics in statistical analysis for assessing the statistical significance of our figures, and to confirm the existence of a correlation among data from compared artefacts.

a) The Mann-Whitney-Wilcoxon (MWW) test: The MWW test is a non-parametric statistical hypothesis test that assesses the statistical significance of the difference between the distributions in two datasets [START_REF] Mann | On a test of whether one of two random variables is stochastically larger than the other[END_REF]. We adopt this test as it does not assume any specific distribution, a suitable property for our experimental setting.

Once the Mann-Whitney U value is computed it is used to determine the p-value. Given a significance level α = 0.001, if p-value < α, then the test rejects the null hypothesis, implying that the two datasets have different distributions at the significance level of α=0.001: there is one chance in a thousand that this is due to a coincidence. b) Spearman's rho: Spearman's rho (ρ), also known as Spearman's rank correlation coefficient, is used in statistics as a non-parametric measure of statistical dependence between two variables X and Y . This measure is used without any assumption that the data is normally distributed, making it a good fit for the datasets that we investigate in this study. The values of ρ are limited to the interval [-1; 1], and a perfect Spearman correlation of -1 or +1 occurs when each variable is a perfect monotone function of the other. The closer to 0 ρ is, the more independent the variables are.

IV. EMPIRICAL EVALUATION

In this section, we report the results of our empirical study. These results are provided as responses to the research questions that were formulated in Section III-C.

A. RQ1: Adoption of Issue Trackers

In the first research question we investigate the distributions of issue trackers across projects. By default, a developer creating a project in GitHub, is provided with a source control repository and a corresponding issue tracker for the project. Nonetheless, the developer can disable the issue tracking making it inaccessible to users. Issue trackers are disabled in only 3.8% of the projects. About 30% of the projects have issue trackers containing some issues, while 66% of the projects have completely unused issue trackers. In the rest of our study, we dismiss projects with issue trackers disabled from our dataset in order to emphasize on the differences between projects that have no issue reports (albeit their issue trackers are enabled) and projects having issues. Thus, 2 projects out of 3 do not have issue reports though their issue trackers are enabled. We thus investigate why.

To this end, we correlate the presence/absence of issues, in enabled issue trackers, with different properties of a project, including its age, the size of its code base, and the number of people in the development team. We also investigate the influence of the project's owner (in general the leader) over the presence of issues. a) Project age: We measure the age of a project solely based on its date of creation on GitHub. For each project, we count the number of weeks that have passed since it has been hosted in GitHub until June 2012. Our goal is to investigate the relationship between the duration of a project's exposure in GitHub and the presence/absence of issue reports. Figure 1 shows the distribution of project age for both datasets drawn as boxplots. The boxplots are drawn using the R statistical analysis tool. Each boxplot contains 5 main horizontal lines. From top to bottom, the first line indicates the MAXIMUM, i.e., the greatest value, excluding outliers (determined by the tool). All data points above this line are outliers. The second line indicates the UPPER QUARTILE, i.e., 25% of data points are above this line. The third line is the MEDIAN, the middle of the dataset. The fourth line is the LOWER QUARTILE, i.e, 25 % of data points are below this line. Finally, the fifth line indicates the MINIMUM, i.e., the least value, excluding outliers. Data points below this line are outliers (determined by the tool).

We find that, on average, projects without issues had been in GitHub for 70.05 weeks, with a median value of 49.57 weeks, while, on average, projects with issues had been in GitHub for 81.57 weeks with a median value of 70.04 weeks. Using the Mann-Whitney-Wilcoxon test, we have established that the difference between the two datasets is statistically significant at 0.001 significance level. This experiment reveals that projects with issues have been hosted in GitHub, in median over half a year longer than projects that do not have issue reports. b) Lines of code: Figure 2 shows the distribution of LOCs in project code bases. Projects with issues have a median value of 1,820 LOC, while projects without issues have a median value of 1,027 LOC. We have tested and confirmed that the two distributions are significantly different using the MWW test at 0.001 significance level.

Projects with smaller code bases are less likely to be the target of issue reports. This finding is in line with the general expectation. We furthermore investigate the size of project teams. Figure 3 shows the distribution of the number of contributors in projects with issue reports and in projects without issue reports. The median number of developers is 2 for projects without issues (mean=43) while the number is raised to 5 for projects with issues (mean=49). We again run the MWW test and find that the difference is statistically significant at a significance level of 0.001.

Fig. 3: Issues and Project Team Size

The conclusion in this experiment is that single-person and small-size-team projects are less likely to receive user feedback.

d) Popularity of project leader: Finally, we investigate the relationship between the popularity of project leader and the number of reported issues. We make the assumption that the project owner, i.e., the developer who created the project repository in GitHub, is the lead developer and that his/her popularity can be inferred from the number of developers that subscribe to his activities by following him. We plot in Figure 4 the distribution of the number of followers for both the owners of projects with issue reports and the owners of projects without issue reports. The median number of followers is 2 for the owners of projects without issues (mean=21), while this number is raised up to 15 for the owners of projects with issues (mean=130). We have done the MWW test and have found that the difference is statistically significant at the significance level of 0.001. We find that projects owned by developers having few followers receive less user feedback than those lead by developers having more followers.

Although there may be correlations among the various dimensions that we have used in this section to investigate the presence/absence of issues in the projects, it leads to the same conclusion: smaller projects developed by small teams consisting of unknown developers do not receive much user feedback, at least not in the form of issue reports. Note that there are many such projects in the open source community.

Many projects in GitHub do not receive any issue in their issue tracking systems although these systems are automatically set up for them. This finding invalidates H1, our first hypothesis from Section III-C. Projects with reported issues tend to be older, have more lines of code, have more number of developers in them, and have more popular owners. Project owners should be better "advertised" to other users of the social coding site GitHub which might help to cause the other users to report issues in the tracking system.

B. RQ2: Number of Issues Tracked

In total, for projects with issue reports (20,041 projects), we have collected 803,840 reports for our analysis. We investigate in the second research question the distribution of the numbers of issues reported per project. Table I details this distribution: 86% of the projects have less than 50 issues, while only 1.14% of the projects have more than 500 issues. Almost 7% of the projects have between 50 and 100 reports while only 0.01% of the projects (i.e., 2 projects) have at least 10,000 issue reports. Also, we investigate the numbers of issues per 1,000 LOC. For each project, we divide the number of issues with the number of kLOC. Figure 5 shows the distribution of the numbers of issues per 1,000 lines of code. The median number of issues per kLOC is 6.32. Nevertheless, the open source software projects in our dataset deal with a variety of applications domains with different complexity which can account for different bug rates. The projects also use different programming languages with disparate verbosity and a wide range of technical difficulties. We therefore investigate whether we can actually establish a correlation between the numbers of LOC and the numbers of issues. In Figure 6 we provide the scatter plot of the numbers of issues and the numbers of LOC from the projects in our dataset. We compute Spearman's rho which yielded a value of 0.341. According to Hopkins [START_REF] Hopkins | A New View of Statistics[END_REF], a Spearman's rho of between 0.3 and 0.4 indicates a moderate correlation.

Fig. 6: Scatterplot of # of LOC and # of Issues

Most projects have only a small number of reported issues. Less than 8% of the projects have more than 100 issues in their issue tracking systems. The amount of LOC is moderately correlated with the number of issues reported.

C. RQ3: Tagging in Issue Trackers

We discuss, in the third research question, the categorization of issue reports. GitHub issue trackers provide a flexible feature for tagging issues to categorize them in order to ease their management by developers. This flexibility, however, comes with a cost since issue reporters may categorize the issues with typographical mistakes, or using various idiosyncrasies. Among the 803,840 issue reports in our dataset, 218,467 (27.17%) contain tags. Issue reporters have used in total 6,951 distinct tags to categorize their reports. Table II presents the top-10 frequent tags. There are two widespread tags, namely bug and feature, which are observed in 18.36% and 10.29% of the tagged issues respectively. Issue trackers are then essentially used in software projects to report bugs and request new features. Unfortunately, since tags are not predetermined by GitHub, many tags can be used to refer bug report or feature request. Table III lists sample tags that are used to refer to a bug report or a feature request. The variety of terms mainly stems from the preferences of issue reporters and typographical mistakes. In order to more accurately quantify the number of issues belonging to the top-2 categories, we cluster the tags that are relevant to bug reports and feature requests. We first manually create two small clusters, of sizes 5 and 7, that correspond to bug reports and feature requests respectively. Next, we want to semi-automatically add more tags to these clusters. Rather than manually checking all of the thousands of tags, we want to partially automate the process. To this end, we compute the Levenshtein distance [START_REF] Levenshtein | Binary Codes Capable of Correcting Deletions, Insertions and Reversals[END_REF] between each tag and the sets of tags in our predefined clusters. Let T t be a tag that we evaluate to include in one of our clusters, and T c a tag that already belongs in the cluster and to which T t is compared. We compute the Levenshtein distance d between T t and T c . Let the string length of T c be l. If d < l 2 , then we add the tag T t to the cluster. Using this formula, we were able to include more tags into each cluster. At the end of the process, we manually check each tag in the clusters. This process has enabled us to identify 91 tags that belong to the category of feature requests and 18 tags in the category of bug reports. Table IV details the results after clustering tags that refer to bug/error reports and feature/enhancement requests. The updated findings suggest that bug/error reports and feature/enhancement requests are equally important for issue reporters.

The second observation from Table II is that there is a significant number of issues that are tagged with internet browsers' names. This suggests that projects related to web development receive more issue reports than the others. This in turn may imply either one of two distinct phenomena : (1) web applications have more implementation issues than other types of software; (2) users of web applications more readily report issues than users of other types of software programs.

To assess the reality of the first phenomenon, we assume that a software application domain can be inferred from the programming language used in its code base. Though this assumption cannot be generalized, we can, to some extent, do this for web programming as many languages targeted at web development are specialized for web programming. We then attempt to establish whether projects written in common languages for web programming, such as Ruby, Python, JavaScript or PHP, contain more issues than the rest. We use David A. Wheeler's SLOCCount [START_REF] Wheeler | SLOCCount: Counting physical Source Lines of Code[END_REF] utility to count actual physical source lines of code and their associated language. We classify the languages according to the number of issues that are filed in all the projects that are written in each language. Figure 7 shows the distribution of the number of issues for the top-10 languages that correspond to the highest number of issues. We observe that PHP is hidden in the middle with a median value comparable to that of system-level languages such as C and C++. Ruby has a slightly higher median value, while JavaScript does not appear in the top-ranked languages. Thus, we dismiss the first phenomenon. We conclude that users of web applications more readily report issues. We hypothesize that since users of web applications are already accessing the web when they encounter the issues, they can directly report them to issue trackers which are online.

• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •
Only less than 30% of issue reports in our dataset are tagged. The two most common tags are bug and feature. More than 12% of issues are tagged with labels related to web browsers and/or web applications

D. RQ4: Who Enter Issues into Issue Trackers

We now discuss the identity of issue reporters. The goal is to investigate the actual feedback provided by the user community as compared to the issues reported by development team members. Thus, for each project with issues, we analyze the number of developers, the number of issue reporters and compute the proportion of the project developers that also file issue reports and vice versa.

Based on developer names, we have identified in total 581,856 distinct developers who have contributed to the code bases of the projects with issues. Those issues were filed by 239,629 reporters. We have furthermore found that, for each project, one third of the developers (33%, the median value) have written some issue reports for the project. On the other hand, 42% of the issue reporters for a given project do not contribute to the code base.

Figure 8 shows the distribution of developers and reporters for all projects. The box plot shows that most developers in a project do not report issues in their own project. However, it indicates that a large portion of issue reporters actually contribute to the code base of the project. Project developers report issues, and issue reporters also often contribute to the code base. Our second hypothesis H2 is therefore disputed. Development teams should therefore acknowledge reporters to encourage them to become active contributors.

E. RQ5: Issue Trackers and Project Success

To provide insights on the fifth research question, we investigate the relationship between the success of a project and the number of its reported issues. To this end, we rely on popularity metrics based on GitHub's social coding featuresforking and watching-to estimate the level of interest in a project. a) Watchers: We first consider that the success of a project is proportional to the number of developers that watch the project. The score of the success metric can change as GitHub allows users to unwatch a project if they lose interest in it. Figure 9 shows a scatter plot that we use to explore the correlation between the number of issues and the number of watchers for each project. The Spearman's rho for the two distributions is 0.628, suggesting a large correlation between the numbers of issues and the numbers of watchers. We proceed to investigate the correlation between the numbers of watchers and the numbers of issue reporters. The scatter plot of Figure 10 shows the correlation between the two datasets. Spearman's rho amounts to 0.789. This suggests that there is a very large correlation between the numbers of issue reporters and the numbers of watchers. Figure 11 shows the scatter plot of the numbers of issue reporters and the numbers of forks from the projects in our dataset. We have computed Spearman's rho which yielded the value of 0.829, suggesting a very large correlation between the number of forks and the number of issue reporters. We also compute Spearman's rho to assess the strength of the dependence between the number of forks and the number of issues as in the previous experiment. In this case, the coefficient drops to 0.669, suggesting simply a large dependence.

Forks vs. Issue Reporters

There is a large correlation between the number of issues and the number of watchers and forks. There is also a very large correlation between the number of issue reporters and the number of watchers and forks. Thus, distributed programming positively impacts the amount of feedback (i.e., issue reports) for a project, suggesting that this software development style should be promoted more. Our hypothesis H3 is thus validated.

F. RQ6: Does the number of reporters impact the time-toclose?

Finally, we investigate the relationship between the number of reporters and the effort made by developers to quickly address issues. In previous work based on issue reports, Van Liere had shown that the large number of Firefox issue reporters has lead to the reduction of the time-to-close interval for software defects [27]. We therefore investigate whether this is a phenomenon common among thousands of projects or whether it is specific to the development setting of Firefox.

Figure 12 shows a scatter plot of projects with their respective numbers of issue reporters and the median timeto-close intervals. A time-to-close interval is computed as the number of days between the creation date of the issue report and its close date. The graph does not show any linear dependence between those two aspects of project development. We furthermore compute Spearman's rho which yields a rho value of 0.161 confirming that there is only a small correlation among the numbers of issue reporters and the speed in which the issues are addressed.

The time-to-fix intervals of issues is only slightly impacted by the number of issue reporters. We therefore note that Hypothesis H4 is disputable.

V. THREATS TO VALIDITY

We have identified the following threats to validity to our study. a) External validity: Our sample set of projects, though sizeable with tens of thousands projects, may not represent the universe of all real-world projects. Furthermore, we have b) Internal validity: Second, although we have curated our datasets to remove projects whose development, in particular issue management, occurs outside GitHub, some project owners may not disable the issue trackers even though they only deal with issues reported from other channels. This can lead to bias in our survey. Nonetheless, we have manually checked that for popularly known projects, such as Linux, that maintain issue trackers in well-known locations outside GitHub, have disabled their issue trackers on GitHub. Finally, we believe that, with a sizeable sample of 100,000 projects, the number of outliers is very small compared to the rest.

In addition, we have filtered our dataset to only consider projects with more than 10 kLOC. A different threshold may yield different findings. c) Construct validity: Finally, we have relied on heuristics for estimating the popularity of a project leader, the success of a project, and the application domain of a project. As we have described for each, those are simple proxies to the ground truth. However, they provide useful metrics and good indications on the popularity of each project, the popularity of each project owner, and the domain of a project in GitHub.

VI. RELATED WORK

GitHub, as a social coding site that hosts millions of software projects, contains a wealth of information about the practice of software development. In previous works, data from GitHub was leveraged to conduct large-scale studies on the popularity of programming languages [START_REF] Bissyandé | Popularity, Interoperability, and Impact of Programming Languages in 100,000 Open Source Projects[END_REF] and the adoption of software testing [START_REF] Singh | Adoption of Software Testing in Open Source Projects -A Preliminary Study on 50,000 Projects[END_REF]. In this paper, we exploit this data to investigate issue reporting in open source projects. To stress the importance of issue reporting we highlight in the following a number of studies related to issue reports in software development projects.

A. Acknowledging user communities

A significant number of studies have discussed the importance of the feedback provided by user communities in the life cycle of a software development project. Bagozzi et al. [START_REF] Bagozzi | Open source software user communities: A study of participation in linux user groups[END_REF], Iivari [START_REF] Iivari | Empowering the users? a critical textual analysis of the role of users in open source software development[END_REF], Hendry [START_REF] Hendry | Public participation in proprietary software development through user roles and discourse[END_REF] and Singh et al. [START_REF] Singh | Users of open source software -how do they get help?[END_REF] have investigated the role of users in open source communities. In these studies they make the point that users actually drive the software project and its evolution. Furthermore, they discuss how users influence the project towards their needs and how developers acknowledge users' input.

Unfortunately, while the importance of user communities is widely accepted, the participation of users in issue tracking is more controversial in the literature. For example, considering the case of the Firefox web browser, Van Liere has concluded that large number of bug reporters reduces the time-to-fix interval for software defects [27], while Ko and Chalina reported for the same system that users file "non-issues that devolved into technical support, redundant reports with little new information, or narrow, expert feature requests" [START_REF] Ko | How power users help and hinder open bug reporting[END_REF]. Based on our findings on thousands of open source projects, it appears that there is only a small correlation between the number of issue reporters and the time-to-fix rates of bugs.

Recently, Grammel et al. have explored the user involvement in issue tracking with a comparative study between open source and commercial development [START_REF] Grammel | Attracting the community's many eyes: an exploration of user involvement in issue tracking[END_REF]. Basing their study on the open source Eclipse project and the closed source project IBM Jazz, they show that closed source projects can also successfully receive user feedback through issue reports as with any open source project. This suggests that our own study, which is based on open source projects, could be generalizable to closed source projects.

B. Exploring issues in software code

A large body of the literature has discussed the correlation between various properties of software code and software development processes with the presence of defects. Nagappan and Ball have shown how churns in code changes correlate with increases in software failures [START_REF] Nagappan | Using software dependencies and churn metrics to predict field failures: An empirical case study[END_REF]. Koponen and Tintula, on the other hand, have investigated on Mozilla and Apache whether the changes made in project code bases were induced by defect reports. They found that this was clearly the case for Mozilla but not for Apache [START_REF] Koponen | Are the Changes Induced by the Defect Reports in the Open Source Software Maintenance?[END_REF]. Posnet et al. have explored how new features, on the one hand, and improvement tasks, on the other hand, affect the quality of code [START_REF] Posnett | Got issues? do new features and code improvements affect defects?[END_REF].

C. Improving issue reporting

Issue reporters may file reports that are incomplete or even invalid, adding to the challenges that developers face in software projects. A number of studies have discussed and proposed approaches to enhance issue reports to make them more useful [START_REF] Spolsky | Joel on Software[END_REF]. Hooimeijer and Weimer have proposed a descriptive model for measuring the quality of a bug report [START_REF] Hooimeijer | Modeling bug report quality[END_REF]. To build the model, they assume that the "time until resolved" is a good indicator for the quality of a bug report. Bettenburg et al. have later relied on actual developer feedback to train CUEZILLA [START_REF] Bettenburg | What makes a good bug report?[END_REF], their utility for providing bug reporters with tips to improve the quality of their reports.

D. Enhancing issue management

Despite the various guidelines on effective bug reporting that float around the internet, development teams are flooded with a considerable number of reports with disparate quality and usefulness. Researchers have thus been devising approaches for automating various processes in the management of issues. Canfora et al. [START_REF] Canfora | Supporting change request assignment in open source development[END_REF] and Anvik et al. [START_REF] Anvik | Who should fix this bug?[END_REF] have proposed different approaches for automatically assigning developers to bug reports. Weiss et al. have proposed to automatically predict the effort for fixing bug reports and estimate the time it will take to fix them [START_REF] Weiss | How long will it take to fix this bug?[END_REF]. Hosseini et al., in their approach for improving bug assignment, rely on the stated characteristics of a bug, such as severity, platform or priority, to predict its time-to-fix [START_REF] Hosseini | A market-based bug mechanism using predictive bug lifetimes[END_REF].

VII. CONCLUSION AND FUTURE WORK

Issue reports are important artefacts in software development. Various issue tracking systems have been developed and widely used, among which, Bugzilla and Jira are well-known. Hosting platforms such as Freecode, Google Code, and GitHub implement in-house issue trackers to collect user feedback. The assumed prevalence of issue reports in software development projects has led to the development of approaches for various purposes, such as improving issue reporting, enhancing issue management, and leveraging issue information for bug fixing.

In this paper, we have investigated the actual adoption of issue trackers in software projects to shed lights to many research questions on the involvement of users and developers in reporting issues. We describe the findings of our empirical study on tens of thousands of open source projects in GitHub. We have found the following results:

• issues are almost exclusively reported in large projects with big development teams led by developers having the most followers; [H1 disputed]

• a large proportion of issue reporters are actually involved in the development of the project; [H2 disputed]

• distributed development, recognized with project "forking", contributes to an increase in the number of issue reporters; [H3 holds]

• there is only a small correlation between the numbers of issue reporters and the time-to-close delays.

[H4 disputed]

• there is a moderate correlation between the numbers of issues tracked and the numbers of lines of code;

• social coding, through the intuitive implementation of project "watching", have an influence on the prevalence of issue reports;

• issue reporters are equally interested in reporting bugs as well as requesting new features in a project;

In this paper, we have limited our study to a snapshot of the projects at a given date: June 2012. In future work, we would like to investigate other questions involving temporal information, such as "When do people start reporting issues?", and "Do issue reports contribute to attract developers in joining the development teams?".

Fig. 1 :

 1 Fig. 1: Issues and the Age of Projects

Fig. 2 :

 2 Fig. 2: Issues and Lines of Code

Fig. 4 :

 4 Fig. 4: Issues and Popularity of Project Owners

Fig. 7 :

 7 Fig. 7: Prevalence of Issue Reports for Common Languages

Fig. 8 :

 8 Fig. 8: Proportions of Developers among Issue Reporters and Vice Versa

Fig. 9 :

 9 Fig. 9: Issues and Project success: Watchers vs. Issue Reports

Fig. 10 :

 10 Fig. 10: Issues and Project Success: Watchers vs. Issue Reporters b) Forks: Here we measure project success by the number of forks. Indeed, the number of forks provides a relatively good indication of the involvement of non-team developers in the development of a project.

Fig. 11 :

 11 Fig. 11: Issues and Project Success:

Fig. 12 :

 12 Fig. 12: Scatterplot of the Number of Issue Reporters and the Time-to-Close focused this study on open source projects found on GitHub which may not perfectly generalize to every software project. Nonetheless, to the best of our knowledge, GitHub is the largest database of projects and has no restriction on the types of project that can be hosted. Moreover, as the study of Grammel et al. suggests, issue reporting in closed source projects yields similar data to that in open source projects [9].

TABLE I :

 I Prevalence of Issues in Trackers

	# Issues	# Projects	% of Projects with Issues
	0 -9	11,602	57.89%
	10 -49	5,526	27.57%
	50 -99	1,411	7.04%
	100 -249	976	4.87%
	250 -499	290	1.44%
	500 -999	163	0.81%
	1000 -4999	69	0.34%
	5000 -9999	2	0.01%
	≥ 10000	2	0.01%

Fig. 5: Number of Issues per 1,000 LOC

TABLE II :

 II Top-10 Popular Tags in GitHub Issue Reports

	Tag	# Tagged Issues	% of Tagged Issues
	bug	40,112	18.36%
	feature	22,477	10.29%
	enhancement	11,584	5.30%
	Win7	9,736	4.38%
	ie8	7,626	3.49%
	chrome	6,817	3.12%
	other	6,667	3.05%
	FireFox	5,669	2.59%
	Feature request	5,594	2.56%
	wrong-or-unclear	5,464	2.50%

TABLE III :

 III Bug Report and Feature Request Tags

bug bug; defect; type:bug; Browser Bug; bugfix; etc. feature feature; request; proposal; featreq; feautre; etc

TABLE IV :

 IV Bug Report and Feature Request Tags

	# Label	# Labeled Issues	% of Labeled Issues
	bug/error	45,123	20.65%
	feature/enhancement	46,402	21.23%

http://bugzilla.org

http://atlassian.com/software/jira

http://developer.github.com

The list is available at http://momentum.labri.fr/orion/project list.txt

http://git-scm.com

http://dwheeler.com/sloccount