Peter Senna Tschudin

Julia Lawall
email: julia.lawall@lip6.fr

Gilles Muller
email: gilles.muller@lip6.fr

3L: Learning Linux Logging (Extended Abstract for BENEVOL'15)

Keywords:

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

I. INTRODUCTION

Logging is the programming practice of inserting statements in the source code to collect and report run time information. This run time information is typically useful for debugging, especially of rare events, which makes logging popular and makes it important for software reliability. In order to log effectively a developer must choose an appropriate position in the source code for the logging statement, a severity level e.g. warning or error, the correct logging function, and how much to log. Deficient logging may leave the user with insufficient information about the program execution, making debugging difficult, while logging incorrectly may result in unintended consequences such as performance overhead and may overwhelm the person trying to understand a problem with redundant information. Thus, developers working on large code bases usually face two distinct, but related problems when deciding how to log: there is no complete specification on how to log, and it is challenging to learn how to log properly, as it involves domain knowledge.

The Linux kernel is an open source operating system with over 13 million lines of C code. In the source code of Linux 4.2, released in August 2015, there are eight logging severity levels, 1,124 logging functions, and 268,156 calls to logging functions. Of the 1,124 logging functions, 81 were not present in Linux 3.16, released around one year previously. While there are 4,396 files in the Linux kernel documentation directory, only two of them contain at least one section that explains how to log properly. An alternative source of information on the latest programming practices, including how to log properly, is the development history of the Linux kernel. Developers may use the history for finding recent changes to existing code, and for finding recently added modules. The development history of the Linux kernel has 535,011 commits since 2005, and 54,827 commits in the last 12 months. The speed at which Linux evolves makes it challenging for developers to stay up-to-date with the latest practices.

Recognizing the difficulty of correct logging, Zhu et al. [START_REF] Zhu | Learning to log: Helping developers make informed logging decisions[END_REF] propose a machine learning framework that learns the common logging practices on where to log from existing logging instances in the source code of large C# user space software. The framework guides developers by suggesting where to log, via the integrated development environment (IDE). Zhu et al. identified the important factors for determining where to log and extracted features from the source code to satisfy their findings. However, the problems of determining the correct logging function, determining the correct severity level, and keeping up to date with the conventions of a fast-changing code base, all of which are particularly relevant to the Linux kernel, remain open.

We propose a framework that learns existing logging practices from features extracted from the software development history, with the goal of enabling our methodology to detect programming practice trends, e.g. API changes, before the new practice becomes widely adopted. We will apply our framework to the development history of the Linux kernel for making suggestions of where to insert logging statements, which logging function to use, and which severity level to use.

Our work proposes to extend previous work by detecting programming practice trends, and by making suggestions of the logging function and severity level to use. Our work will also be different from previous work in terms of the source used for extracting features: we propose to extract features from the development history, and weight the features by age, while previous work only considered the latest version of a code base.

In this paper we present only our preliminary work, focusing on the motivation and methodology of our approach. We leave a complete design, implementation and evaluation for future work.

II. BACKGROUND

Logging in the Linux kernel is challenging at two main levels. The first challenge is to determine where to log. Here, the developer must take into account the interaction between the overhead of logging and the stringent performance constraints of some parts of an operating system kernel. The second challenge is to determine how to log. Here, the developer has to carefully choose the right logging function and the right severity level, to ensure that the information provided will be helpful but not overwhelming. We examine these issues in more detail in the rest of this section.

Performance constraints (where to log): One of the main requirements on operating system code is that it be unobtrusive. Thus logging, which incurs a performance overhead, can be incompatible with critical execution paths, such as the processing of high speed network traffic. As a concrete example of the difficulty of choosing where to log and the performance consequences of logging in the wrong place, we consider the Linux BCM4706 ethernet driver 1 of which an extract is shown in Figure 1. This driver contains three badly placed logging function calls, of which one is shown in the final if statement of Figure 1 (lines 31-32). The if condition shown here (line 30) is the same at all three logging calls, and is related to the hardware properties of the device. Thus, if one of the tests is true, then they are all true, and they are all true for every processed packet. We found that the execution time overhead caused by emitting these three logging messages per network packet caused the driver throughput to slow down by 37.93 times, from 4.4MBps to 116KBps. The first author has submitted a fix for this issue, which has been accepted into the Linux kernel. 2 This example shows that logging must be done only when it is absolutely needed, and that it should be limited to code that is not performance critical. Performance constraints, however, are not always explicit in the source code, making correctly localizing logging code hard for developers.

Debugging challenges (how to log): A general challenge at the kernel level is debugging itself, because of the high degree of concurrency between processes and interrupts, because of the lack of the runtime support required to use conventional debuggers, and because of the close interaction with hardware, which itself can have unpredictable behavior. These factors can make error conditions difficult to reproduce, implying that it is essential that any log messages that are produced be comprehensive and unambiguous. These issues in turn have led to evolution in the Linux kernel logging API, and have resulted in a huge number of different logging functions with different properties.

The top of Figure 2 shows the growth in the number of Linux kernel logging functions over the last roughly 10 years, from Linux 2.6.12 released in 2005 to Linux 4.2 released in 2015. During that period, the number of logging functions increased by 3.42 times. This increase mirrors the increase in the number of lines of source code, which has grown by 3.17 times in the same period, as shown in the bottom of Figure 2. As the kernel indeed continues to grow, we may expect that the 1 drivers/net/ethernet/broadcom/bgmac.c 2 Linux kernel commit 8edfe3b6. number of logging functions will also keep growing, further complicating the logging choices facing the developer. Among the more than 1,000 logging functions provided by the Linux kernel in its most recent release, we can distinguish between a small set of generic logging functions, complemented by a huge collection of logging functions that are dedicated to a particular service, such as a particular device driver or a particular file system. The original kernel logging function, which was already present in Linux 1.0 in 1994, was printk, which was accompanied by a family of 8 macros representing different severity levels. These severity levels allow the developer to globally turn on or off log messages with various degrees of criticality, to address performance considerations and to avoid overloading the kernel logs with too much information. Starting in 1996, a new set of API ). Specific services may in turn use these logging API functions to define service-specific logging functions that encapsulate the logging of various servicespecific information. The latter functions are typically both defined and used in a single module. For example, Figure 3 shows the logging functions defined by the XFS file system. These logging functions support a variety of severity levels and make it possible to systematically include environment information such as the module name and the mount point used by the file system, when this information is available. The difficulty of choosing among the available logging functions is further compounded by their changing usage over time. Figure 4 shows the evolution in the use of logging functions, again over roughly the last 10 years. In the figure, the lower three lines represent the rate of calls, i.e., the number of calls per line of code, to printk (dashed and dotted red), to the generic logging API functions ("new api", solid blue), which we recognize as those logging functions beginning with the prefixes pr_, ata_, dev, fs_, hid_, net, or v4l, and to all other logging functions ("others", dotted grey). The upper line ("total", dashed black) indicates the rate of any of these calls. We observe that consistently around half (51.84% in Linux 4. the service-specific functions. However, the rate of use of the generic API functions, which was initially almost nonexistent, has been steadily growing, while that of printk, which was initially highly prevalent, has been steadily shrinking. Finally, we can also observe that the overall rate of logging calls is decreasing slowly, reducing by 9.26% in the last 10 years. This may suggest that there are contexts in which logging is no longer needed or no longer desired, further complicating the task of the developer.

III. METHODOLOGY

The goal of our tool is to provide relevant information to help developers decide when to log, which severity level to use, and which logging function to use, focusing on error paths. We are designing our tool to be a patch and file verification tool, as might be used in a commit hook of a source code management system such as git. An overview of the usage of our tool is shown in Figure 5. Our tool takes as input a patch or file, and then passes this patch or file to the feature extraction pipeline, shown in detail in Figure 6. The obtained features are then used to query a previously trained model that returns a log label. The log label represents the set of logging decisions, including when to log, which logging severity level to use, and which logging function to use. Finally, the inferred log label is compared to the current logging properties of the code, and fixes are proposed for any inconsistencies.

In the rest of this section, we describe the model training process. The result of the training process can be saved for reuse and can be shared to simplify and speed up the use of our tool. pipeline [START_REF] Kotsiantis | Supervised machine learning: A review of classification techniques[END_REF]. The goal of the machine learning process is to construct a model that is able to label data elements based on their features. Features and labels are representations of the data that must be created to feed the model training process. A feature represents one attribute of the data, such as the error code returned in the error path of a function. A label represents a property of the data, such as the need to log the code snippet. The combination of all features extracted from one instance of the data makes up a training example, and the label of that data instance represents the knowledge the model is learning.

A. Machine Learning Overview

Our training framework, as illustrated in Figure 6, has three parts, the dotted line divides the graph into three data representation spaces: data space, feature space, and classification category. The data space contains the Linux kernel source code, patches and meta data, all provided by the Linux kernel git repository. Git is the source code management system used by the Linux kernel, and makes available all of this information. The feature space shows the processing pipeline of the features extracted from data using the Coccinelle C-code matching and transformation engine [START_REF] Padioleau | Documenting and automating collateral evolutions in Linux device drivers[END_REF]. The classification category contains the log label, representing the name of the logging function and the log severity level. The log label is also extracted from the data using Coccinelle. Finally, the circle in the lower right of Figure 6 represents the model being trained with a set of features and a label. We use the J48 decision tree algorithm provided by Weka [START_REF] Hall | The WEKA data mining software: an update[END_REF] for training the model.

B. Finding the logging functions

Choosing a label for a given code fragment in the training phase relies critically on knowing whether the code fragment includes logging code. Zhu et al. [START_REF] Zhu | Learning to log: Helping developers make informed logging decisions[END_REF] in their work on userlevel C# code have identified logging functions by searching for some relevant keywords in method names. This, however, seems potentially insufficient for the wide variety of logging functions used in the Linux kernel. A possible alternative solution would be to unfold function and macro definitions for calls that contains string parameters, and verify the presence of calls to basic printing functions such as printk. We have found, however, that non-logging functions may also contain calls to printk-like functions, and thus this approach may result in false positives. We thus decided to rely on properties of call sites rather than names or function and macro definitions for finding logging functions.

Our methodology for finding logging functions combines three semantic patterns. The first is that the function should be called at least once on an if branch ending in a return statement, representing a typical value-error-check scenario. However, many other functions such as resource release functions and recovery operations are called in the same way, making this criterion not sufficient. The second pattern is that the function should have at least one argument that is a string constant, which we expect will represent the message string. Again, this pattern is not exclusive to logging functions; for example, we have observed that functions like strcmp (string comparison) may also be used according to this pattern. The third pattern is that the function should have a variable number of arguments, which is common to printf like functions, but is also not exclusive to logging functions. We found that none of these patterns alone produces a list with a low number of false positives, but the combination of the three produces very good results. We used Coccinelle [START_REF] Padioleau | Documenting and automating collateral evolutions in Linux device drivers[END_REF] to describe the patterns, and to extract the function names.

C. Features

In this section we will describe our feature extraction and processing pipeline which is used during the model training and by our tool.

Zhu et al. [START_REF] Zhu | Learning to log: Helping developers make informed logging decisions[END_REF] identified important factors for determining where to log in large user space software written in the object-oriented language C#. They identified three categories of source-code features representing the attributes of error paths that are relevant for logging: structural features, textual features, and syntactic features. Structural features refer to the structure of the source code, including the error type, or the exception type, and the methods and functions called. Textual features represent the code as flat text attributes for classification using a bag-of-words model in which the features are the frequency of occurrence of each word. Syntactic features capture eight semantic patterns from the source code, such as the number of called methods and functions.

The Linux kernel is written in the C language, which is not object-oriented, but it makes heavy use of object-oriented concepts such as classes and methods, implemented mostly through the use of structures and pointers. This has made it possible to create a mapping between the features defined by Zhu et al. [START_REF] Zhu | Learning to log: Helping developers make informed logging decisions[END_REF] and the features that are found in the Linux kernel. Many of Zhu et al.'s features could be adapted directly, such as using functions instead of methods, but we added a feature named entry point, and we removed a feature named EmptyCatchBlock.

An entry point is a pointer to a function associated with an event. The Linux kernel activates the entry point in response to a known event such as the user requesting a file open operation. Entry points are widely used in the Linux kernel, and all functionalities provided by modules such as device drivers and file systems are registered in the Linux kernel through the declaration of entry points. We added entry points to the structural and textual features because knowing the entry points that lead to a particular fragment of code, potentially interprocedurally, is essential to understanding the code fragment's functionality. In the case of EmptyCatchBlock, another difference is the lack of recognizable try-catch-throw exception handling in the Linux kernel and consequently the lack of empty catch blocks as a particular feature. We use Coccinelle to extract features from the source code.

We apply a set of feature selection and noise handling techniques before training the model to increase the accuracy of the suggestions of our tool. As our approach is similar to the approach used by Zhu et al. [START_REF] Zhu | Learning to log: Helping developers make informed logging decisions[END_REF], we do not include a detailed description of this process due to space limitations.

D. Labels

The label represents the knowledge that the model is learning: where to log, which logging function to use and which logging severity label to use. The label plays a somewhat different role when training the model, where it is part of the training data, and when using the tool, where it is the result produced by the model.

When training the model, the label is chosen based on the presence of a call to a logging function in the data. Our training framework infers the label automatically, based on the set of logging functions identified as described in Section III-B. For the cases in which there is no call to a logging function, the label is not logged. For the cases in which a call to a logging function is present, the label encapsulates the function name and the logging severity level. The logging severity level is determined by following the call graph of the logging function until reaching a call to a logging function that explicitly includes a log level, such as KERN_WARNING.

When using the tool, as illustrated in Figure 5, only the features are used to query the machine learning model. The model then produces a label that represents the correct logging decision for that set of features. However, a label is also extracted from the data, as done in the training phase, to identify the current state of logging in the code. This extracted label is then compared to the label inferred by the model. If the labels are identical, it means that the logging strategy taken by the patch or file is already correct. If they differ, however, it means that the logging performed by the patch or file can be improved, and we are designing our tool to emit a fix suggestion.

IV. THREATS TO VALIDITY

In the current state of our work, the main threat to validity is an internal threat to validity, in the correctness of our strategy for identifying logging functions. This identification strategy determines how our model is trained, and thus the correctness of its results. To address this threat, we have carefully studied the list of identified logging functions, and cross checked it against the results of other more approximate strategies, such as searching for certain kinds of substrings in function names, to ensure that no obvious candidates are overlooked.

V. RELATED WORK

The work most closely related to ours is that of Zhu et al. [START_REF] Zhu | Learning to log: Helping developers make informed logging decisions[END_REF] on providing logging suggestions for C# code that we have already described extensively. ErrLog [START_REF] Yuan | Be conservative: Enhancing failure diagnosis with proactive logging[END_REF] is a tool with similar goals that has been applied to various userlevel infrastructure software and that tries to recognize error conditions and ensure that all are logged. Other tools that have been developed around logging include an empirical study of logging practices in open-source software [START_REF] Yuan | Characterizing logging practices in open-source software[END_REF], LogEnhancer that extends existing log messages with relevant information [START_REF] Yuan | Improving software diagnosability via log enhancement[END_REF], and SherLog for helping developers effectively use information in log messages to guide debugging [START_REF] Yuan | SherLog: error diagnosis by connecting clues from run-time logs[END_REF].

Various machine-learning approaches have been applied to improving the quality of Linux kernel code, including the work of Engler et al. [START_REF] Engler | Bugs as inconsistent behavior: A general approach to inferring errors in systems code[END_REF] on identifying bugs from deviant code patterns, and an extension of that work by Li and Zhou [START_REF] Li | PR-Miner: automatically extracting implicit programming rules and detecting violations in large software code[END_REF]. Our work, in contrast, specifically targets logging, and uses specific properties of logging code.

VI. CONCLUSION

In this paper, we have identified the unique features and challenges of correct logging at the operating system kernel level, exemplified by the Linux kernel, and we have proposed a machine-learning based methodology for helping developers make correct logging decisions. In the future, we plan to complete and implement our methodology, and evaluate our approach on the recent history of Linux kernel code.

1 static

 1 Fig. 1. Logging causing performance problems

Fig. 4 .

 4 Fig. 4. Number of calls to logging functions per SLOC

Figure 6

 6 Figure 6 shows our feature and label extraction framework, which represents a typical supervised machine learning

Fig. 6 .

 6 Fig. 6. Feature and label extraction framework

  2) of the calls are to printk or the generic API functions, while the other half (48.16% in Linux 4.2) are to

	printk	new api		others	total
	0.02
	0.01
	0.00
	v 2 .6 .1 2 v 2 .6 .1 7 v 2 .6 .2 3 v 2 .6 .2 9 v 2 .6 .3 2 v 2 .6 .3 5 v 2 .6 .3 8 v 3 .1	v 3 .4	v 3 .7	v 3 .1 0	v 3 .1 3	v 3 .1 6	v 3 .1 9	v 4 .2


Ripon K Saha

Julia Lawall
email: julia.lawall@lip6.fr

Sarfraz Khurshid

Dewayne E Perry
email: perry@ece.utexas.edu

Are These Bugs Really

Keywords: Bug Severity, Bug Tracking System, Mining Software Repositories

come

I. INTRODUCTION

Bug tracking systems are among the most frequently used resources for research in mining software repositories [START_REF] Guo | Not my bug! and other reasons for software bug report reassignments[END_REF]- [START_REF] Bettenburg | What makes a good bug report?[END_REF]. They are also often used in developing new techniques for automated software engineering such as automatic bug triaging [START_REF] Bortis | Porchlight: A tag-based approach to bug triaging[END_REF], bug assignment [START_REF] Anvik | Who should fix this bug?[END_REF], bug-fix time prediction [START_REF] Giger | Predicting the fix time of bugs[END_REF], [START_REF] Zhang | Predicting bug-fixing time: an empirical study of commercial software projects[END_REF], severity prediction [START_REF] Lamkanfi | Predicting the severity of a reported bug[END_REF], bug prioritization [START_REF] Tian | Drone: Predicting priority of reported bugs by multi-factor analysis[END_REF], and bug localization [START_REF] Saha | Improving bug localization using structured information retrieval[END_REF], [START_REF] Zhou | Where should the bugs be fixed?more accurate information retrieval-based bug localization based on bug reports[END_REF]. A critical element of much of this work is understanding the importance of the bug reports found in these bug tracking systems. As this information is difficult to accurately infer, and may depend on the priorities and point of view of the bug reporter, studies typically rely on the "severity" label provided in the bug report [START_REF] Saha | An empirical study of long lived bugs[END_REF]. While the labels vary by project, they typically amount to some variant of Severe, Normal, and Minor.

In many bug tracking systems, Normal is provided as the default. This may raise questions about the validity of a Normal severity label. Indeed, the person who files a bug report may be an ordinary user who has no expertise in the implementation of the affected software, or even no technical expertise at all. Such a person may find it difficult to accurately assess the severity of a bug. Thus, the bug reporter might not fill in the severity field, leaving it at its default Normal value. As a result, studies that use the severity field to investigate if there exists any relationship between bug severity and factors such as bug-fix time, amount of discussion, etc. are open to criticism that the results found in the Normal case may be invalid, as Normal may not reflect the actual severity. Simply excluding the Normal reports, however, may distort the results in the opposite direction, if the Normal reports represent a large percentage of the available data.

These issues have been highlighted in a number of research studies. For example, in their two studies of severity prediction, Lamkanfi et al. [START_REF] Lamkanfi | Predicting the severity of a reported bug[END_REF], [START_REF] Lamkanfi | Comparing mining algorithms for predicting the severity of a reported bug[END_REF] excluded all the normal bugs stating: "In our case, the normal severity is deliberately not taken into account. First of all because they represent the grey zone, hence might confuse the classifier. But more importantly, because in the cases we investigated this "normal" severity was the default option for selecting the severity when reporting a bug and we suspected that many reporters just did not bother to consciously assess the bug severity". Similarly, Tian et al. [START_REF] Tian | Information retrieval based nearest neighbor classification for fine-grained bug severity prediction[END_REF] excluded Normal bugs stating: "Following the work of Lamkanfi et al., we do not consider the severity label normal as this is the default option". Along the same lines, in the submission of our previous work on "long lived bugs" [START_REF] Saha | An empirical study of long lived bugs[END_REF], when we drew our conclusions that most long lived bugs were important and adversely affected users' normal working experience, all three reviewers expressed their concerns: Reviewer 1: "since you used data from the severity field, I would suggest to discuss the fact that the level of this field could be somewhat subjective." Reviewer 2: "In most cases, 'Normal' is the 'default' value of the severity, thus most of the users reporting a bug leave the default value since they simply don't know or are not interested in precisely defining the value." Reviewer 3: "Firstly we have to agree with the finding that eclipse severity is meaningful. If it is then you can cite other work that shows it to be meaningful, otherwise this claim does not hold up."

A researcher is thus faced with a dilemma: either include information that may be unreliable, or discard potentially valuable information. To the best our knowledge no study has investigated either the amount of noise in the severity data (except w.r.t. enhancements [START_REF] Herzig | It's not a bug, it's a feature: How misclassification impacts bug prediction[END_REF], [START_REF] Antoniol | Is it a bug or an enhancement?: a text-based approach to classify change requests[END_REF]) or the amount of value in this information.

In this paper, to better understand how severity information can be used, we investigate the following hypotheses, summarizing the apparent current consensus, as reflected by the above citations: H1: Normal bugs do not reflect the actual severity level.

H2: Bug reports are mislabeled as Normal because reporters do not bother to change the default (Normal) severity. Furthermore, we investigate the reasons for these problems and their impact in a representative software-engineering application. Our analysis is carried out in the context of four systems from the Eclipse product family. These are open source systems, that have publicly available bug databases, and that have been used in a number of previous software engineering studies [START_REF] Lamkanfi | Predicting the severity of a reported bug[END_REF], [START_REF] Saha | An empirical study of long lived bugs[END_REF]- [START_REF] Tian | Information retrieval based nearest neighbor classification for fine-grained bug severity prediction[END_REF]. Our findings indicate that:

• Around 80% of the bugs reported in the studied software projects are classified as Normal. Excluding them from any automatic software engineering techniques could substantially distort the results. • A manual reclassification of 500 Normal bugs in the studied software projects by pairs of students showed that 65% of the Normal bugs are not normal. Indeed, almost 25% of the Normal bugs are severe. These results support Hypothesis 1. • Contradicting Hypothesis 2, we find the main reason for misclassifications in the Normal bugs is not that it is the default severity level. Rather, this field is very subjective and thus users may follow different criteria. Indeed, the pairs of students provided different opinions for more than half of the Normal bugs. We provide a taxonomy of the most common rationales used in these dissimilar assessments.

• The presence or absence of Normal bugs in training and test sets can significantly affect the actual and measured effectiveness of automatic software engineering techniques that rely on bug severity information. In our experiment with a basic bug severity predictor, we find that misclassification in the training data can reduce the accuracy of the severity prediction considerably. On the other hand, a tool accuracy excluding Normal bugs from both training and testing data is likely to be an overestimation if the tool is intended to be used on unlabeled data containing Normal bugs. We conclude that while the classification of Normal reports is not very accurate, excluding them from software engineering studies can significantly distort the results.

The rest of this paper is organized as follows. Section II describes bug tracking systems and the various relevant features that they provide. Section III presents our research questions and our dataset. Sections IV through VII consider our research questions. Finally, Section VIII analyzes threats to validity, Section IX presents related work, and Section X concludes.

II. BACKGROUND

We first briefly present the notion of a bug tracking system, and the levels of severity that such systems commonly use.

A. Bug Tracking System

Generally project stakeholders maintain a bug database for tracking the bugs found in their projects. A bug database may collect bug reports from developers, testers, or ordinary users, according to the policies of the project. Widely used online bug tracking systems include Bugzilla, JIRA, and Mantis. 1 Different bug tracking systems may have different data structures and follow different life cycles of bugs. In this paper, we focus on data extracted from Bugzilla.

Any person who has a Bugzilla account for a given project can post a change request. A change request could be either a bug report or a request for a feature enhancement. In Bugzilla, both are represented similarly and are referred to as "bugs," with the exception that for a feature enhancement, the severity field is set to "enhancement". Generally, the bug reporter provides a bug summary, a bug description, the names of the suspected product and component, and an indication of the bug's severity. The bug reporter also specifies the software version, the platform and operating system where the bug was encountered, so that developers can easily reproduce the bug.

B. Bug Severity

Each project that uses Bugzilla can define its own severity levels. Since we study the projects from the Eclipse product family, we discuss the severity levels defined by Eclipse community. According to Eclipse Bugzilla documentation,2 the severity level can be one of the following: Blocker, Critical, Major, Normal, Minor, Trivial, or Enhancement. These values are intended to describe the impact of the reported bug on the operation of the software. The definitions of these values provided by the Eclipse documentation are given in Table I.

At the start of the bug fixing process, each project or component team leader triages NEW bug reports to determine whether the bug is really a bug and if the provided information is correct. In case of any inconsistencies, the bug triager can correct them or request more information from the person who originated the report. For example, during triaging, a bug may be moved to another component/product or the triager can adjust the severity level. The developer who fixes a bug can also adjust the severity level. If the severity field is changed, the Bugzilla report contains the history of the assigned values.

III. STUDY SETUP

A. Research Questions

Our study investigates the following research questions: RQ1. What proportion of the bugs are Normal in the bug repository?

Motivation: The first question of any empirical study is how large is the population that we want to study. Indeed, if the population is small, there may be little reason to worry about it. For this study, our population of interest is the set of bug reports having the severity level Normal. RQ2. What proportion of bug reports classified as Normal are actually "normal"?

Motivation: This is one of the main research questions of our study. The fewer bug reports classified as Normal that are actually "normal," the more effect this will have on the validity of any study that somehow depends on the bug report severity classification. This research question also addresses Hypothesis 1: Normal bugs may not represent the actual severity level.

RQ3. What are the main sources of misclassifications?

Motivation: To reduce misclassifications, we first need to understand the reasons behind it. Delineating the common reasons for misclassifications will help researchers or practitioners deal with the problem more systematically. This research question also addresses Hypothesis 2: Bug reports are mislabeled as Normal because reporters do not bother to change the default (Normal) severity level.

RQ4. Can misclassification or exclusion of Normal bugs affect previous study results?

Motivation: We investigate whether the noise in Normal bugs can affect tool results. If there is no impact, we would have little reason to worry about the issue.

B. Subject Systems

Our study focuses on four open-source projects, JDT, CDT, PDE, and Platform, from the Eclipse product family. 3 JDT and CDT provide an Integrated Development Environment based on the Eclipse platform for developing Java applications and for developing C and C++ applications, respectively. The Plug-in Development Environment (PDE) provides tools to create, develop, test, debug, build and deploy Eclipse plugins, fragments, features, update sites and Rich Client Platform (RCP) products. Finally, the Eclipse Platform defines the set of frameworks and common services that make up the infrastructure required to support the use of Eclipse. These projects are widely used in the real world, and have also been extensively used in software engineering research [START_REF] Lamkanfi | Predicting the severity of a reported bug[END_REF], [START_REF] Saha | An empirical study of long lived bugs[END_REF], [START_REF] Lamkanfi | Comparing mining algorithms for predicting the severity of a reported bug[END_REF]. Furthermore, although these projects belong to the same product family, they are from various domains.

We have used Lamkanfi et al.'s [START_REF] Lamkanfi | The Eclipse and Mozilla defect tracking dataset: a genuine dataset for mining bug information[END_REF] bug dataset, obtained from the Eclipse Bugzilla database, 4 to obtain the bug information associated with these projects. This dataset includes all the bug reports and their histories from the project inception to March 2011 for these four projects. Table II describes the dataset in more detail. Although this dataset is few years old, it was part of the MSR data showcase in 2013 and similar datasets have been used in many studies [START_REF] Lamkanfi | Predicting the severity of a reported bug[END_REF], [START_REF] Saha | An empirical study of long lived bugs[END_REF]- [START_REF] Tian | Information retrieval based nearest neighbor classification for fine-grained bug severity prediction[END_REF], [START_REF] Murgia | Predicting bug-fixing time using bug change history[END_REF] IV. PROPORTION OF NORMAL BUGS

In this section, we investigate our first research question: What proportion of bugs have Normal severity level?

A. Methodology

A straightforward methodology would be to just compare the number of reports labelled Normal with the total number of reports. However, a bug tracking system may contain many invalid and duplicate bug reports, as well as feature requests or enhancements. There are also some bug reports that developers think are not worth fixing. In Bugzilla, the status and resolution fields together keep track of the current status of each bug. More specifically, the status field holds at most one of the values: UNCONFIRMED, CONFIRMED, IN_PROGESS, RESOLVED, and VERIFIED. The resolution field holds at most one of the values: FIXED, INVALID, WONTFIX, DUPLICATE, and WORKSFORME. Using this information, we extracted all the unique and valid bug reports, specifically those whose status field was set to either RESOLVED or VERIFIED and the resolution field was set to FIXED. We note that, as our bug reports date from 2011 at the latest, almost all of the reports have either been classified as uninteresting (INVALID, WONTFIX, DUPLICATE, or WORKSFORME) or have been fixed. We also removed all the reports marked as enhancements. We use the resulting set of reports in this and all of the subsequent research questions. Then, we counted all the bugs for each severity level.

B. Results

Table III provides detailed results regarding severity. For all of the considered systems, Normal is the dominant severity category, with 78-82% of the bug reports. The next most dominant category is Major, representing only 8-10% of the reports. Blocker bugs are rarest, at around 1%. The proportions of other types of bugs (Critical, Minor, and Trivial) are between 2% and 4% in most cases. Our results suggest that any research based on bug severity that ignores Normal bugs faces a severe threat to validity, since a large percentage of bug reports would likely not be taken into account. V. ACTUAL SEVERITY OF "NORMAL"-LABELED BUGS

In the previous section, we saw that a large proportion of bugs are classified as Normal. However, we do not yet know if these Normal bugs are really normal according to the Eclipse Bugzilla definition. In this section, we investigate the second research question: What proportion of bugs having Normal severity level is actually normal?

A. Methodology

Since to the best of our knowledge, there is no clean dataset of bug reports that have actual severity levels, classifying bug reports using automated machine learning techniques would likely be inaccurate. Therefore, we conduct a manual investigation of their actual severity. Our methodology takes into account the fact that bug severity may be subjective, as well as the high cost of doing such an analysis.

1) Design: The severity field represents the impact of a given bug on the operation of the software, and thus it may be subjective. Indeed, even the Eclipse documentation mentions that the bug reporter's perspective on the severity can depend on how the bug reporter wants to use the software. 5 Therefore, to get reliable results, we have each bug report assessed by multiple users. In such a study, the cost depends on two factors: 1) the number of bug reports to be assessed, and 2) how many assessments are made.

To keep the cost reasonable, we made two decisions. First, we randomly selected a sample of 500 bug reports labelled Normal, representing 125 bug reports from each project, from within the last five years of our dataset, i.e., from 2006 to 2011. Second, we recruited a group of assessors, such that each report would have at least two assessors. If the two assessors had different opinion about a given bug report, we analyzed both the report and the assessments to make a decision.

2) Users/Assessors Selection: All the assessors in our study are either graduate or undergraduate students of the University of Texas at Austin. We sent a general email to all the students in the senior "Software Engineering" class and to some graduate students in the software engineering track of the Electrical and Computer Engineering department. Good programming knowledge and substantial experience in working with the Eclipse IDE were requirements to participate in the study. Based on these criteria, we selected 6 graduate students and 4 senior undergraduate students. 9 of them have experience in industry either as an intern or as a full-time programmer. Table IV gives the students' qualifications in more detail.

3) Procedure: Our study was divided into two sessions: training and assessment. In the training session, we conducted a 30-minute tutorial. The tutorial included:

1) Providing a brief overview of our study, 2) Explaining a real Eclipse bug report and giving a brief overview of Bugzilla, 3) Showing how to submit a bug report in Eclipse, to show that Normal is the default severity level, 4) Explaining the definition of each severity level from the Eclipse documentation (Table I), 5) Showing a representative example in each severity level to deepen their understanding about bug severity, 6) Explaining the structure of the expected feedback.

We divided the 500 bug reports into five sets of 100 bug reports each. Sets 1-4 had 100 bug reports from the same project and Set 5 had 25 bug reports from each of the four projects. This strategy allowed most of the students to focus on a single project. We then assigned the students to five groups, pairing a graduate student with an undergraduate, when possible. The group members were not informed of each other's identity. Then we assigned each set of bug reports to a group randomly. The students were given 10 days to complete the assessments. We recommended to the students that they carefully read each part of the bug report, including at least the bug summary, the bug description, and the developers' comments, to make their decision. All of the students completed their task. After the study, each student was rewarded with a $50 Amazon Gift Card.

We note that the students have access to more information than the original bug reporter, as the students have access to the comments that were added after the original bug report was made. Our goal, however, is not to simulate the conditions under which bugs are reported, but to obtain accurate severity information. Furthermore, in the total of 1,000 assessments produced by the students, the students used the bug summary, bug description, and comments in 68%, 62%, and 39% of the cases, respectively. For only 15% of the bug reports did both students report that the comment information was helpful to make a decision, and for only 2% of the bug reports did the students only use the comments to make a decision. Therefore, for most bug reports, the students were able to make a decision based on the information available to the original reporter. 4) Feedback: We designed a Google form to receive students' feedback. 6 Our form contains: 1) the bug id, 2) the actual severity of the bug, 3) the specific reason for the decision (free text), 4) the parts (summary, description, and comments) of the bug report that helped make a decision, and 5) the assessor's name. All of the fields were required to submit a response. Students could provide "Not Sure" for the actual severity if they really were not sure about it.

B. Results

We got 1000 responses from the students, comprising two responses for each bug report. There were only 16 responses where at least one student was undecided and thus chose "Not Sure". For only one bug report, from the Platform project, were both responses "Not Sure" due to insufficient information. We investigated all these 16 bug reports and were able to assign a severity level in 15 cases. However, we were not able to assign any severity level to the bug report where both responses were not "Not Sure". We eliminated this report, leaving the responses for 499 bug reports for analysis.

Among the 500 bug reports, there were only 164 reports, i.e., 33%, for which both students gave the same severity level. To refine the results, we focused on the difference between Normal and the other severity levels. Like other work [START_REF] Lamkanfi | Predicting the severity of a reported bug[END_REF], [START_REF] Lamkanfi | Comparing mining algorithms for predicting the severity of a reported bug[END_REF], we merged the Blocker, Critical, and Major categories into a higher-level category, Severe, and the Minor and Trivial categories into a higher-level category, Non-Severe (NS). We refer to two responses that are in the same higherlevel category as similar. After merging the categories, as shown in Table V, we obtain 210 similar responses, amounting to 42% of the reports, leaving 289 where the students disagree. For Platform, the proportion of similar responses is only 35%. The highest rate of agreement (50%) is found for CDT. These results confirm that severity is highly subjective.

Given the high rate of dissimilarity among the severity levels provided by the students, we cannot use the results directly to obtain the proportion of Normal bug reports that are actually normal. Instead, we consider the results from three perspectives: best case, worst case, and optimal. For the best case, i.e., the most optimistic view of the state of the software, we take the lowest level of severity between the two students' responses. For example, if one student categorized a bug as Major and another categorized it as Normal, we would consider the actual severity to be Normal. On the other hand, for the worst case, i.e., the most pessimistic view of the state of the software, we take the highest level of severity between the two responses. Finally, for optimal case, we investigated all the 289 bug reports where students' responses differed and tried to come to a consensus. We read all the bug reports (summary, description, and comments) and students' responses (assigned severity level and specific reason for assigning that level). Then, we made a decision by either taking the one of the students' responses that we agreed with, which was possible in most cases, or assigned a different severity level.

Tables VI and VII present the proportion of Normal reports that are classified by the students into the different high-level categories for the best, worst, and optimal cases. In all cases, the Enhancement columns are the same; since enhancements are not bugs, in both tables we use the optimal-case results. From the results, we can see that the proportion of Severe, Normal, and Non-Severe bugs could vary between 7%-34%, 32%-40%, and 15%-34% respectively, depending on how the results are calculated. More specifically, from Table VII, representing the optimal classification, we see that the actual proportion of Normal bugs among those originally labelled Normal is only 35%, and that 19% of the reports originally labelled as Normal are not reports of bugs at all. Furthermore, among the bugs originally labelled Normal, 24% are Severe and 22% are Non-Severe. For JDT, the proportion of Severe bugs is even higher, 33%. Among the 109 reports that are Non-Severe, 84 are Minor and only 25 are Trivial. Therefore, our overall results suggest that the dataset of Normal bugs has serious noise, validating Hypothesis 1: Normal bugs may not represent the actual severity level.

VI. SOURCES OF MISCLASSIFICATION

In the previous section, we showed that 65% of Normal bugs are not actually normal according to the definition of Eclipse severity levels. There may be numerous reasons for these misclassifications, from "leaving the severity field at its default value" to "too subjective to decide". In this section, we answer RQ3: What are the main sources of misclassifications?

A. Methodology

To understand the reasons for misclassifications, first we investigate the main severity levels that confused assessors. Then, we read all the bug reports where students' responses differed by high-level category (Severe, Normal, or Non-Severe). In almost all of the cases, it was possible to determine why the student chose a particular severity level by reading the reasons the student provided.

To categorize the common reasons for different responses, we analyzed all the bug reports, following an open-ended taxonomy. For a given bug report, first we identified the highlevel reason for the difference, and then we checked whether the reason fits into any of the existing categories. If it did not, we created a new category. We provide concrete examples for each category below, to better understand the categorization procedure. We selected examples that: i) cover the range of subject systems, and ii) have a summary or description in the bug report that is concise enough to present in the paper.

B. Results

Table VIII shows the number of bug reports for each pair of dissimilar responses. For example, the value in the rightmost cell indicates that for 13 bug reports, one student's response is Trivial but the other's response is Enhancement. From the results we see that students were mostly confused between the Normal and Critical, Normal and Major, and Minor and Normal severity levels. Interestingly, Normal bugs are present in each confusion pair. Therefore, it is evident that even after careful assessment, users can be confused between Normal and other categories. The following summarizes our taxonomy of common reasons for the different responses.

1) Focusing on different aspects of a bug report: A bug report may describe several aspects of a bug. Different persons can focus on different aspects, causing them to map the bug to different severity levels. Consider the following example:

Platform # 210946, Description: A caught Throwable is not written to the Eclipse log. It is just written on the console.

One student thought that this bug is Severe since it involves losing data from the Eclipse log. Their rationale was that since information is normally written to both the console and the log, the user may close the console and rely only on the log file. Such a user would not even realize that some data were missing. On the other hand, the other student assigned a severity level Minor thinking that there is an easy workaround, since the Throwable is at least written on the console.

2) Same aspect but different perception: In some bug reports both students focused on the same aspect of the bug but their perception of it was different. For example:

CDT # 332915, Summary: [tracepoint] Refreshing the Trace Control view blocks the UI thread.

Bug Description: We've noticed that when heavily using the tracepoint interface, deadlocks can happen due to the UI thread being blocked. Once [sic] case is that the refresh operation of the Trace Control view is done within a Query, which locks the UI thread.

One student responded, "One feature enabled ends up impeding another feature -even though both features work in isolation." Thus, she chose Severe. The second student responded, "GUI and refresh are on the same thread". Thus, it is a regular issue and the student assigned a Normal severity.

3) Different acceptance or tolerance level: Sometimes users may have the same perception about the problem but a different level of tolerance to deal with it. For example:

Platform # 172321, Summary[Commands] [GTK] Handler activation in editor when a dialog is closed is delayed One student thought that this is a "loss of functionality (delay time of feature) only on linux platform" and thus tagged it as Normal. The other student responded, "delay issue for activating handler is a major issue to me". So he chose Major.

4) Impact: Some bugs can seem to fall into the category Minor if the definitions of severity levels are strictly followed. However, the impact of the bug can be annoying enough to make the bug Normal or even Severe. For example:

JDT # 231887, Summary: [actions] cannot refresh working sets through Package Explorer Bug Description: Steps To Reproduce: 1. Import some Java projects and put them into some working sets. Change the Top Level Elements in the Package Explorer to Working Sets 2. Externally modify some of the files from different working sets 3. Select the working sets in the Package Explorer, right-click, and choose Refresh. Nothing happens. (Verify by opening files that have been modified -instead of opening the file, you get the "This file is out of sync" editor) 4. If you expand all the working sets and refresh the individual projects, it works.

From the bug summary and description we can see that the user has provided a workaround. However, every time the user changes something, he must refresh each project related to the working set, which is annoying. Thus, one student marked the bug as Severe saying that it hinders the workflow. However, the other student responded, "Easy workaround. Not so much important bug." We also noticed these dissimilar responses when keyboard shortcuts do not work properly (e.g., Platform # 262593). Again, there is in principle an easy workaround, using menu commands, but some users may be so used to keyboard-shortcuts that they do not feel comfortable with the menu, causing them to view the bug as Severe.

5) Different cost of the same bug: development perspective vs. release perspective: This is the most frequent category in our sample, especially for those dissimilar responses where students are confused between Critical and another category. In our study, in most cases, when a report indicates a program crash, e.g., due to a null pointer exception, the students marked it Critical, which is appropriate according to the definition of the severity levels. Examples include:

JDT # 325523, Summary: NPE when deleting resource PDE # 275921, Summary: NPE with update classpath However, in some cases, the students analyzed the bug from a developer's perspective and marked it as Normal. For example, they said that this was an easy fix or occurred in infrequent cases. Indeed, when an exception is thrown during development and testing, this can be considered as normal since this is a common mistake that developers can fix quickly. However, if a stable version crashes for a given task and the user must wait for the next update to get it resolved, the effect is a lot costlier than the development scenario.

6) Bug vs. Enhancement: Whether a given issue is a bug or an enhancement is subjective, and is thus itself a reason for misclassification. Furthermore, even if a reporter correctly classifies an enhancement, s/he may end up representing it incorrectly in the Bugzilla database due to the tricky Bugzilla configuration used by Eclipse. Indeed, in the configuration of Bugzilla used by the Eclipse projects there is no separate field for distinguishing bugs from enhancements. Instead, Enhancement is just one possible bug severity level. Therefore, if a change request is an enhancement, the reporter should set the severity label to Enhancement regardless of the request's importance. In practice, however, we found many cases where reporters marked enhancements as Major, Normal, or Minor depending on their perceived importance, thus implicitly misclassifying the change request as a Bug, not as an Enhancement. We discuss some examples:

Platform # 185067, Summary: [KeyBindings] New Keys pref page: cannot sort 'User' column PDE # 330943, Description: [plug-in registry] View initialization takes too much time

In the first example, the issue reporter is asking for sort functionality to be added to the "new keys" preference page. Since what is asked for is a new feature, it is an enhancement, not a bug, even if the reporter finds it inconvenient or inconsistent that the feature is not currently available. Likewise, in the second example, there is no error in "View initialization." Instead, the reporter is requesting that the performance be improved. However, in the students' assessment, one response for each bug was Minor, since the students thought that these issues would not affect users much.

C. Discussion for Hypothesis 2

We now investigate the existing Hypothesis 2, whether "leaving the severity field at its default value" is the main reason for severity misclassification. For this, we try to infer possible motivations from the experiences of our student assessors and from the bug report characteristics themselves.

Experiences of the student assessors: All the 500 bug reports in our manual investigation are marked as Normal in the Eclipse Bugzilla. However, for 65% of these reports, the student assessors had a different opinion of the severity from the original labeller. We try to understand why differences of opinion can occur by studying the differences of opinion that occurred within our manual study. Specifically, for 58% of the reports, each of the students evaluating the report assigned it a severity in a different higher-level category (e.g., severe, normal or NS). Careful investigation into the students' responses reveal that most of these discrepancies were due to the subjective nature of the assessment. There are indeed many factors to consider, such as minor or major loss of functionality, frequent or infrequent use cases, the convenience of any workaround, etc. Each of these factors is itself subjective, and our analysis in Section VI-B shows that different choices by the students often resulted from their putting different weights on these different subjective criteria. As the students were told to pay careful attention to the evidence available in choosing a bug severity, but still often came up with different labels, it is possible that the original reporters were also paying attention to the choice of severity, but made choices that were different than those of the students.

Furthermore, we observe that our optimal strategy classifies only 3 of the selected 500 Normal reports as Blocker and only 25 of these reports as Trivial, implying that most of the differences of opinion between the original reporter and the students was among the severity levels closer to Normal. These differences are again likely to be more subjective.

Report characteristics: We also performed a simple automatic analysis on all the fixed bug reports to get an idea about the proportion of bugs that should be Trivial but are categorized as Normal. We found that many of the reports classified as Trivial by our optimal strategy contain the keywords typo, spell, and documentation, either in the report or in the students' comment, and that these words appear rarely in the other reports in our sample. Text search of the summary and description of all the fixed Normal bugs in the complete dataset showed that only 1% of such bug reports contain these words. Again, we find little overlap between levels that are far apart, suggesting that misclassifications are between similar categories for which the differences are more subjective.

VII. MISCLASSIFICATION OR EXCLUSION OF NORMAL BUGS: DO THEY MATTER?

In this section, we investigate RQ4: Can misclassification or exclusion of Normal bugs affect previous study results?

A. Methodology

As we already noted, many previous studies use the bug severity field as a feature in various techniques such as bugfix time prediction, modeling bug report quality, severity prediction, etc. A number of previous studies have ignored Normal bug reports, on the assumption that Normal does not correctly reflect the severity level. In this study, we have confirmed this assumption. However, a tool that has not been trained on Normal bugs may subsequently give meaningless results on Normal input. Such a tool is thus unusable in a real-world setting unless accurate severity information is already available. Therefore, we investigate two phenomena: 1) whether there can be any effect of misclassification on previous study results, and 2) whether there can be any impact on the results if the Normal bugs are eliminated from the study.

To investigate the impact of these phenomena, we chose bug severity prediction as a representative application. Generally a bug severity prediction algorithm takes a set of bug reports known to be from various categories (e.g., Severe and Non-Severe) as training data and uses the properties inferred from this training data to predict the severity of bug reports in a test set. Lamkanfi et al. [START_REF] Lamkanfi | Predicting the severity of a reported bug[END_REF] showed that taking into account bug summaries is sufficient to get accurate results. Since our objective is to investigate the effect of misclassification or exclusion of Normal bugs on accuracy, not to propose new techniques for bug severity prediction, we have just implemented a simple approach. Our bug severity prediction system takes a set of bug summaries labelled with severity as a training set and predicts the severity of an input report represented by its summary. Our predictions are coarse-grained: Severe, Normal, and Non-Severe. We use Mallet's implementation of Naive Bayes out-of-the-box as our underlying classifier. Then we measure accuracy in terms of the proportion of correctly classified items. Specifically, the accuracy of our classifier is m/n * 100% if it classifies m instances correctly out of n instances. For a comprehensive description of bug severity prediction, please see [START_REF] Lamkanfi | Predicting the severity of a reported bug[END_REF], [START_REF] Lamkanfi | Comparing mining algorithms for predicting the severity of a reported bug[END_REF], [START_REF] Tian | Information retrieval based nearest neighbor classification for fine-grained bug severity prediction[END_REF].

Training and Test Set Creation: We distinguish between a clean dataset, in which we have good confidence that the severity labels are accurate, and a noisy dataset, in which it is not known whether the severity labels are accurate. Either kind of dataset furthermore may or may not contain Normal bugs. This leads to four training sets, TR clean , TR noisy , TR clean -Normal , and TR noisy -Normal , having various permutations of these properties. We train our severity prediction algorithm on each of these training sets, resulting in four instances of the tool. To assess the impact of noisy data and of excluding Normal bugs, we then test each of these tool instances on a clean test set, TE , and compare the accuracy of the resulting predictions with the known labels.

A challenge in our experimental methodology is obtaining sufficient clean data. Indeed, our training and test sets must respect a number of constraints. First, the training set and the test set should be disjoint. Furthermore, previous research has shown that to avoid bias due to the over-prevalence of data in one class, all of the training sets (clean or otherwise) and the test set should have the same number of bugs at each severity level [START_REF] Japkowicz | Learning from imbalanced data sets: A comparison of various strategies[END_REF]. Specifically, if one class has few instances in the training set, any learning algorithm would know less about that class. Likewise, if a one class is very dominant in a test set, the evaluation result would be biased toward that class. For example, in a two-class (C A and C B ) test set , if one class (C A ) represents 90% of instances and if a naive classifier says all the instances are C A , its accuracy would be still 90%. Finally, Lamkanfi et al. [START_REF] Lamkanfi | Predicting the severity of a reported bug[END_REF] have found that a training set of 500 bug reports in each category gives stable results.

In addressing our previous research questions, we have manually investigated only 500 bug reports, and among them 120, 174, and 109 are classified as Severe, Normal, and Non-Severe, respectively. Using this dataset, and respecting the constraint that there should be the same number of bugs at each severity level, we can obtain a data set of at most only slightly over 300 elements. Concretely, we take the 100 most recent reports in each severity level, resulting in a dataset of 300 elements. As this does not satisfy the requirement of 500 reports in each category, we cannot use this as a training set. Thus, we use it as the test set, TE .

For the training sets, we need a low-cost way to obtain more data with reliable severity labels. For this, we focus on the bug reports in which the severity information has been changed at least once in any way. This may not result in a completely clean set, but it should be acceptable, since developers, who we assume to be experts in the software, have reviewed those bug reports and adjusted their severity level. Starting from the year 2006, we take the first 500 bug reports having this property in each severity category. For example, the training set of Normal bugs consists of those bug reports that ended up being Normal after one or more severity changes. The resulting set of 1500 reports makes up TR clean . To create TR noisy , we follow the same procedure, without the requirement of a change in the severity information. Then, from TR clean and TR noisy , we obtain TR clean -Normal and TR noisy -Normal , respectively, by removing the Normal reports. As we have taken the training data TR from the start of the time period and the test data TE from the end of the time period, they are likely disjoint. We have furthermore verified this in practice.

Finally, we note that all of the datasets contain bug reports from all four subject systems.

B. Results

We present our results in terms of the confusion matrix and accuracy. Precision, recall, and F-measure can all be calculated from the confusion matrix. Table IX shows the impact of misclassification on the accuracy of the severity predictor trained on the full clean dataset TR clean . In the results, each row is the number of predicted results for a given category. For example, the first row represents the 100 actual Severe bugs in TE . Of these, 57 are predicted to be Severe, 26 are predicted to be Normal, and 17 are predicted to be Non-Severe. Based on the actual severity level of bug reports in TE , the We next perform the same experiment, but where all Normal bugs have been removed from the clean training set, producing TR clean -Normal . Since the training dataset contains no Normal data, no bugs in the test set TE will be classified as Normal (see Table X) and the accuracy of the resulting classifier is only 47%. However, if we also exclude Normal bugs from TE , as done in previous studies, the accuracy increases to 71%. Therefore, the accuracy reported in the existing literature is likely overestimated if the tool is intended to be used on unlabeled data that may contain Normal bugs.

Next, we repeat both experiments for the case of noisy training data. First, we consider TR noisy , containing all categories of bugs. Table XI shows that we obtain an accuracy of 41% for TE with this training data. This value is 8% less than that obtained when we trained our classifier with TR clean . Therefore, misclassification in the training data can reduce the accuracy of the severity prediction considerably.

Next, we consider TR noisy -Normal , in which the reports labelled Normal have been removed. As compared to the use of the clean training set TR clean -Normal , the accuracy only slightly declines, from 47% to 45% (Table XII), which is less than the decline between the results obtained using TR clean and TR noisy . Furthermore, as compared to TR noisy , the accuracy actually improves, from 41% to 45%. This result

VIII. THREATS TO VALIDITY

Construct Validity: The set of bug reports is the only artifact used in our study; bug reports are generally well understood. We have also used well known metrics in our data analysis such as proportion and classification accuracy, which are straightforward to compute. We use a publicly available dataset, which enables the replication of this study. Therefore, we argue for a strong construct validity.

Internal validity: A bug report may be filed by either an Eclipse developer or a real user. The students who participated in our study are not involved in Eclipse development. However, they frequently use Eclipse for their own software development, e.g., research and class projects. Furthermore, 9 out of the 10 students had previous experience in industrial software development. Therefore, we believe that the students have the background necessary to assess bug severity.

All the bug reports used in our study are extracted from Bugzilla. There are many other bug tracking systems such as Jira, Mantis, etc. Since the severity levels may vary across projects and bug tracking systems, we may get different results for the bug reports in other bug tracking system.

To assess the severity of bugs, the students manually analyzed bug reports. To delineate the common reasons for misclassification, we also manually analyzed bug reports and students' responses. There might have been some unintentional misinterpretations during the manual verification due to the lack of domain knowledge. However, we held extensive discussions to minimize this threat.

To construct a clean training set, we selected bug reports whose severity had been changed at least once. Although we did not manually check that these bug reports have the actual level of severity, we believe the dataset should be fairly accurate since either bug triagers or developers examined those bug reports and adjusted the severity level accordingly.

External Validity: Eclipse may not be representative of all software. Still, it has been used in a number of studies, and so the conclusions drawn from it are at least applicable to those studies. Furthermore, we find similar results across the different Eclipse sub-projects.

Since manual investigation of bug reports is expensive, we investigated only 500 bug reports. We plan to conduct a more large-scale manual investigation in the future. Still, it has been shown that a sample of size 500 has suffient power to detect all but the smallest effects [START_REF] Rosenthal | Essentials of behavioral research: Methods and data analysis[END_REF]. In our manual investigation, each report was assessed by two students. More assessments may further increase the confidence in the results. However, we separately investigated all the dissimilar responses. Therefore, this threat should have little impact on our results.

IX. RELATED WORK

Our work is related to the study of bug reports, and more specifically bug severity. We review some recent work that has relied on this information. Our work specifically investigates a source of bias or noise in bug reports. We also review some work that has investigated other such issues.

Bug reports are one of the main artifacts in software maintenance research. They have been used to understand various phenomena about software bugs and to design tools to help developers in the bug fixing process. Bettenburg et al. [START_REF] Bettenburg | What makes a good bug report?[END_REF] investigated what kind of information developers think is the most helpful in a bug report. They also investigated the extents and reasons of duplicated bug reports [START_REF] Bettenburg | Duplicate bug reports considered harmful...really?[END_REF]. Bortis et al. [START_REF] Bortis | Porchlight: A tag-based approach to bug triaging[END_REF] proposed to tag bug reports automatically to help with bug triaging. Tian et al. [START_REF] Tian | Drone: Predicting priority of reported bugs by multi-factor analysis[END_REF] proposed a machine-learning based approach for assigning a priority to each bug report. Anvik et al. [START_REF] Anvik | Who should fix this bug?[END_REF] and Shokripour et al. [START_REF] Shokripour | Why so complicated? simple term filtering and weighting for location-based bug report assignment recommendation[END_REF] proposed approaches for automatic assignment of bug reports to developers. Saha et al. [START_REF] Saha | Improving bug localization using structured information retrieval[END_REF] and Zhou et al. [START_REF] Zhou | Where should the bugs be fixed?more accurate information retrieval-based bug localization based on bug reports[END_REF] proposed approaches for automatic bug localization. Huo et al. [START_REF] Huo | An empirical study of the effects of expert knowledge on bug reports[END_REF] investigated the role of expert and non-expert knowledge in bug reports and its impact on the results of bug localization tools.

Bug Severity is one of the key features of a bug report, to understand the bug's importance. Researchers have used this attribute in numerous software engineering studies. Menzies and Marcus [START_REF] Menzies | Automated severity assessment of software defect reports[END_REF] and Lamkanfi et al. [START_REF] Lamkanfi | Predicting the severity of a reported bug[END_REF], [START_REF] Lamkanfi | Comparing mining algorithms for predicting the severity of a reported bug[END_REF] proposed a text mining and machine learning based approach to predict bug severity. Tian et al. [START_REF] Tian | Information retrieval based nearest neighbor classification for fine-grained bug severity prediction[END_REF] also predicted bug severity, based on information retrieval. Bhattacharya et al. [START_REF] Bhattacharya | Graphbased analysis and prediction for software evolution[END_REF] proposed a graph-based approach to estimate bug severity. Hooimeijer and Weimer [START_REF] Hooimeijer | Modeling bug report quality[END_REF] used bug severity to investigate and model bug report quality. They concluded that self-reported severity is an important factor in the model's performance. Giger et al. [START_REF] Giger | Predicting the fix time of bugs[END_REF] and Zhang et al. [START_REF] Zhang | Predicting bug-fixing time: an empirical study of commercial software projects[END_REF] used bug severity (with several other bug report features) to predict bug-fix time. Saha et al. [START_REF] Saha | An empirical study of long lived bugs[END_REF] used severity to understand the importance of long lived bugs.

Bias or noise in bug-relevant datasets is a well-known problem in software engineering research. Bird et al. [START_REF] Bird | Fair and balanced?: bias in bug-fix datasets[END_REF] investigated the potential biases in defect datasets in terms of bug features and commit features. They evaluated a popular defect prediction algorithm and showed that bug feature bias (e.g., unequal proportion of bug reports in terms of bug severity and developers' reputation) affects the performance of the algorithm. Later Nguyen et al. [START_REF] Nguyen | A case study of bias in bug-fix datasets[END_REF] confirmed that the bias in the bug-fix dataset exists not only in open-source projects but also in the datasets of commercial projects. Kim et al. [START_REF] Kim | Dealing with noise in defect prediction[END_REF] proposed an algorithm to detect such noisy instances in bug datasets so that they can be eliminated. However, these studies did not investigate noise in bug severity. Rahman et al. [START_REF] Rahman | Sample size vs. bias in defect prediction[END_REF] showed that consideration of the sample size of a dataset is as important as bias in the dataset. Antoniol et al. [START_REF] Antoniol | Is it a bug or an enhancement?: a text-based approach to classify change requests[END_REF] showed that not all the bug reports in bug tracking systems are actually bugs. Later, based on a comprehensive manual investigation on 7,000 issue reports, Herzig et al. [START_REF] Herzig | It's not a bug, it's a feature: How misclassification impacts bug prediction[END_REF] reported that one-third of the bugs in the issue tracking systems are not actually bugs and this misclassification affects bug prediction algorithms. Kochhar et al. [START_REF] Kochhar | Potential biases in bug localization: Do they matter?[END_REF] investigated the potential biases in the dataset of mappings between bug reports and corresponding fixed files, and described their impact on bug localization.

X. CONCLUSION

In this paper, we have studied the mislabeling of Normal bugs, and the impact that it can have on tools that rely on bug severity. Based on the studied software projects, we confirm that the bugs labeled Normal are often not normal according to the bug repository criteria. Furthermore, we find that the inclusion or exclusion of these reports, as well as their consideration as Normal bugs or according to their actual severity, can have a major impact on the accuracy of tools that rely on bug severity values. This raises a real dilemma for the software engineering researcher. Normal reports are very prevalent, around 80% of the reports in our study, but cannot be relied on and are damaging to tool evaluations.

A partial solution is to create a clean dataset. Indeed, our results show that a bug severity prediction tool gives better results when trained on clean data than when trained on noisy data. We have proposed two approaches to creating a clean dataset: manual inspection and selecting only reports where the severity has been changed after the original submission. The former, however, is time-consuming, and the latter is more approximate. The wide use of bug reports by the software engineering community thus suggests that the community may want to invest resources into creating larger clean datasets.

We have also observed that misclassification of bugs and enhancements is a severe problem, which also may affect many studies. It appears that distinguishing enhancements from bugs through the severity field is not effective, because it does not allow the user to express the urgency of the enhancement request. Users could be less tempted to create noisy data if bug tracking systems such as Bugzilla would provide a dedicated field to separate bugs from enhancements. Our future work includes manually validating more Normal bug reports to create a large-scale clean dataset, and improving the stateof-the-art of severity prediction algorithms.

Data. Our data and results are publicly available at: http://www.riponsaha.com/data/severity-assessments.csv.

TABLE VI PROPORTION

 VI OF NORMAL BUGS AT EACH SEVERITY LEVEL FROM THE BEST AND WORST CASE PERSPECTIVES

	System		Best Case			Worst Case
		Sev. Norm. NS	Enh. Sev. Norm. NS	Enh.
	JDT	14	48	48	15	50	40	20	15
		11% 38%	38% 12% 40% 32%	16% 12%
	CDT	14	66	21	24	45	52	4	24
		11% 53%	17% 19% 36% 42%	3% 19%
	PDE	5	46	45	29	40	31	25	29
		4% 37%	36% 23% 32% 25%	20% 23%
	Platform 4	38	54	28	36	36	24	28
		3% 30%	43% 22% 29% 29%	19% 22%
	Total	37	198	168 96	171 159	73	96
		7% 40%	34% 19% 34% 32%	15% 19%

TABLE VII PROPORTION

 VII OF NORMAL BUGS AT EACH SEVERITY LEVEL FROM THE OPTIMAL CASE PERSPECTIVE

	System		Same Response	Different Response		Total
		Sev. Norm. NS Enh. Sev. Norm. NS Enh. Sev. Norm. NS Enh.
	JDT	13 19	20 0	28 21	9	15 41 40	29 15
		10% 15% 16% 0% 22% 17% 7% 12% 33% 32% 23% 12%
	CDT	12 32	3	15 22 27	6	9	34 58	9	24
		10% 26% 2% 12% 18% 22% 5% 7% 27% 46% 7% 19%
	PDE	4	16	25 8	12 30	9	21 16 46	34 29
		3% 13% 20% 6% 10% 24% 7% 17% 13% 37% 27% 23%
	Platform 3	13	20 7	26 17	17 21 29 30	37 28
		2% 10% 16% 6% 21% 14% 14% 17% 23% 24% 30% 22%
	Total	32 80	68 30 88 95	41 66 120 174 109 96
		6% 16% 14% 6% 18% 19% 8% 13% 24% 35% 22% 19%

TABLE XII ACCURACY

 XII FOR SEVERITY PREDICTION CLASSIFIER TRAINED FROM TR noisy -Normal

			Predicted
		Severe Normal Non-Severe
	Actual	Severe Normal Non-Severe 29 63 52	. . .	37 48 71	Accuracy: 45%
		Accuracy: 67% if we exclude Normal bugs from TE
	indicates that the most noise is in Normal bugs. Finally,
	the reported accuracy again increases greatly if our test set
	does not include Normal bugs, reaching 67%. But knowing
	in advance whether a bug is Normal is not a reasonable
	assumption for the input of a bug severity prediction tool.
	Therefore, our overall results suggest that both misclassifi-
	cation and exclusion of Normal bugs may significantly affect
	any results based on bug severity.

http://www.bugzilla.org, https://www.atlassian.com/software/jira, https://www.mantisbt.org

http://wiki.eclipse.org/Eclipse/Bug_Tracking

http://www.eclipse.org/jdt, https://eclipse.org/cdt, http://www.eclipse.org/pde, https://projects.eclipse.org/projects/eclipse.platform

https://bugs.eclipse.org/bugs/

http://wiki.eclipse.org/Eclipse/Bug_Tracking

http://goo.gl/XP03JZ

Acknowledgments. We would like to thank the students who participated in the bug severity labeling study. This work was funded in part by the National Science Foundation (NSF Grant Nos. CCF-1117902 and CCF-0845628).


Yann Guédon
email: guedon@cirad.fr

Slope heuristics for multiple change-point models

Keywords: Data-driven slope estimation, Latent structure model, Model selection, Multiple change-point detection

With regard to multiple change-point models, much effort has been devoted to the selection of the number of change points. But, the proposed approaches are either dedicated to specific segment models or give unsatisfactory results for short or medium length sequences. We propose to apply the slope heuristic, a recently proposed non-asymptotic penalized likelihood criterion, for selecting the number of change points. In particular we apply the data-driven slope estimation method, the key point being to define a relevant penalty shape. The proposed approach is illustrated using two benchmark data sets.

Introduction

The slope heuristics were introduced by [START_REF] Birgé | Gaussian model selection[END_REF] as a new non-asymptotic penalized likelihood criterion for model selection. They showed that there exists a minimal penalty such that the dimension of models (and the associated estimator risk) selected with lighter penalties becomes very large. Moreover, they proved that considering a penalty equal to twice this minimal penalty allows to select a model close to the best possible (or oracle) model in terms of estimator risk. This approach has been recently popularized by the introduction of the data-driven slope estimation method by [START_REF] Baudry | Slope heuristics: overview and implementation[END_REF] which is a practical method for implementing slope heuristics. In the maximum likelihood estimation framework, this practical method is based on the expectation of a linear relation between the penalty shape (a function of the model dimension) and the maximized log-likelihoods for overparameterized models. We focus here on the application of the slope heuristics for selecting the number of change points in This paper was published as a part of the proceedings of the 30th International Workshop on Statistical Modelling, Johannes Kepler Universität Linz, 6-10 July 2015. The copyright remains with the author(s). Permission to reproduce or extract any parts of this abstract should be requested from the author(s). multiple change-point models.

2 Defining the log-likelihood function and the penalty shape for multiple change-point models

For multiple change-point models, the two possible log-likelihood functions are:

• log f (s * , x; J), the log-likelihood of the most probable segmentation s * in J segments (the number of change points is therefore J -1) of the observed sequence x. [START_REF] Lebarbier | Detecting multiple change-points in the mean of Gaussian process by model selection[END_REF] used this log-likelihood function for defining a slope heuristic for Gaussian models.

• log f (x; J), the log-likelihood of all the possible segmentations in J segments of the observed sequence x with f (x; J) = s f (s, x; J).

These log-likelihoods for K = 2, . . . , J can be exactly computed by a single application of a dynamic programming algorithm for log f (s * , x; J) and a smoothing algorithm for log f (x; J) [START_REF] Guédon | Exploring the latent segmentation space for the assessment of multiple change-point models[END_REF]. The slope estimation relies on maximized log-likelihoods computed for overparameterized models and, as shown in [START_REF] Guédon | Exploring the latent segmentation space for the assessment of multiple change-point models[END_REF][START_REF] Guédon | Segmentation uncertainty in multiple change-point models[END_REF], the most probable segmentation is often meaningless for overparameterized models. Consistently with our view of multiple change-point models as latent structure models [START_REF] Guédon | Exploring the latent segmentation space for the assessment of multiple change-point models[END_REF][START_REF] Guédon | Segmentation uncertainty in multiple change-point models[END_REF], we will thus focus on the log-likelihood of all the possible segmentations log f (x; J) for defining a slope heuristic. We investigated on several data sets this log-likelihood over the range of J values corresponding to overparameterized models and noted that this log-likelihood function is markedly concave for overparameterized models if J < T (e.g. 10 < T /J < 100), where T is the sequence length, but far less if J T . To apply the slope heuristics, it is required that [START_REF] Baudry | Slope heuristics: overview and implementation[END_REF]): (C1) The log-likelihood increases with J. (C2) The penalty shape pen shape (J) increases with J. To these two standard requirements, we add the two following specific requirements for multiple change-point models: (C3) The penalty shape pen shape (J) depends on the sequence length T . Adding a segment for say J = T /10 entails a smaller increase of the penalty shape than adding a segment for J T . (C4) The first-order differenced penalty shape pen shape (J)-pen shape (J -1) decreases with J. This decrease is not a linear function of J. For the definition of the penalty shape, our starting point was pen shape (J) = log n J where n J = T -1 J-1 is the number of possible segmentations in J segments. Consider the limiting case where all the segmentations are equally probable for a fixed J, then log f (x; J) = log γ J + log n J . In fact for overparameterized models, as shown in [START_REF] Guédon | Exploring the latent segmentation space for the assessment of multiple change-point models[END_REF][START_REF] Guédon | Segmentation uncertainty in multiple change-point models[END_REF] using different examples, log f (x; J) decomposes into a structural part corresponding to true change points and a noise part that increases with J. To respect the monotonicity of pen shape (J) as a function of J, we finally propose pen shape (J) = log T J-1 (J -1)! , with pen shape (J) -pen shape (J -1) = log T J -1 .

Illustrations on benchmark data sets

The use of the proposed slope heuristic for multiple change-point models is illustrated using two benchmark data sets corresponding to different segment models and sequence lengths. The slope heuristic is compared with the "exact"ICL criterion proposed by [START_REF] Rigaill | Exact posterior distributions and model selection criteria for multiple change-point detection problems[END_REF].

British coal mining disasters

The data consist of the dates of 191 coal mining disasters between 1851 and 1962 summarized as annual counts during the 112-year period. We assume that the number of disasters in any year has a Poisson distribution, and the underlying Poisson distribution parameter is piecewise constant through time.

TABLE 1. British coal mining disaster data: comparison of the exact ICL criterion and the slope heuristics (SH) with J as the penalty shape (pen shape 0) and the proposed penalty shape (pen shape 1). The criterion value and the corresponding posterior model probability P (MJ |x) are given for each J.

J 1 2 3 4 5 ICL J -413. 14 -358.70 -362.31 -369.34 -375.74 P (M J |x) 0 0.855 0.141 0.004 0 SH J (pen shape 0) -419. 68 -358.81 -357.04 -358.55 -361.01 P (M J |x) 0 0.2 0.49 0.23 0.07 SH J (pen shape 1) -407. 72 -360.19 -368.05 -377.00 -385.38 P (M J |x) 0 0.98 0.02 0 0

The slopes were estimated over the range J = 6, . . . , 20 and, we obtained a residual standard deviation of 1.05 with the naive penalty shape J instead of 0.04 with the proposed penalty shape. Both the exact ICL criterion and the slope heuristic with the proposed penalty shape favour 2 segments while the slope heuristic with the naive penalty shape J favours 3 segments and puts weight on 4 segments which is not consistent with the outputs of the different validation approaches shown in [START_REF] Guédon | Exploring the latent segmentation space for the assessment of multiple change-point models[END_REF][START_REF] Guédon | Segmentation uncertainty in multiple change-point models[END_REF]; see Table 1.

Well-log data

The data consist of 4050 measurements of the nuclear-magnetic response of underground rocks. The underlying signal is roughly piecewise constant, with each segment relating to a single rock type that has constant physical properties. We estimated Gaussian change in the mean and variance models on the basis of these data. The slope was estimated over the range J = 30, . . . , 80. The exact ICL criterion favours 18 and 19 segments while the slope heuristic with the proposed penalty shape favours mainly 16 segments; see Table 2. The 16segment model selected by the slope heuristic is far more consistent with the analysis of the segmentation space presented in Guédon ( 2013) than the 18-or 19-segment models selected by the exact ICL criterion.

TABLE 2 .

 2 Well-log data: comparison of the exact ICL criterion and the slope heuristic (SH) with the proposed penalty shape. The criterion value and the corresponding posterior model probability P (MJ |x) are given for each J.

	J	15	16	17	18	19	20
	ICL J	-69355.4 -69330.0 -69316.3 -69309.7 -69309.6 -69313.3
	P (M J |x)	0	0	0.014	0.378	0.403	0.063
	SH J	-69479.6 -69461.8 -69466.0 -69478.6 -69482.8 -69495.4
	P (M J |x)	0	0.89	0.11	0	0	0