Sara Abbaspour Asadollah

Daniel Sundmark
email: daniel.sundmark@mdh.se

Sigrid Eldh
email: sigrid.eldh@ericsson.com

Hans Hansson
email: hans.hansson@mdh.se

Eduard Paul Enoiu
email: eduard.paul.enoiu@mdh.se

A Study of Concurrency Bugs in an Open Source Software

Keywords: Concurrency bugs, Bug severity, Fixing time, Open source software

Concurrent programming puts demands on software debugging and testing, as concurrent software may exhibit problems not present in sequential software, e.g., deadlocks and race conditions. In aiming to increase efficiency and effectiveness of debugging and bug-fixing for concurrent software, a deep understanding of concurrency bugs, their frequency and fixing-times would be helpful. Similarly, to design effective tools and techniques for testing and debugging concurrent software understanding the differences between non-concurrency and concurrency bugs in real-word software would be useful. This paper presents an empirical study focusing on understanding the differences and similarities between concurrency bugs and other bugs, as well as the differences among various concurrency bug types in terms of their severity and their fixing time. Our basis is a comprehensive analysis of bug reports covering several generations of an open source software system. The analysis involves a total of 4872 bug reports from the last decade, including 221 reports related to concurrency bugs. We found that concurrency bugs are different from other bugs in terms of their fixing time and their severity. Our findings shed light on concurrency bugs and could thereby influence future design and development of concurrent software, their debugging and testing, as well as related tools.

Introduction

With the introduction of multicore and other parallel architectures, there is an increased need for efficient and effective handling of software executing on such architectures. An important aspect in this context is to understand the bugs that occur due to parallel and concurrent execution of software. In this paper we look into how the increase of such executions have impacted a number of issues, including the occurrence of related bugs and the difficulty to fix these bugs compared to fixing non-concurrent ones.

Testing and debugging concurrent software are faced with a variety of challenges [START_REF] Lu | Learning from mistakes: a comprehensive study on real world concurrency bug characteristics[END_REF]. These challenges concern different aspects of software testing and debugging, such as parallel programming [START_REF] Peri | Software development tools for multi-core/parallel programming[END_REF], performance testing, error detection [START_REF] Zhang | Conmem: Detecting crash-triggering concurrency bugs through an effect-oriented approach[END_REF] and more. Since concurrent software exhibit more non-deterministic behavior and nondeterministic bugs are generally viewed to be more challenging than other types of bugs [START_REF] Desouza | Automated, scalable debugging of MPI programs with Intel Message Checker[END_REF][START_REF] Godefroid | Concurrency at Microsoft: An exploratory survey[END_REF][START_REF] Süß | Common mistakes in OpenMP and how to avoid them[END_REF], testing and debugging concurrent software are also considered to be more challenging compared to testing and debugging of sequential software.

Developing concurrent software requires developers to keep track of all the possible communication patterns that evolve from the large number of possible interleavings or concurrently overlapping executions that can occur between different execution threads through utilizing the shared memory. Handling the many execution scenarios that this results in is a notoriously difficult task in debugging and makes it equally hard to create test cases [START_REF] Fonseca | A study of the internal and external effects of concurrency bugs[END_REF].

In the study presented in this paper we are particularly interested in isolating concurrency bugs from other types of bugs (non-concurrency bugs) and analyzing the distinguishing features in their respective fixing processes. Hence, the main emphasis of this research is on concurrency bugs, and to explore the nature and extent of concurrency bugs in real-world software. This exploration of bugs can be helpful to understand how we should address concurrency bugs, estimate the most time-consuming ones, and prioritize them to speed up the debugging and bug-fixing processes. Also it could be helpful for designers to avoid the errors that are more likely to occur during the early phases of the software lifecycle.

In our study we address the following research questions:

-RQ1: How common are different types of concurrency bugs, compared to nonconcurrency bugs? -RQ2: How much time is required to fix concurrency bugs, compared to fixing non-concurrency bugs? -RQ3: Are concurrency bugs more severe than non-concurrency bugs?

In this study we investigate the bug reports from an open source software project. We classify bugs into two distinct groups, i.e., concurrency bugs and non-concurrency bugs. We classify the concurrency bugs based on bug type, severity and fixing time. We compare the non-concurrency and concurrency bug in terms of their reporting frequency, severity and fixing time. Our results indicate that a relatively small share of bugs is related to concurrency issues, while the vast majority are non-concurrency bugs. Fixing time for concurrency and non-concurrency bugs is different but this difference is not big. In addition, concurrency bugs are considered to be slightly more severe than non-concurrency bugs.

Methodology

In this study first we start with Bug-source software selection in order to select a proper open source software for our study. Second, we identify the set of concurrency bug reports in the issue tracking database of the selected project through a keyword search in Bug report selection. Then we manually analyze the full set of identified bug reports in order to exclude those that are not concurrency-related. Finally, in bug reports classification process, we collect data for the concurrency bugs, and classify the bug reports based using the classification scheme described in Section 3. The following subsections describe the steps of this research process in further detail.

Bug-source Software Selection

We were interested in an open source application that coordinates distributed processes with significant number of releases and an issue management platform for managing, configuring and testing. We selected the Apache Hadoop project 1 as the open source project for our study. The full justification for selecting Hadoop as our study object is provided in the list below.

(1) Hadoop has changed constantly and considerably in 59 releases over six years of development. [START_REF] Peri | Software development tools for multi-core/parallel programming[END_REF] Due to Hadoop's key concept of parallel and distributed abstraction it is recently adopted by several big companies (i.e., Facebook, Ebay, Yahoo, Amazon and more). (3) Detailed information on bugs and bug fixes are openly available. (4) The Hadoop framework has been widely adopted by both the industry and research communities [START_REF] Polato | A comprehensive view of Hadoop research?A systematic literature review[END_REF]. [START_REF] Godefroid | Concurrency at Microsoft: An exploratory survey[END_REF] It has a web interface for managing, configuring and testing its services and components.

Hadoop tracks both enhancement requests and bugs using JIRA 2 . JIRA is an issue management platform, which allows users to manage their issues throughout their entire lifecycle. It is mainly used in software development and allows users to track any kind of unit of work, such as project task, issue, story and bug to manage and track development efforts.

Bug Reports Selection

In this stage we selected the concurrency bugs from the bug report database including bugs from different versions of Apache Hadoop, including bug reports from the period 2006-2015, i.e., the last decade. In total, the Hadoop bug report database contains 4872 issues in this period that are tagged as "Bug" 3 .

We automatically filtered reports that are not likely to be relevant by performing a search query on the bug report database. Our search query filtered bugs based on (1) "Bug" as report type, (2) the status of the report, and (3) keywords relevant to concurrency. Figure 1 summarizes the bug report selection process.

In filtering based on "Bug" as report type step, we practically searched in the Apache Hadoop report database for the reports with issue type "Bug" according to our main objective and bug definition.

In filtering based on the status of the report step, we searched for bugs with "Closed" (i.e., this report considered finished, the resolution is correct) and "Fixed" resolution status (i.e., fix for this issue has been implemented). We only selected "Fixed" and "Closed" reports since unfixed and open bug reports might be invalid and root causes described in the reports could be incorrect. It would then be impossible for us to completely understand the details on these bugs and determine their types. In filtering based on the keywords relevant to concurrency step, we decided to use the keywords that could help us to include the bug reports were compatible with the scope of this study. In identifying such keywords, we reviewed the keywords utilized in similar previous studies [START_REF] Abbaspour | Towards classification of concurrency bugs based on Observable properties[END_REF][START_REF] Lu | Learning from mistakes: a comprehensive study on real world concurrency bug characteristics[END_REF]. The keywords included in the search, i.e. the terms, are as follows. After filtering we obtained a final set with 411 reports.

thread, blocked, locked, race, dead-lock, deadlock, concurrent, concurrency, atomic, synchronize, synchronous, synchronization, starvation, suspension, "order violation", "atomicity violation", "single variable atomicity violation", "multi variable atomicity violation", livelock, live-lock, multi-threaded, multithreading, and multi-thread.

Table 1 shows the bug count across the different stages of the bug report selection process. Note that this selection process may have some limitations, discussed in more detail in subsection 5.1.

Manual Exclusion of Bug Reports and Sampling of Non-concurrency Bugs

In this stage we manually analyzed the 411 bug reports obtained in the previous step 4 . The manual inspection revealed that some of the bugs that matched the search query were not concurrency bugs. Thus, we excluded them. More specifically, we determined the relevance of the bugs by checking (1) if they describe a concurrency bug, and if they do, ( 2) what type of concurrency bug is it. The latter is done, by comparing their descriptions (or explanations) with our concurrency bug classification (Section 3.1). If we could not map a report with any class we excluded that report from our set. We also excluded reports with very little information, since we could not analyze them properly. After filtering we obtained a final set with 221 concurrency bugs.

As explained in Section 1, our main objective is understanding the differences between non-concurrency and concurrency bugs. For comparison purposes, we randomly sampled an equally sized subset of non-concurrency bugs that were reported during 2006-2015 and were "Fixed" and "Closed". These bugs were used for comparative analysis between the concurrency and non-concurrency bug sets. In this study, we use the term non-concurrency bugs instead of sample of non-concurrency bugs for all comparative analysis.

Bug Reports Classification

We analyzed the issues and information contained in the reports using them to map to the concurrency bug classification manually. Each bug report contains several types of information, which were valuable in recognizing and filtering the concurrency bugs with other types of bugs to aids us understand the characteristics of bugs. The bug reports contained for example the description of the bug with some discussions among the developers on how to detect, where to detect (bug localization) and how to fix the bugs. Typically most of the reports include a description of the correction of the bug, and a link to the the version of the software where the bug has been corrected, and even the scenario of reproducing the reported bug. The reports also contain additional fields such as perceived priority, created date, resolved date, version affected and more.

We used different types of fields in order to explore the concurrency bug issues in the Hadoop project. We used the priority field to estimate the severity of the bug. The interval between the Created date and Resolved date fields was used to calculate the amount of (calendar) time required to fix the bug (fixing time).

Study Classification Schemes

In order to perform the bug classification process we defined three main classifiers and grouped the reports based on these classifiers. The classifiers were Type of concurrency bug, fixing time and severity. These three classification schemes are described in detail below.

Concurrency Bug Classification

In [START_REF] Abbaspour | Towards classification of concurrency bugs based on Observable properties[END_REF], our main contribution is a better understanding of the different types of concurrency bugs. We classified and mapped the relevant bug reports related to the types of concurrency bugs using a classification of concurrency bug types. It categorizes concurrency bugs into seven disjoint classes (i.e., Deadlock, Livelock, Starvation, Suspension, Data race, Order violation and Atomicity violation).

Fixing Time Calculation

This class shows the time duration (days) which developer (or debugger) spend to fix a reported bug. Fixing time, calculated by subtracting the Created date and Resolved date fields of the bug.

Bug Report Severity Classification

In order to define priority for each issue based on developers' perspective we used a classification scheme similar to the classification defined in [START_REF]What is an Issue -Atlassian Documentation[END_REF]. Blocker shows the highest priority. It indicates that this issue takes precedence over all others. Critical indicates that this issue is causing a problem and requires urgent attention. Major shows that this issue has a significant impact. Minor indicates that this issue has a relatively minor impact. Trivial is the lowest priority.

Results and Quantitative Analysis

This section provides the analysis of the data collected for bugs obtained from the Hadoop project bug database. We used 442 bugs (i.e., 221 are concurrency bugs while the rest are non-concurrency bugs sampled for our analysis) reported between 2006 and 2015. The bug selection process is described in Section 2.

RQ1: How common are different types of concurrency bugs, compared to non-concurrency bugs?

As seen in Figure 2(a), out of the 3591 bugs reported in the Hadoop database, 221 (i.e., 6.15%) bugs are related to concurrency issues and are causing a certain type of concurrency bugs, while the rest (i.e., 93.85%) are identified as non-concurrency bugs.

The 221 concurrency bugs were further categorized according to the concurrency bug classification in [START_REF] Abbaspour | Towards classification of concurrency bugs based on Observable properties[END_REF]. As mentioned already in Section 3.1, this taxonomy defines seven types of concurrency bugs. For the sake of this study, we have added Not clear category to the taxonomy. The Not clear category includes reports that cover bugs related to concurrency and parallelism, but are not classified according to the concurrency bugs taxonomy. For these bugs, the summary and description of the report shows it is a concurrency bug, but further classification of bug type is prohibited by a very project implementation-specific explanation of the bug details and solution.

In addition, we investigated the frequency with which each type of concurrency bug appears, with the aim of getting insights into bug prioritization. In Figure 2(b) we show the number of concurrency bugs according to their category and how often they are reported in the data we collected. From a total of 221 bug reports, almost half of them (i.e., 44%) concern data races (or race conditions), a well-known and common concurrent bug [START_REF] Qi | Pacman: Tolerating asymmetric data races with unintrusive hardware[END_REF]. In addition, about 15% of the reports reported the Suspension bug type and only two bug reports were categorized as Livelock bugs. Answer RQ1: Only 6.15% of the total set of bugs are related to concurrency issues, while the majority of bugs (i.e., 93.85%) are of non-concurrency type.

RQ2: How much time is required to fix concurrency bugs, compared to fixing non-concurrency bugs?

In order to gain better understanding on how difficult is to fix concurrency bugs in comparison with non-concurrency bugs, we conducted a quantitative analysis of the effort required to fix both concurrency and non-concurrency bugs. This effort was measured as explained in Section 3.2.We used this time as an indicator for the complexity involved in fixing these bugs. Table 2(a) lists the detailed statistics on the obtained results for fixing time of concurrency and non-concurrency bugs. These results are also summarized in Figure 3(a) in the form of box-plots (the vertical axis scale of the plot is logarithmic). Interestingly, the fixing time for concurrency and non-concurrency bugs is very similar, with an average of 58 days and 54 days for fixing concurrency and non-concurrency bugs, respectively.

To evaluate if there is any statistical difference between concurrency and nonconcurrency bugs fixing time we use a Wilcoxon Signed Rank test, a non-parametric hypothesis test for determining if there is any statistical difference among two data sets, with the assumption that the data is drawn from an unknown distribution. We use 0.05 as the significance level. In addition, we calculate the Vargha-Delaney A-statistic as a measure of effect size [START_REF] Vargha | A critique and improvement of the CL common language effect size statistics of McGraw and Wong[END_REF] for analyzing significance. This statistic is independent of the sample size and has a range between 0 and 1. The choice of what constitutes a significant effect size can depend on context. Vargha and Delaney [START_REF] Vargha | A critique and improvement of the CL common language effect size statistics of McGraw and Wong[END_REF] suggest that A-statistic of greater than 0.64 (or less than 0.36) is indicative of "medium" effect size, and of greater than 0.71 (or less than 0.29) can be indicative of a "large" effect size.

We are interested in determining if the fixing time for concurrency bugs is similar to the one for non-concurrency bugs. We begin by formulating the statistical hypotheses as follows: the null hypothesis is that fixing time of the concurrency and non-concurrency bugs sets have identical distributions (H 0 ) and the alternative hypothesis is that the distributions are different (H 1 ). Based on the p-value of 0.027, which is less than 0.05, we reject the null hypothesis. That is, the fixing time of concurrency bugs and non-concurrency bugs are statistically different. When calculating the Vargha-Delaney A-statistic we obtained a value of 0.560 which indicates a "small" standardized effect size [START_REF] Vargha | A critique and improvement of the CL common language effect size statistics of McGraw and Wong[END_REF]. From our results, we can see that the fixing time for concurrency bugs is different from the fixing time for non-concurrency bugs, but that this difference corresponds to a "small" standardized effect size.

We were also interested in understanding the differences between fixing time for each type of concurrency bugs. Figure 3(b) summarize our results in the form of box plots. It is obvious that Data races took the longest time to fix (i.e., 80 days on average). Order violation and Deadlock type of bugs took less time (55 and 43 on average, respectively) while Livelock and Starvation type of bugs took shorter fixing time (17 and 24 days on average, respectively). Table 2(b) lists the detailed statistics on the obtained results for each type of concurrency bugs. To evaluate if there is any signif-q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q C bugs Non-C bugs q q q q q q q q q q q q q q q q q q q q q q q q q q q q DeadLock Data race icant statistical difference between the different types of concurrency bugs, we use a Wilcoxon Signed Rank test and calculate the A-statistic effect size. To this end, we report in Table 3 the p-values and the effect size for each type of concurrency bugs. The tested hypotheses are formulated as follows: the null hypothesis is that fixing time results between two different bug types are drawn from the same distribution and the alternative hypothesis is that the fixing time results are drawn from different distributions. We use a traditional statistical significance limit of 0.05 and Vargha and Delaney's suggestion [START_REF] Vargha | A critique and improvement of the CL common language effect size statistics of McGraw and Wong[END_REF] for statistical significance. Examining Table 3, we can conclude that the null hypothesis is accepted with p-values above the traditional statistical significance limit of 0.05 for the majority of bug types except for "Deadlock-Data race" and "Deadlock-Suspension" pairs where the null hypothesis is rejected. This shows that the bug fixing time is not different except between "Deadlock-Data race" and "deadlock-Suspension". For example, in Table 3 we show the obtained p-value of 0.008 for testing the pair "Deadlock-Data race", which is less than 0.05, and therefore we can reject the null hypothesis: the fixing time for Deadlock and Data Race bug types are different. In addition, the A-statistic for the same pair of bug types is about 0.796 (or 0.204 in the second row), which is greater than the significance level of 0.71. We can say that in this case the effect size is "large". We can conclude that fixing time for deadlock and data race bug types is different with a "large" effect size. It should however be noted that the likelihood of statistical errors vastly increases when doing multiple tests using the same dataset. The results from the inter-bug-type comparisons are thus less reliable than the results from the comparison between concurrency and non-concurrency bugs.

Livelock

Answer RQ2: Concurrency bugs do require longer fixing time than non-concurrency bugs, but the difference is not very large.

RQ3: Are concurrency bugs more severe than non-concurrency bugs?

We analyzed the difference between concurrency and non-concurrency bug severity in order to understand if the severity of bugs is differently distributed. Figure 4 shows the severity distributions. In order to statistically compare the severity between concurrency bugs and non-concurrency bugs, we apply a Two-Sample Kolmogorov-Smirnov test (also known as two-sample K-S test) to find if the frequency between these two types of bugs is significantly different. Our null hypothesis can be formulated as follows: are the severity level results of concurrency bugs and non-concurrency bugs drawn from the same distribution. In this test, if the D-value is larger than the critical-D-value, the observed frequency is distributed differently. Table 4 shows that the D-value is 0.131, which is larger than the Critical-D-value of 0.091. Thus, statistically we have enough evidence to conclude that there is a difference between the concurrency and non-concurrency bug severity distribution. In other words, the concurrency and non-concurrency severity types are distributed differently.

Finally, we are also interested to identify the severity distribution difference between different concurrency bugs classes. The results obtained for this analysis are shown in Figure 5. The results indicate that the highest severity is observed for the "Blocker" class. We expected that most of the bugs to be of Deadlock type. In reality, as shown in Figure 5, most of the bugs are of Data race type. We can interpret this fact in the following way: the Data race type might represent the most problematic bug type in terms of severity in the Hadoop project.

On the other hand, after comparing the different type of concurrency bugs we found that most of the bugs categorized as being part of the Data race type in terms of severity belongs to the Major class; the highest population of Deadlock bugs belong to Critical class; the highest population of bugs categorized in the Suspension type belongs to Critical class; the highest population of bugs corresponding to Atomicity violation type belongs to Major and Minor class; the highest population of Order violation bugs belongs to Minor class and the highest population of Starvation bugs belong to Major class. We can interpret that the Deadlock and Suspension bugs have higher severity.

Answer RQ3: Concurrency bugs are considered to be more severe than nonconcurrency bugs, but the difference is not that large.

Discussion

In our study, we found a much smaller share of concurrency bugs than the one found by other similar studies. This could possibly be due to the different time span of our study and that of other similar studies. An interesting observation is that 70% of the bugs that we found were reported in the five-year interval of 2006-2010, and the remaining 30% were reported in the five-year interval of 2011-2015.

Similarly, the fixing time found by other studies is much larger for concurrency bugs than for non-concurrency bugs. We find a difference, but it is relatively small. This could be due to a large portion of fixing time in other studies relate to reproducing the bugs using the bug scenario in the bug description. In our study we found surprisingly few reports stating difficulties in reproducing the bug.

The involvement of more than one thread cause a concurrency bug. For this reason we predicted to find that concurrency bugs were more severe than non-concurrency bugs. However, we expected most of the "Blocker" bugs to deadlock type due to its characteristic and properties but this was not the case. In our study Data race was the biggest portion of "Blocker". We can interpreter that Data race is the most problematic bug to fix in Hadoop project.

Moreover, our investigation shows that about half of the concurrency bugs are of Data race type. The reason could be that Data race is more severe than other type of bugs (as shown in Figure 5) and it would be normal if it takes longer time to fix. About 15% of the bugs belongs to the Suspension type. By investigating the bug reports' description and comments we noticed that most of the Suspension bugs occurred when the developer put a block of code in waiting mode for an unnecessary long time, thereby causing a Suspension bug.

Validity Threats

In the design and execution of this study, there are several considerations that need to be taken into account as they can potentially limit the validity of the results obtained.

-Some concurrency bugs might go unfixed or unreported because they occur infrequently, only on certain platforms/software configurations, or are hard to reproduce. It would be interesting to consider these kinds of bugs, but they are not likely to have detailed discussions. As a result, these bugs are not considered as important as the reported and fixed concurrency bugs that are used in our study. However, based on our investigation , 81 bug reports out of 4872 bug reports tagged as "Cannot reproduce" while 25 of them mentioned at least one of the concurrency keywords (listed in Section 2.2) in their description or comments. -The reports with other status (i.e., "In Progress" -this issue is being actively worked on at the moment by the assignee-or "Open" -This issue is in the initial 'Open' state, ready for the assignee to start work on it-) were not considered in this study and there is a chance that we did not include the relevant reports. -Even if the obtained results (for RQ1, RQ2 and RQ3) are based on data samples from a single project, these results might apply to other software as well. More analysis is required to confirm whether this is in fact the case.

Related Work

A series of related studies on debugging, predicting and fixing concurrent software have been conducted. In particular, there is a large body of studies on prediction [START_REF] Neuhaus | Predicting vulnerable software components[END_REF][START_REF] Nagappan | Static analysis tools as early indicators of pre-release defect density[END_REF][START_REF] Rahman | Comparing static bug finders and statistical prediction[END_REF][START_REF] Lewis | Does bug prediction support human developers? findings from a google case study[END_REF] and propagation [START_REF] Voinea | How do changes in buggy Mozilla files propagate?[END_REF][START_REF] Pan | Measuring structural quality of object-oriented softwares via bug propagation analysis on weighted software networks[END_REF] of bugs in source code.

Most of these studies strive to identify the components or source code files, that are most prone to contain bugs. Fault prediction partially focuses on understanding the behavior of programmers and its effects on software reliability. This work is complementary to the study conducted in this research, which is concentrated on a specific type of bugs (i.e., concurrency bugs) and on understanding their consequences.

[19], Vandiver et al. analyzed the consequences of bugs for three database systems. This work is focused on presenting a replication architecture, instead of on studying bugs. The authors did not distinguish between concurrency and non-concurrency bugs, and only evaluated whether they caused crash or Byzantine faults.

Three open-source applications bug databases (Apache web server, GNOME desktop environment and MySQL database) are investigated by Chandra and Chen [START_REF] Chandra | Whither generic recovery from application faults? A fault study using open-source software[END_REF], with a slightly different focus than ours. The authors analyzed all types of bugs (only 12 of them were concurrency bugs) to determine the effectiveness of generic recovery techniques in tolerating the bugs. Concurrency bugs are only one possible type of bug that affects their results. In contrast, based on our main objective we focus on a more narrow type of bugs by limiting ourselves to concurrency bugs, but provide a broader analysis (comparing concurrency and non-concurrency bugs) taking into consideration several types of these bugs.

Farchi et al. [START_REF] Farchi | Concurrent bug patterns and how to test them[END_REF] analyzed concurrency bugs by creating such bugs artificially. They asked programmers to write codes which have concurrency bugs. We believe that artificially creating bugs may not lead to bugs that are representative of the real-world software bugs. We, on the other hand, analyze the bug database of an open-source software, which is well maintained, and widely used software.

Lu et al. examined concurrency bug patterns, manifestation, and fix strategies of 105 randomly selected real-world concurrency bugs from four open-source (MySQL, Apache, Mozilla and OpenOffice) bug databases [START_REF] Lu | Learning from mistakes: a comprehensive study on real world concurrency bug characteristics[END_REF]. Their study concentrated on several aspects of the causes of concurrency bugs, but the study of their effects was limited to determining whether they caused deadlocks or not. We use the same study methodology to find relevant bug reports but we provide a complementary angle by studying the effects of recent concurrency bugs not limited to deadlock and not-deadlock bugs. In other words, according to our objective we used other classification(s) for our study.

Conclusion and Future Work

This paper provides a comprehensive study of 4872 fixed bug reports from a widely used open source storage designed for big-data applications. The study covers the fixed bug reports from the last ten years, with the purpose of understanding the differences between concurrency and non-concurrency bugs. Two aspects of these reports are examined: fixing time and severity. Based on a structured selection process, we ended up with 221 concurrency bugs and 221 non-concurrency bugs (sampled). By analyzing these reports we have identified the frequencies of concurrency and nonconcurrency bugs. The study also helped us to recognize the most common type of concurrency bugs in terms of severity and fixing time. The main results of this study are: (1) Only a small share of bugs is related to concurrency while the vast majority are non-concurrency bugs. (2) Fixing time for concurrency and non-concurrency bugs is different but this difference is relatively small. (3) Concurrency and non-concurrency bugs are different in terms of severity, while concurrency bugs are more severe than non-concurrency bugs. These findings could help software designers and developers to understand how to address concurrency bugs, estimate the most time-consuming ones, and prioritize them to speed up the debugging and bug-fixing processes.

Fig. 1 :

 1 Fig. 1: Bug report selection workflow

Fig. 2 :

 2 Fig. 2: Distribution of bugs

  Fixing time comparison for concurrency (C bug) and nonconcurrency bugs (Non-C bugs).

Fig. 3 :

 3 Fig.3: Fixing time analysis; boxes span from 1 st to 3 rd quartile, black middle lines are marking the median and the whiskers extend up to 1.5x the inter-quartile range while the circles represent the outliers.

Fig. 4 :

 4 Fig. 4: Concurrency and non-concurrency bug severity

Fig. 5 :

 5 Fig. 5: Concurrency bugs severity

Table 1 :

 1 Report counts from different stage of bug report selection process

	Filter	Selected reports	# of reports
		Total Hadoop bug reports	3591
	2006-2015 & Bug & Fixed & Closed	Keywords match related bug reports	411
		Concurrency bug reports analyzed	221
	2006-2015 & Bug & Fixed & Closed Sample of non-concurrency bug reports 221

Table 2 :

 2 Descriptive statistics results for concurrency and non-concurrency bug sets in terms of

	fixing time.		(a) Fixing time comparison
	Bug	Fixing time	Average Minimum Maximum Median	Standard Deviation
	Concurrency	58.3	0.1	1221.0	13.1	143.4
	Non-concurrency	54.2	0.1	998.1	7.9	133.3
		(b) Fixing time comparison for concurrency bugs
	Concurrency bug Fixing time	Average Minimum Maximum	Median	Standard Deviation
	Deadlock		43.1	0.1	943.2	4.3	171.5
	Data race		80.0	0.1	1221.0	17.0	181.4
	Order violation		54.9	0.1	471.3	20.4	100.6
	Atomicity violation		39.1	0.2	253.7	19.7	72.3
	Livelock		16.9	15.0	18.9	16.9	2.7
	Starvation		24.4	1.0	89.2	12.8	29.6
	Suspension		38.5	0.5	197.2	17.9	46.9

Table 3 :

 3 Wilcoxon test for concurrency bugs fixing time comparison

	H0	HA	Hypothesis test result	Deadlock Data race	Order violation	Atomicity violation	Livelock Starvation Suspension
	Deadlock	P-value A-statistic	--	0.007389 0.7951961 0.2046976 0.0892762 0.0154108 0.0735466 0.2636306 0.03635 0.06946 0.1476 0.128 0.001921
	Data race	P-value A-statistic	0.007389 0.2048039	--	0.9368 0.1302476 0.0572856 0.0100967 0.0507493 0.1589967 0.9595 0.9702 0.646 0.5778
	Order		P-value	0.03635	0.9368	-	0.9452	0.8894	0.6391	0.6888
	violation	A-statistic	0.2826018 0.8697524	-	0.0990541 0.0181741 0.0821554 0.2936019
	Atomicity	P-value	0.06946	0.9595	0.9452	-	0.9231	0.7902	0.6644
	violation	A-statistic	0.2983314 0.9427144 0.2428526	-	0.0195557 0.0879477 0.3191625
			P-value	0.1476	0.9702	0.8894	0.9231	-	0.8128	0.8869
			A-statistic	0.3081093 0.9899033 0.2548624 0.1121267	-	0.0919864 0.3364332
	Starvation	P-value A-statistic	0.128 0.2998193 0.9492507 0.2444468 0.107716 0.0195026 0.646 0.6391 0.7902 0.8128	--	0.4343 0.3216601
	Suspension	P-value A-statistic	0.001921 0.2806356 0.8410033 0.2166543 0.0958657 0.017377 0.0797641 0.5778 0.6888 0.6644 0.8869 0.4343	--

https://issues.apache.org/jira/browse/hadoop

https://www.atlassian.com/software/jira

Bug is "a problem which impairs or prevents the functions of the product"[START_REF]What is an Issue -Atlassian Documentation[END_REF].

We provide the raw data of this study at https://goo.gl/sr6iDQ.

Acknowledgment

This research is supported by Swedish Foundation for Strategic Research (SSF), SYN-OPSIS project and the Swedish Knowledge Foundation (KKS), TOCSYC project.

Authors Suppressed Due to Excessive Length


Stamatia Bibi
email: sbibi@uowm.gr

Apostolos Ampatzoglou
email: a.ampatzoglou@rug.nl

Ioannis Stamelos
email: stamelos@csd.auth.gr

A Bayesian Belief Network for Modeling Open Source Software Maintenance Productivity

Keywords: maintenance, productivity, software metrics, Bayesian networks

Maintenance is one of the most effort consuming activities in the software development lifecycle. Efficient maintenance within short release cycles depends highly on the underlying source code structure, in the sense that complex modules are more difficult to maintain. In this paper we attempt to unveil and discuss relationships between maintenance productivity, the structural quality of the source code and process metrics like the type of a release and the number of downloads. To achieve this goal, we developed a Bayesian Belief Network (BBN) involving several maintainability predictors and three managerial indices for maintenance (i.e., duration, production, and productivity) on 20 open source software projects. The results suggest that maintenance duration depends on inheritance, coupling, and process metrics. On the other hand maintenance production and productivity depend mostly on code quality metrics.

Introduction

Software maintenance is, according to van Vliet [START_REF] Van Vliet | Software Engineering: Principles and Practice[END_REF], one of the most effort and time intensive activities in the software lifecycle, since it consumes 50-75% of the overall project resources. According to ISO 14764-2006 the most common maintenance activities include the adaptation of a system to new environments, the implementation of additional requirements, the improvement of run-or design-time quality properties, or the identification of latent defects. It is expected that the worse the internal quality of a system is, the more difficult to maintain the software will be [START_REF] Ampatzoglou | The Financial Aspect of Managing Technical Debt: A Systematic Literature Review[END_REF]. However, there are several precautionary actions that managers can take (e.g., software refactorings [START_REF] Fowler | Refactoring: Improving the Design of Existing Code[END_REF]) so as to improve the structural quality of the software, which in turn will lead to decreased maintenance effort and cost. Nevertheless, the time budget that is usually allocated on such maintenance activities is usually limited. Thus, it should be cautiously allocated to the modules that suffer from the most important bad smells [START_REF] Fowler | Refactoring: Improving the Design of Existing Code[END_REF].

In this study we focus on the relationship between structural software quality characteristics (expressed through metrics), maintenance production (i.e., software units maintained) and duration. Although we acknowledge the fact that structural quality is not the only parameter that should be used in such a model (i.e., a complete model should consider the number of changing requirements, number of defects, etc.), we isolate software structure, in order to explore its influence exclusively. In classical economics production and duration are combined through the measure of productivity, which is defined as an average measure of production in the unit of time [START_REF] Boehm | Software Engineering Economics[END_REF]. In this line of thought, we define maintenance productivity as the average lines of code that are being maintained (software units maintained) in a given time unit. Although the exploration of the relationship between software units maintained and structural quality measures is not novel (see Section 2), to the best of our knowledge this is the first study that considers maintenance productivity. To achieve this goal we used a Bayesian Belief Network (BBN) to model the relationships between the input (structural software metrics) and output parameters (maintenance production, maintenance duration, and maintenance productivity). The network has been trained through a case study on 454 versions of 20 Open Source Software (OSS). The rest of the paper is organized as follows: Section 2 presents research efforts on studies related to maintenance production; Section 3 presents background information on BBN and the metrics that have been used in the developed network; Section 4 presents the case study design; in Sections 5, we present and discuss our results and the implications to the researchers and practitioners; in Section 6 we present threats to validity and finally, and in Section 7 we conclude the paper.

Related Work

Research on software maintenance effort can be categorized into two groups: (a) development of effort estimation models, and (b) development of methods that improve the maintenance outcomes.

Among popular effort estimation models we can find Estimation by Analogy (EbA) and estimation based on Regression. Shepperd et al. [START_REF] Shepperd | Effort estimation using analogy[END_REF] suggested EbA for allocating staff in software maintenance projects based on similar completed historical maintenance projects. De Lucia et al. [START_REF] De Lucia | Assessing effort estimation models for corrective software maintenance through empirical studies[END_REF] use multiple regression analysis to construct corrective maintenance effort estimation models. The results show that the performance of the model is improved when the different types of maintenance tasks are included. The use of function points are proposed for estimating the effort of software maintenance projects [START_REF] Abran | Measurement of the Maintenance Process from a Demandbased Perspective[END_REF]. Bayesian Networks are also used in Software Maintenance [START_REF] Van Koten | An Application of Bayesian Network for Predicting Object -Oriented Software Maintainability[END_REF]. Van Koten and Gray [START_REF] Van Koten | An Application of Bayesian Network for Predicting Object -Oriented Software Maintainability[END_REF] use Bayesian Networks to predict the maintainability measured as the change in software lines of code between subsequent releases. The results are improved compared to regression models. In [START_REF] De Melo | Bayesian networks in software maintenance management[END_REF] BBNs aim to predict delays in software maintenance tasks. Bayesian Networks have been also applied in software fault prediction [START_REF] Fenton | Predicting software defects in varying development lifecycles using Bayesian Nets[END_REF] and cost estimation [START_REF] Stamelos | On the use of Bayesian belief networks for the prediction of software productivity[END_REF]. The application of the method indicated certain advantages regarding their ability to be combined with expert judgement and provide flexible and informative estimates.

On the other hand there are studies found in literature that examine the factors that affect software maintenance effort. System size, system age, number of input/output data items, application type, and programming language are among the parameters that can affect software maintenance effort according to [START_REF] Abran | Measurement of the Maintenance Process from a Demandbased Perspective[END_REF] and [START_REF] Ghods | Contributors to quality during software maintenance[END_REF]. Also there is a vast literature on code metrics that affect the maintainability of software and consequently the corresponding effort required. According to Riaz et al. [START_REF] Riaz | A systematic review on software maintainability prediction and metrics[END_REF] the metric suites proposed by Li and Henry [START_REF] Li | Object-Oriented Metrics that Predict Maintainability[END_REF] and Chidamber and Kemerer [START_REF] Chidamber | Managerial Use of Metrics for Object Oriented Software: An Exploratory Analysis[END_REF] are the most accurate ones for assessing software maintainability. On the other hand there are studies that explore the specific parameters that affect open source project development, quality and evolution [START_REF] Koch | Exploring the effects of process characteristics on product quality in open source software development[END_REF], [START_REF] Midha | Governance practices and software maintenance: A study of open source projects[END_REF], [START_REF] Mockus | Two case studies of open source software development: Apache and Mozilla[END_REF]. Developer participation, core team size, code ownership, productivity, defect density, and problem resolution intervals were examined in these studies leading to the conclusion among others that defect density and productivity benefit from the large open source community of testers and bug fixers. Speed of releases seems to require highly modularized software. To the best of our knowledge this is the first study that considers maintenance productivity.

Background Information

In this section we present some background information that is needed to comprehend this study. In particular, we provide information related to Bayesian Belief Networks, and structural quality metrics associated to maintainability.

Bayesian Belief Networks are Directed Acyclic Graphs (DAGs), which are causal networks that consist of a set of nodes and a set of directed links between them, in a way that they do not form a cycle [START_REF] Jensen | Bayesian Networks and Decision Graphs[END_REF]. Each node represents a random variable that can take mutually exclusive values according to a probability distribution, which can be different for each node. Each link expresses probabilistic cause-effect relations among the linked variables and is depicted by an arc starting from the influencing variable (parent node) and terminating on the influenced variable (child node). The relation between the two nodes is based on Bayes' Rule:

(1)

A simple BBN example is presented in Figure 1. The model consists of two nodes. The first node (NOC) represents the number of classes in a software package and the second node (Maintenance Effort) represents the effort required for package maintenance measured in Person-Months widely reported in literature as Man-Months (MMs). We consider that the values of these two nodes fall into two discrete categories (Low and High). For the node NOC, let's suppose that Low values range between 1 class and 10 classes (Similarly for Maintenance Effort values). For example the first column of the Node Probability Table states that if the number of classes is low then there is 70% probability that the maintenance effort will be low and 30% percent probability that the maintenance effort will be high. The algorithm used for learning Bayesian Networks is analytically presented in [START_REF] Cheng | Learning Bayesian Belief Network Classifiers: Algorithms and System[END_REF]. The tool used for constructing the networks and the classifier can be found in the web (http://www.cs.ualberta.ca/~jcheng/bnpc.htm).

Maintainability Predictors:

In this study we focus on metrics that are the most accurate software maintainability predictors. According to Riaz et al. [START_REF] Riaz | A systematic review on software maintainability prediction and metrics[END_REF] the metric suites proposed by Li and Henry [START_REF] Li | Object-Oriented Metrics that Predict Maintainability[END_REF] and Chidamber and Kemerer [START_REF] Chidamber | Managerial Use of Metrics for Object Oriented Software: An Exploratory Analysis[END_REF] are the most accurate ones, while the model of van Koten and Gray [START_REF] Van Koten | An Application of Bayesian Network for Predicting Object -Oriented Software Maintainability[END_REF] is the most stable one [START_REF] Riaz | A systematic review on software maintainability prediction and metrics[END_REF]. The metric suite that is used from the aforementioned study is presented in Table 1. In order to automate the calculation of software quality metrics we used Percerons Client (http://www.percerons.com), i.e., a tool developed in our research group.

Case Study Design

In order to investigate the influence of structural quality in several managerial maintenance indices (i.e., duration, effort and productivity), we performed a case study on 20 open-source software (OSS) projects. The case study is designed and reported according to the guidelines of Runeson et al. [START_REF] Runeson | Case Study Research in Software Engineering: Guidelines and Examples[END_REF].

Objectives and Research Questions

The objective of this study is to analyze software metrics and investigate their influence on the maintenance production, duration and productivity. In particular we investigate three research questions: RQ 1 : Which quality metrics are related to the duration of maintenance among two successive releases?

This question aims at identifying the metrics that mostly influence the time required for developing a new version of an OSS project measured in days elapsed from one release to another. Quick successive releases may indicate the level of readiness of the project community, the tendency to launch quick releases, and short time to fix bugs or add functionality.

RQ 2 : Which quality metrics are related to the production of maintenance among two successive releases?

The objective of this question is to reveal which quality metrics are more associated to the production of the maintenance in OSS, measured as the number of lines of code added from previous release. The target of this analysis is to identify the quality characteristics whose levels enable a substantial change in a single version. We expect that low values of coupling and complexity, and high values of cohesion, will yield for projects in which maintenance production is large and efficient.

RQ 3 : Which quality metrics are related to the productivity of maintenance among two successive releases? This research question is relevant to productivity. Productivity is calculated as the ratio between production and duration. We note that this calculation of productivity deviates from the typical one, because duration in OSS development might include idle time, when developers are inactive. However, to the best of our knowledge, the actual development time cannot be retrieved from source code repositories. The objective is to confirm if high levels of quality lead to increased productivity, and to some extent eliminate the bias caused by change load.

Case Selection & Data Processing

To collect subjects for our case study we retrieved a set of popular and active OSS projects. In particular, we selected only projects that were active in 2014 (i.e., published at least one version), and when sorted by popularity (based on the default sourceforge.net measure) they were among the top ranked ones. Additionally, we selected projects containing more than 300 classes to ensure that sufficient units of analysis will participate in the study and that toy-applications would be excluded. Therefore, we downloaded and collected metrics data from 20 popular OSS projects so as to build a dataset that is adequate for statistical analysis. Some demographics for these projects are presented in Table 2. The case study is an embedded multiple-case study, in the sense that for every OSS project (i.e., case), we analyze every transition between two versions (i.e., units of analysis) [START_REF] Runeson | Case Study Research in Software Engineering: Guidelines and Examples[END_REF]. For each release of the projects presented in Table 2, we collected three sets of data: the structural quality metrics presented in Table 1(see Section 3), the targeted maintenance managerial indices (see Section 4.1), and some process metrics. The obtained process metrics include the date each release was uploaded, the sequential number of release that indicates the maturity of the project (i.e., if we are in the beginning of the project lifecycle or not), and the number of downloads recorded for each release (considering the previous version each time) from the date launched to 01/01/2015. At this point the resulting project releases were further filtered to exclude releases that presented negative values in the production variable (deletion of functionality) and releases that presented zero duration time (releases uploaded all at once). After this pre-processing phase the remaining project releases were selected for analysis. Descriptive statistics for all dependent variables of our analysis are presented in Table 3. To apply the BBN analysis we had to further process the gathered metrics, since quality metrics present continuous arithmetic values. In particular, continuous values had to be transformed to categorical ones, so as to be utilized in BBN analysis. The transformation method used was the equal frequency binning [START_REF] Witten | Data Mining: Practical machine learning tools and techniques[END_REF]. This method automatically sets the boundaries of each bin (category), so as to ensure that all of them have an equal number of observations. Each metric that is represented in a categorical form has five distinct categories (VL-Very Low, L-Low, A-Average, H-High, and VH-Very High).

Data Analysis

On the completion of the aforementioned process we applied the Bayesian Network analysis and utilized heatmaps to graphically represent our data. Our goal was to better depict the influence of the independent variables on the dependent ones.

The first step of the analysis includes the selection and tuning of the variables that affect the structure of the BBN. In this step expert input is required to set the order of the variables that will affect the causal relationships that the model will define. Metrics order was selected taking into account two considerations: (a) the development phase, when the values of the metrics will be available in the sense that we first define certain design metrics, while some time afterwards certain source code metrics are available, and (b) the relatedness of metrics, based on their calculation method.

example coupling metrics like CBO and MPC are closely related. In the model CBO precedes MPC, in the sense that the value of CBO "includes" the value of MPC.

After extracting the model we employed heatmaps to visualize the obtained outcome. Heatmaps are graphical representations that use color intensity in order to de-note occurrence frequency. While applying heatmaps to our results set, we developed a matrix, in which row represents a value of the dependent variable (e.g., low maintenance duration) and a column a value of the independent variable (e.g., systems with a very low level of polymorphism). Each cell is a percentage that shows the probability that the independent variable will present a certain value according to the value of the dependent variable (e.g., 10% of the cases with a very low level of polymorphism are maintained very quickly-low maintenance duration).

Results & Discussion

In this section we present the results obtained from the BBN analysis, organized by research question. The extracted network including all structural quality metrics, process measures, and managerial maintenance indices is presented in Figure 2. We note that since this study aims to investigate the interdependencies between the nodes of the graph, the estimation accuracy of the network is not explored. A discussion / interpretation of the most important relationships presented in the network is provided later in this section, while answering each research question.

Duration

In this section we identify the structural properties that are related to maintenance duration (RQ 1 ). From Figure 2, we can observe that variables that are directly connected to DURATION are Number of downloads, Depth of Inheritance Tree (DIT), and Message Passing Coupling (MPC). The heatmaps visualizing the relationships of these variables to the duration of the maintenance activities is presented in Table 4. By interpreting the left-most part of the heatmap, we can observe that as the number of downloads increases, the maintenance duration becomes larger. This result can be intuitively interpreted by the fact that projects with many downloads are expected to serve larger communities, leading to the creation of more feature requests and bug fixing activities. The number of extra requirements that are requested in each mainte-PRODUCTION nance cycle increases their duration. Based on the central part of the heatmap, our results suggest that an average depth of inheritance tree offers shorter maintenance cycles. On the other hand, extensive or limited inheritance leads to larger maintenance cycles. Again, this can be characterized as an expected result, in the sense that: (a) using very large inheritance trees decreases software understandability since the full list of accessible variables and metrics from a class can spread to many classes, and (b) using limited inheritance does not make use of one of the most important benefits from using the object-oriented paradigm, and at the same time hinders the applicability of well-known object-oriented principles (e.g., open-closed principle [START_REF] Martin | Agile software development: principles, patterns and practices[END_REF]) and design patterns (e.g., template method, state/strategy, etc. [START_REF] Gamma | Design patterns: elements of reusable Object-Oriented software[END_REF]) that have been reported as beneficial to maintainability. Finally, very high coupling leads to high or very high maintenance duration (see right-most part of the heatmap). This result is expected in the sense that according to the high coupling principle [START_REF] Martin | Agile software development: principles, patterns and practices[END_REF] dense inter-connection of classes reduces their understandability, increases the probability of initiating a ripple effect, and reduces their changeability. On the other hand, by focusing on very small coupling we get interesting results. Intuitively one would expect that very small coupling would increase modularity (low ripple effect, increase changeability) and therefore be connected to quick maintenance cycles, but this is the 2 nd most frequent event. The most frequent event is that very small coupling is connected to large maintenance cycles. This result, although initially surprising, can be interpreted by the inherent relationship/trade-off between coupling and cohesion (as it is confirmed by the network of Figure 2). Specifically, it is expected that low coupling can also lead to bad design, in cases that specific classes are growing so large that they do not need to call methods from other classes (they are self-contained). This leads to reduced coupling, but probably these classes are assigned to more than one responsibility. This decision overrules the Single Responsibility Principle, which according to Martin [START_REF] Martin | Agile software development: principles, patterns and practices[END_REF], in turn leads to maintainability problems, especially focused on understandability and change proneness.

Production

In this section, we explore the BBN for mining relationship related to RQ 2 , i.e., metrics that are related to maintenance production, measured as lines of code added from one version to the other. Similarly to Section 5.1, in Table 5, we present the heatmap for the two variables that are directly related to maintenance production. Interestingly, we can observe that the two most important parameters are: one complexity metric (namely AMS) and the same coupling metric as in Table 4 (namely MPC). This is considered important from two perspectives:

(a) Average Method Size (AMS) has, until now, not been explored as a possible maintainability predictor, although it appears to be related to maintenance production. However this result can be considered intuitive, in the sense that the size of the method is related to one the most persistent and frequently occurring bad smells, namely Long Method [START_REF] Fowler | Refactoring: Improving the Design of Existing Code[END_REF]. According to the literature methods of large size are very difficult to understand, modify and extend, leading to difficulties in increasing maintenance production.

(b) Message Passing Coupling (MPC) is consistently the coupling metric that can be used as the optimal maintainability index. The superiority of MPC with regard to assessing ripple effects (a significant aspect of maintainability) compared to the other examined coupling metrics is also discussed by Arvanitou et al. [START_REF] Arvanitou | Introducing a Ripple Effect Measure: A Theoretical and Empirical Validation[END_REF]. In particular MPC is the only (among the investigated metrics) that captures both coupling volume (number of relationships) and coupling intensity (how closely connected the two classes are). An additional important characteristic of MPC is that it counts coupling intensity using the discrete count function, and therefore is not biased from the number of times one method is being called [START_REF] Ampatzoglou | The Effect of GoF Design Patterns on Stability: A Case Study[END_REF].

Productivity

Similarly to RQ 1 and RQ 2 , in Table 6, we present the heatmap that visualizes the parameters that are more closely related to maintenance productivity. By comparing the results of Table 6, with those of Tables 4 and5, we can observe that productivity is more closely related to maintenance production rather than maintenance duration. In particular we observe that the only metric that is consistently among the optimal maintenance predictors is MPC, whereas AMS is only related to maintenance production and productivity. Concerning the range of values for these variables we can reach the following conclusions: According to Martin [START_REF] Runeson | Case Study Research in Software Engineering: Guidelines and Examples[END_REF], the average method size should not exceed 5 lines of code. Based on the findings of this case study, very small methods provide a maximum productivity in 42% of the cases, which is a rather intuitive result. The counter point, as discussed by Fowler [START_REF] Fowler | Refactoring: Improving the Design of Existing Code[END_REF] (i.e., long methods are hard to maintain) cannot be discussed based on our results, since in order for such an investigation to be possible, it would be required to count the number of long method bad smells and not an aggregated/average measure. Concerning MPC, we can observe that tightly coupled systems are exhibiting small and very small productivity rates-a result that is expected based on the negative effect of coupling on maintainability.

Implications to Practitioners & Researchers

The results of our case study have provided some interesting insights and implications to both practitioners and researchers. Concerning practitioners, our study provides evidence that indeed structural quality metrics are affecting maintenance indices. However, from the vast amount of metrics that are available, only a limited number of them are highly influential. As expected the most significant metrics cover the most frequently quantified quality attributes, directly or indirectly, i.e., complexity, cohesion, coupling, and inheritance. In particular, the study pointed out that practitioners should pay attention on balancing the values of the following metrics: Average Method Size (AMS), Message Passing Coupling (MPC), and Depth of Inheritance Tree (DIT). Thus, practitioners should not only include these metrics in their quality dashboards, but also manage their natural trade-off (e.g., optimizing coupling on the expense of cohesion).

Furthermore, the outcome of this study indicated some interesting future research directions. First a replication of the case study can be performed including metrics of different categories and suites (e.g., architecture level) and also metrics that describe the communication and message exchange of the OSS developers' community. Secondly, the development of a complete estimation model that additionally takes into account factors other than software structure is expected to fully capture and predict all aspects of software maintenance productivity. Finally, tailoring the proposed model in an industrial context is necessary for the adoption of such approaches in closed source software development.

Threats to validity

In this section we discuss the threats to validity of our case study, based on the categorization described by Runeson et al. [START_REF] Runeson | Case Study Research in Software Engineering: Guidelines and Examples[END_REF]. With regard to internal validity, we need to clarify that the relationships obtained through the network are influenced by some confounding factors. For example, the load of high-level changes from one version of the system to another (e.g., new features, number of fixed bugs, etc.) is not considered in this study. However, we believe that the large size of our dataset imposed a normal distribution of actual changes that minimizes the effect of the confounding factor. Additionally, we remind that the investigation of the strength of relationships is out of the scope of this study, since it only aims at their identification.

Concerning external validity, we have identified two threats to generalization. First, since all the examined projects come from one source code repository the results cannot be generalized to the OSS population. Second, since we only explored Java projects (due to the limitation of the used tool) the results cannot be generalized to other programming languages. However, we believe that projects popularity and size are indicative factors of successful OSS projects.

Regarding reliability, based on the fact that our study is purely quantitative and the developed protocol (see Section 4) is thoroughly described, we believe that the process is easily replicable by other researchers. Though, concerning construct validity, Bayesian network model construction includes by its nature many subjective decisions made by the modeler. The parameters of the model such as the selection of the binning method for the discretization of the values of the continuous variables and the ordering of the independent variables may also affect the model extracted. This provides an interesting direction for future studies that could include additionally sensitivity analysis regarding the parameters of the model.

Conclusions

Long-term monitoring of maintenance effort requires a measurement process of respective attributes that affect software maintenance and a model to represent the interrelations of the attributes gathered. Open source software is a suitable paradigm of continuous, sustainable maintenance efforts. Understanding why each system requires more or less maintenance effort is necessary for better monitoring software maintenance process and releases launch. In this study we have performed an empirical study on the factors that affect software maintenance duration, production and productivity on 20 open source software projects. We suggest Bayesian Networks as a tool to model all attributes and factors that can affect maintenance efforts. The created model included structural quality and process metrics, and managerial indices. The results indicate that maintenance duration is affected by the popularity of a project and DIT (a design metric) and MPC (a source code metric). On the other hand Production and Productivity are solely affected by quality metrics AMS and MPC.

Fig. 1 .

 1 Fig. 1. a. A BBN for maintenance effort b. Node Probability table for Fig 1a

Figure 2 .

 2 Figure 2. Maintenance Indices Bayesian Belief Network

Table 1 .

 1 Maintainability PredictorsDepth of Inheritence Tree (DIT). Inheritence level number, 0 for the root class.

	Metric / Description	Phase	Quality Attribute (QA)

design inheritance Number of Children Classes (NoCC). Number of sub-classes that the class has. design inheritance Message Passing Coupling (MPC). Number of send statements defined in the class. code coupling Response For a Class (RFC). Number of local methods plus the number of methods called by local methods in the class. code coupling Lack of Cohesion of Methods (LCOM). Number of disjoint sets of methods (number of sets of methods that do not interact with each other), in the class. code cohesion Data Abstraction Coupling (DAC). Number of abstract types defined in the class. design coupling Weighted Method per Class (WMC). Average cyclomatic complexity of all methods. code complexity Number of Methods (NOM). Number of methods in the class. design size Lines of Code (SIZE1/LoC). Number of semicolons in the class. code size Number of Properties (SIZE2). Number of attributes and methods in the class design size Coupling Between Objects (CBO). Number of classes that one class depends on design coupling Average Method Size (AMS). Avereage number of semicolons in a method code complexity

Table 2 .

 2 Case Study Subjects

	Name AoI Azureus Checkstyle Dr Java File Bot FreeCol FreeMind Hibernate Home Player Html Unit	NOC 749 3,888 1,186 3,464 7,466 794 443 3,821 457 920	Versions 31 24 32 55 20 40 41 48 30 27	Cases 27 22 16 43 17 33 35 31 25 25	Name Java Game Library iText ZDF MediaThek Pixelator Mondrian Open Rocket Subsonic Sweet Home 3D UMS Tux Guitar	NOC 645 654 617 827 1,471 3,018 4,688 341 5,499 745	Versions 22 40 40 33 32 26 40 24 50 17	Cases 20 33 33 33 25 21 11 17 9

Table 3 .

 3 Descriptive Statistics for Maintenance Indices

	Maintenance Index Production (ΔLOC) Duration (in days) Productivity	N 454 2629.16 Mean 454 79.85 454 76.03	Minimum Maximum Std. Dev. 1 162429 10748.11 1 1055 91.6 0.01 9962 522.47

Table 4 .

 4 Parameters influencing maintenance duration

	Duration Very Small Small Average High Very High	VS 28 21 15 18 18	#Downloads S A H 23 18 16 15 VH VS 19 21 18 21 18 18 24 22 20 19 22 17 22 21 23 23 15 20 22 25 18	DIT A 16 24 17 20 S H VH VS 22 18 24 21 18 17 23 19 21 19 18 22 18 18 20 19 21 15 23 23 24	MPC A 19 22 21 16 S H VH 19 25 20 19 24 17 21 18 24 17 19 23 14 19 19 24

Table 5 .

 5 Parameters influencing maintenance production

	Production Very Small Small Average High Very High	VS 16 21 21 24 19	S 27 23 17 15 16	AMS A 25 18 15 17 24	H 20 21 21 22 20	VH 13 17 26 22 20	VS 15 25 23 20 18	S 22 20 21 19 18	MPC A 23 25 19 15 19	H 17 14 21 23 24	VH 22 17 16 24 21

Table 6 .

 6 Parameters influencing maintenance productivity

	Productivity Very Small Small Average High Very High	VS 19 19 23 17 25	S 26 22 20 12 16	AMS A 21 19 14 16 28	H 20 19 22 28 14	VH 14 21 20 27 17	VS 16 23 19 27 15	S 27 23 17 15 18	MPC A 22 22 24 13 22	H 13 14 18 28 28	VH 23 18 22 17 18