Fabio Mulazzani
email: fmulazzani@unibz.it

Bruno Rossi
email: brrossi@unibz.it

Barbara Russo
email: brusso@unibz.it

Maximilian Steff
email: maximilian.steff@unibz.it

Building Knowledge in Open Source Software Research in Six Years of Conferences

Keywords:

Since its origins, the diffusion of the OSS phenomenon and the information about it has been entrusted to the Internet and its virtual communities of developers. This public mass of data has attracted the interest of researchers and practitioners aiming at formalizing it into a body of knowledge. To this aim, in 2005, a new series of conferences on OSS started to collect and convey OSS knowledge to the research and industrial community. Our work mines articles of the OSS conference series to understand the process of knowledge grounding and the community surrounding it. As such, we propose a semi-automated approach for a systematic mapping study on these articles. We automatically build a map of cross-citations among all the papers of the conferences and then we manually inspect the resulting clusters to identify knowledge building blocks and their mutual relationships. We found that industry-related, quality assurance, and empirical studies often originate or maintain new streams of research.

Introduction

Since its origins, the diffusion of the OSS phenomenon and the information about it has been entrusted to the Internet and its virtual communities of developers. As such, information on OSS has grown exponentially resulting in a vast quantity of data readily available. This data has attracted the interest of researchers and practitioners aiming at formalizing this information into a body of knowledge (e.g. [START_REF] Feller | Understanding Open Source Development[END_REF], [START_REF] Feller | Perspectives on Free and open Source Software[END_REF]). For this reason, in 2005, a new series of conferences on OSS (OSS conference series 1 ) and, in 2009, a new journal 2 have been established. These initiatives have substantially contributed to initiate a long process to ground knowledge in OSS that masters data from different sources and consolidates them into well-accepted concepts and their mutual relations. As such, they represent a valuable source for understanding how OSS knowledge has been created and how it will evolve in the future. Our work tackles this issue proposing a systematic mapping study of the papers of the OSS conference series.

Systematic Mapping Studies (SMS) and Systematic Literature Reviews (SRL) are techniques of knowledge synthesis. These techniques are typically based on manual inspections of articles ( [START_REF] Kitchenham | What's up with software metrics? -A preliminary mapping study[END_REF], [START_REF] Kitchenham | Systematic literature reviews in software engineering -A systematic literature review[END_REF], [START_REF] Kitchenham | Guidelines for Performing Systematic Literature Reviews in Software Engineering[END_REF], and [START_REF] Stol | The use of empirical methods in Open Source Software research: Facts, trends and future directions[END_REF]). Manual inspection requires significant effort in mining large sets of articles. On the other hand, a complete automated inspection can produce inaccurate results. In our work, we propose a semiautomated approach to mine articles' repositories for systematic mapping studies. We automatically inspect articles to build a map of cross-citations and then we manually inspect the resulting clusters to identify the building blocks of knowledge in OSS and the social network of the community maintaining it. We selected the complete database of articles of the OSS conference series since its origin in 2005 3 . Our choice is driven by three criteria: 1) papers are peer reviewed -this excludes for example the MIT repository 4 , Apache conferences5 , OpenOffice.org conferences 6 , etc…; 2) the series' mission is to disseminate knowledge in OSS -this excludes traditional journals in software engineering; and 3) papers report more than a group discussion -this excludes workshops or one day events co-located with larger non OSS events. The result of this study aims at answering two major questions:

RQ1. Is there any social network underlying the research production at the OSS conference series?

RQ2. What are the major streams of research proposed at the OSS conference series?

Our answer to RQ1 will identify the cornerstone papers and the links among them across the years. Links will express the relation among the authors by means of the connection of their research production. The analysis will reveal unexpected and undeclared connections among authors as well as lack of connections among conceptually related papers. This will also illustrate the self-sustainability of the OSS conference series and the value that it provides to the OSS community. In addition, using the results in [START_REF] Stol | The use of empirical methods in Open Source Software research: Facts, trends and future directions[END_REF], our research will also discuss how empirical studies fit the network. An answer to RQ2 will help build the baseline for future investigations in OSS research or to extend existing ones.

In the following section, we introduce related work and motivate our work. Section 3 presents our method of SMS and Section 4 explains our analysis methodology. In Section 5, we explore the results of our analysis and describe the clusters of papers we identified, followed by a summary of our findings in Section 6. We close with the conclusions and limitations.

Background and Motivation

The software engineering community has been increasingly adopting Evidence-Based Software Engineering (EBSE) approaches to build discipline-specific bodies of knowledge such as Inspection, Testing, and Requirements Engineering ( [START_REF] Budgen | Investigating the applicability of the evidence-based paradigm to software engineering[END_REF], [START_REF] Ven | The Organizational Adoption of Open Source Server Software by Belgian Organizations[END_REF]). Apart from the traditional ways of doing literature review, also called ad-hoc reviews, the EBSE practice uses Systematic Literature Reviews (SLRs) ( [START_REF] Brereton | Lessons from applying the systematic literature review process within the software engineering domain[END_REF], [START_REF] Kitchenham | Systematic literature reviews in software engineering -A systematic literature review[END_REF], [START_REF] Kitchenham | Guidelines for Performing Systematic Literature Reviews in Software Engineering[END_REF], and [START_REF] Kitchenham | Procedures for Performing Systematic Reviews[END_REF]) and Systematic Mapping Studies (SMSs) ( [START_REF] Budgen | Investigating the applicability of the evidence-based paradigm to software engineering[END_REF], [START_REF] Kitchenham | What's up with software metrics? -A preliminary mapping study[END_REF], and [START_REF] Petersen | Systematic mapping studies in software engineering[END_REF]) as robust methodologies of searching, selecting, analyzing, and synthesizing literature and aggregate evidence on a specific topic. As such, SLRs and SMSs are called secondary studies as they aggregate research of other, so-called primary studies. SLR is "a means of evaluating and interpreting all available research relevant to a particular research question, topic area or phenomenon of interest" ( [START_REF] Dybå | Evidence-Based Software Engineering for Practitioners[END_REF], [START_REF] Kitchenham | Procedures for Performing Systematic Reviews[END_REF]). Research in SE has provided guidelines and lessons learned for performing SLR ( [START_REF] Biolchini | Systematic Review in Software Engineering[END_REF], [START_REF] Brereton | Lessons from applying the systematic literature review process within the software engineering domain[END_REF], [START_REF] Dybå | Applying Systematic Reviews to Diverse Study Types: An Experience Report[END_REF], [START_REF] Kitchenham | Guidelines for Performing Systematic Literature Reviews in Software Engineering[END_REF], and [START_REF] Staples | Experiences using systematic review guidelines[END_REF]). SMS is used to draw a landscape of reported research on a particular topic ( [START_REF] Gao | Network Analysis of the SourceForge.net Community[END_REF], [START_REF] Kitchenham | What's up with software metrics? -A preliminary mapping study[END_REF], [START_REF] Kitchenham | Guidelines for Performing Systematic Literature Reviews in Software Engineering[END_REF], and [START_REF] Petersen | Systematic mapping studies in software engineering[END_REF]). Being less specific, SMS requires significantly less effort than SLR; however, it provides only a course-grained overview of the published literature. An SMS can also serve as a preparation activity before doing an SLR. SLR and SMS in OSS has been typically used to provide evidence of practices and methods for a more general SE research purpose Only recently, secondary studies have been published to investigate specific areas of OSS. In 2010, Hauge et al. performed an SLR of research on OSS adoption [START_REF] Hauge | Adoption of open source software in softwareintensive organizations -A systematic literature review[END_REF]. In 2009, Stol et al. ([36], [START_REF] Stol | The use of empirical methods in Open Source Software research: Facts, trends and future directions[END_REF]) presented an SLR on empirical papers published at the OSS conference series. To our knowledge, the first secondary study that investigated OSS as a holistic phenomenon is the work in [START_REF] Aksulu | A Comprehensive Review and Synthesis of Open Source Research, Special Issue[END_REF]. In their work, the authors have published taxonomy of OSS mining 623 journal papers -excluding conference papers, though. In this context, our work provides an SMS on OSS as a holistic phenomenon. Our work is complementary to the work in [START_REF] Aksulu | A Comprehensive Review and Synthesis of Open Source Research, Special Issue[END_REF] in terms of papers investigated, method of analysis, and research goal.

Research Method

Our method follows the concepts of a systematic mapping study [START_REF] Petersen | Systematic mapping studies in software engineering[END_REF]. In the introduction, we illustrate our search criteria for inclusion and exclusion. Following them, we select all the papers of the OSS conference series. In this section, we describe how we identify and apply classification criteria on the selected papers. Classification categories are taken from the Calls for Papers of the OSS conference series and are used to label cross-citation clusters. In particular, we automate papers classification to reduce effort of articles' inspection and enable future replications. Following [START_REF] Anel | The Importance of Reviewing the Code[END_REF], to increase the transparency of our method, we also detail the tools we used and the process we follow. Worth noticing that this approach differs from the one proposed by Kitchenham in 2010 ( [START_REF] Kitchenham | What's up with software metrics? -A preliminary mapping study[END_REF]). Kitchenham uses citations to identify most and least cited papers. We propose here to extend this approach using citations to determine streams of research.

Creating the Directed Graph of Cross-citations

We collected the PDF files of papers from the Springer repository. We developed an application (PDF Analyzer 7 ) that (a) converts the papers from the PDF format to a textual representation, and then (b) injects this representation into an XML file with nodes corresponding to the paper's sections. The application allows user intervention during the conversion process. We have also sampled part of the XML files to verify and validate the tool output. When problems occurred, we manually corrected them with the aid of the original PDF file. We create a Python script that 1) extracts all papers' titles and conference years from the XML files, 2) parses all the references for all the paper titles, 3) for every hit, extracts conference names, and 4) goes over the references again to identify possibly missing titles using the different variations of conferences' names. We noticed that conferences' titles significantly vary in that we identified 17 different variations. At the end, we also manually further checked for missed citations. The final output of the Python tool classifies papers by year of conference and by citations and passes them to GraphViz8 to display the final graph (Fig. 11).

Descriptive Analysis of Cross-citations

Before performing any inspection of the clusters, we have analyzed the citations of the papers we found. Table 1 shows the distribution of citations over the years. Articles refer to full and short papers if any. In particular, Table 1 shows that there are a good number of isolated articles in that they do not cite other papers.

Inspecting the Graph

Fig. 11 illustrates the complete directed graph of articles that cite or are cited by other articles. Each article is a node and citations are edges. Articles-nodes mapping is provided at https://pro.unibz.it/staff/brusso/PapersUsed.html. We define fan in as the number of citations to a paper and fan out as the number of citations from a paper. We define the distributor of knowledge as a node in the graph that has at least fan-in equals two and an attractor of knowledge a as node with at least fan-out equals two. A pure distributor has zero fan out and a pure attractor has zero fan in. A node can also simultaneously be a distributor and an attractor. A node that is a distributor with fan out one is a router as it branches the knowledge of the single citation into different articles. For example, node #107 is a router that distributes the knowledge of paper #82 to five other articles (Fig. 2). Fig. 1 displays the types of nodes. A path is a set of nodes connected by edges following the direction of the graph.

Fig. 1. Types of nodes

To determine a research area in OSS, we make three assumptions: i) Pure distributors determine research areas; ii) A path originating from a pure distributor and leading to a pure attractor or a dead end (a paper that has only one fan out) determines an area of research in OSS; iii) Paths starting from a pure distributor determine a cluster. Thus, we start from a pure distributor, for example #82, and then follow one of its links. We follow links in opposite direction downwards in the graph until we reach a pure attractor or a dead end. Then we aggregate all the paths from a pure distributor to define a cluster. Finally, we add all the dead ends cited by an attractor of the cluster. For example, the edge linking #82 to #107 determines an area of research originated from paper #82 including the dead ends node #41 and #81. The five paths originated by paper #82 determine a cluster.

To label paths, we have used the taxonomy of the Call for Papers (CfP) of the OSS conference series 9 . Typically, a CfP includes the major topics of the conference. Consequently, accepted papers concern topics listed in the CfP. We have also considered the classification proposed in [START_REF] Stol | The use of empirical methods in Open Source Software research: Facts, trends and future directions[END_REF]. Unfortunately, the classification was too high-level for our analysis.

We have identified twenty -one clusters in the graph (Fig. 11). Four are bipolesclusters of two articles -and two are isolated clusters. Eleven in 2005, two in 2006, and two in 2007 originate the fifteen clusters. Thirty-four are empirical papers according to [START_REF] Stol | The use of empirical methods in Open Source Software research: Facts, trends and future directions[END_REF]. Note that the majority of the pure distributors that originate the largest clusters are empirical. Table 2 lists pure distributors and attractors that define the largest clusters -papers with more than three fan-out or fan-in. It also classifies them as empirical according to [START_REF] Stol | The use of empirical methods in Open Source Software research: Facts, trends and future directions[END_REF]. To identify the reason of the citation, we have manually looked up each citation in the text. We have used the wording of the authors and the position of the citation within the article structure to understand the reason for each citation. If, for example, a citation is located in the "Background" section only and it is given to justify the work, then we label the corresponding link as motivation of work.

Classification of the Articles

We have read all the articles following the patterns defined by the clusters. The reading confirms that each path originated from a pure distributor determines a welldefined perspective of research that takes its motivation from the distributor. In many cases, we are also able to identify authors that contributed the most to a given research area and determine the semantic of the cross-citations besides the motivation of work. In the following, we report of this analysis per cluster. The number of the pure distributor names clusters.

Cluster #82. The largest cluster originates from node #82. Paper #82 introduces the OSSmole project (later called FLOSSmole). OSSmole is a repository of data, scripts, and analysis of data collected from OSS projects. The cluster is then branched into five links (Fig. 2). Three links identify three major sub-clusters, defined by router #119, router #107, and node #97. The sub-cluster defined by router #107 is generally concerned with developer communities and social network analysis of these communities. Links are rather strong. Paper #174 cites paper #107 to motivate its research and the use of a metric (outdegree centrality), and cites paper #204 as example of method of analysis. Two of the articles in this cluster focus on the same project (Apache). Cluster #44. Paper #44 introduces to practices for quality assurance in OSS projects (Fig. 3). Paper #44 generates three sub-clusters on OSS deployment, project's activity, and community building and participation. Article #156 connects the last two topics by means of increase of the community size and their activity growth. In cluster #44, we were not able to identify major contributors as different authors contributed to the research streams and all the citations were to motivate the work. engagement in OS projects. Developers' engagement is used to correlate OSS to agile methods (#195), to evaluate the involvement of companies (#172), or to investigate communication among developers (#235). In the majority of the subtopics, Barahona and Robles and Capiluppi and Adams have been the major contributors. Citations define rather strong connections among papers of this cluster. For example, articles #102, #120, #195, #235 use data from KDE, whereas papers #304 and #301 use the same metric of paper #177 and #102 respectively. In some of the cases, they share some of the authors, too.

Cluster #5. Cluster #5 is a big cluster connected to three other sub-clusters (#24, #84, and #62 in Fig. 4). These clusters focus on the relation between OSS and industry. The empirical pure distributor #5 summarizes the works of an international workshop on pros and cons of the use of OSS in industry. This paper spins off into two contributions of people that participated in the workshop (#90) and of some of its authors (#146 and #88). Looking at the affiliations of the authors, we call cluster #24 the industrial "Scandinavian case": it starts with a case of interest in OSS in the Worth noticing is that all the three papers of 2006 (#84, #88, #90) report of a case study in industry, but only the first two are as empirical according to [START_REF] Stol | The use of empirical methods in Open Source Software research: Facts, trends and future directions[END_REF]. Cluster #20 and #2. Two pure distributors (#2 and #20) originate this cluster (Fig. 5). We call it the "Italian case" as the major contributors are Ardagna, Damiani, Frati and other Italians. In their three works, they deal with security of OSS. In the last two years, the area has evolved to the larger problem of selecting and integrating OSS for industry (#233 and #312) where security is especially relevant (like telecom applications). 2). In #139, Jensen and Scacchi present a reference model to discover OSS processes. Paper #312 presents a framework that compares methods for evaluating OSS. Paper #305 includes a systematic literature review to identify benefits and drawbacks of OSS (Fig. 9). Paper #230 builds a conceptual framework for planning release processes of OSS. Paper #175 defines the design of a platform for investigating the OSS phenomenon through multidisciplinary research. This work also provides recommendations and critical issues in OSS research. Paper #233 includes an extensive background section that motivates the work.

Measures of success of OSS project Fig. 9. Major Pure Attractor #305

Inter-cluster Connections

There are several interlinks among clusters (Figure 10). We mark a link between two clusters if there is a node that directly connects them, i.e. a pure attractor citing papers of two clusters. Connections among clusters describe mutual influence among topics and have the potential to converge to a single cluster 10 . The major findings in Figure 10 show that:

1. "Tools for data mining" is a cross cutting topic connecting various research areas, from the analysis of social structures and networks of OSS communities and their developers' engagement, software evolution, OSS security, and models of OSS quality. 2.

OSS in industry specifies into "adoption in industry," "role of industry," "case studies in industry" and a stream in "benefits, risks, and drawbacks of OSS." Fig. 10. Links between research streams 10 Figure 10 does not show papers that connect clusters. This information is accessible at https://pro.unibz.it/staff/brusso/LinkingPapers.html Fig. 11. The directed graph of citing articles within the OSS conference series. Grey nodes are empirical papers according to [START_REF] Stol | The use of empirical methods in Open Source Software research: Facts, trends and future directions[END_REF].

Discussion

Our analysis reports of well-scoped clusters of citations that highlight major areas of research in OSS and their evolution across the six years of OSS conferences. We report of a lively social network of citations (RQ1), and several streams of research established across years (RQ2).

In particular, we have found that the creation of a big repository for data mining (FLOSSmole) has originated research in social network analysis, tools for data mining, and analysis of code artefacts to understand maintenance processes, specifically bug fixing. The topic "tools for data mining" also appears to be a crosscutting theme that connects several other research areas.

Quality assurance is another hot topic. It motivates studies on community building (as a means for QA), OSS deployment (as a factor to consider in QA), project's activity (as a measure of QA) and on comparative models (for QA).

A specific concern at the OSS conference series is the perception of OSS in industry. Research has focused on the role of industry in OSS development and after 2008 on OSS adoption. A report of a large workshop in OSS that joined researchers and practitioners of the industry in 2005 initiated this research.

A good number of papers at the OSS conference series have been dedicated to developers' engagement. Developers' engagement is used to correlate OSS methods to agile methods, to evaluate the involvement of companies, or to investigate communication among developers. This topic is also cited in works on software evolution and community building and participation.

There are themes still little cited that might have some future potential: OSS process (meta-) modelling, OSS security, Agile and OSS development methods, and teaching OSS in universities.

We have found the majority of large clusters to have a group of authors that contribute to the research stream over the years. Clusters also reveal authors' interrelations that define informal research groups and outline social network structures, like in the "Scandinavian case."

Large clusters are initiated by empirical papers with the only exception being the paper on the FLOSSmole repository.

Papers with a large number of citations are synthesizers of research often presenting a framework or a platform to guide research in OSS.

As a final remark, we want to stress that we are aware that the sample we selected does not include all the crucial papers that may have contributed to ground knowledge in OSS. Some of the relevant contributions have been published in journals not dedicated to OSS or have simply been reported on the Internet. History of knowledge synthesis is full of such examples as illustrated by the FODA paper [START_REF] Kyo | FODA: Twenty years of Perspectives on feature Models[END_REF], a technical report cited more than 1500 times although not peer reviewed. Our future work will extend this analysis to OSS article hubs like for example http://flosshub.org or http://pascal.case.unibz.it or the MIT repository.

We also acknowledge that fact that cross-citations are indicators of research connections, but are not unique and not necessarily the best ones. A textual similarity analysis could give finer results. Namely, we have already applied some known algorithms (e.g. cosinus, Jaro Winkler) for text similarity on the XML sections of the papers. Despite some preliminary good results on titles and authors, for longer sections the algorithms proved to be inaccurate. We plan to use more sophisticated techniques of string similarity and a better data cleansing to get finer results.

Conclusions

In our research, we aim at understanding the major research topics outlined in six years of conferences dedicated to OSS. We have clustered articles by their crosscitations and inspected each single cluster to search for major research themes, evolution of research topics, and major initiators or synthesizers of research in OSS.

We have also introduced a social network underlying the research production. This network assumes that authors are connected by their common research interest revealed by citations as a community of practice. This network is different from a social network defined by co-authorship. In this case, links may be transversal to research topics and are generated by existing collaborations. Differently, our network may reveal hidden and potential collaborations among researchers.

Finally, we have used a semi-automated approach for systematic mapping studies that automatically creates clusters from cross-citations and manually inspect the clustered papers. Our research has identified cornerstone papers and links among them revealing unexpected and undeclared connections among authors as well as lack of connection among potentially connected topics. Large clusters are mostly initiated and sustained by empirical papers. Since 2008, synthesizers of research have introduced frameworks and platforms to perform OSS research paving the way for future work. The analysis of non-cited papers indicates that significant research has not been exploited, yet. Therefore, we recommend the OSS community to exploit further the potential provided by the OSS conference series while maintaining the interest in its major research streams.

Fig. 2 .Fig. 3 .

 23 Fig. 2. Citations cluster originated from paper #82 Howison and Crowston have been the major contributors. The branch ends in 2010 with the work of Conaldi and Rullani that proposes a global perspective of F/OSS network structure mining the SourceForge repository. Paper #119 on the future of OSS data mining starts a new branch that focuses on analyses and improvements of project mining tools. In this sub-cluster, we found citations motivated by the use of the same repository or the same research goals. The branch ends in 2008 with recommendations for the design of research infrastructure in OSS by Gasser and Scacchi (paper #175). Paper #97 originates a research branch on the analysis of code artefacts for modelling maintenance processes and specializes over the years in bug fixing processes. Although article #97 cites #82, it does not use OSSmole directly mining CVS log files. The major contributors in this branch are Dalle and den Besten and the majority of the articles focus on the Firefox community.

Fig. 4 .Cluster # 8 .

 48 Fig. 4. The four clusters #5, #24, #62, and #84 Cluster #8. Paper #8 concerns the involvement of volunteers in OS projects (Fig. 3). A first branch defined by router #102 evolves into the topic on developers'

  Finnish industry (#24), then includes the Swedish one in 2006 (#88) and closes with the Norwegian one in 2008 (#180). From paper #180, Hauge, Sorensen and Conradi initiate a new research theme on OSS adoption in industry (#233, #220, #309, #312, and #305). Paper #62 uses the Capability Maturity Model to assess the migration from closed to open software development in industry. This paper initiates a research investigation on the role of industry in Open Source Development (OSD) (router #138) and in particular, the migration of an existing business model to an open source one (#233). It further evolves in OSS adoption in industry (#180). Cluster #84 originates from the paper of Ven and Verelst (#84) and gives impulse to the specific perspective of OSS adoption in industry that relates to benefits, drawbacks, and risks. The authors that contribute the most to cluster #84 are Hauge and Conradi. The majority of the citations of the four clusters concerns motivation of work, but there are links that connect papers by the same method of analysis (e.g. #90 and #311).

Fig. 5 .Cluster # 17 .

 517 Fig. 5. Clusters #17 and #20_2

Fig. 6 .

 6 Fig. 6. Clusters #127 and #73Cluster #127. Cluster #127 (Fig.6) originated in 2007 with the results of the European project QualiPSo and enforced with the results on OSS trustworthiness (router #234). The majority of the works proposes and compares frameworks and comparative models of OSS quality. Major contributors are Morasca, Lavazza, and Taibi, members of the above project. In this cluster, the majority of the papers and in particular, all those in 2010 cite #127 and #234 to use the model defined there.Cluster #73 and Cluster #121. These clusters (Fig.6 and 7) concern process modelling and they are connected through the paper of Jensen and Scacchi in 2007 on guiding the discovery of OSS processes with a reference model (#139). Two branches

Fig. 7 .Fig. 8 .

 78 Fig. 7. Clusters #128, #121 Cluster #128. A study of network analysis on SourgeForge originates this cluster (Fig. 7). The cluster concerns social structures of OSS communities, as membership networks or communication networks. In particular, Gasser and Scacchi propose an infrastructure for research in social networks of OSS communities whereas Balieiro et al. introduce a study on the meso-level structure of OSS development as collaboration environment. Papers are authored by various members of the OSS research community and are connected by a net of citations that concerns mainly the motivation of work.Isolated clusters. There are six isolated clusters on the right of Fig.11(Fig.8). The four bipoles concern innovation (#164 -#193), adoption (#149 -#181), services (#87 -#137) and requirements (#211 -#241). In all but the bipole on adoption, the two papers share part or all of the authors. Cluster #7 concerns measures of success of OS projects. Over the years, the concept of success evolves from a static meaninge.g. number of hits of web searches -to an evolutionary perspective of development efficiency as the code produced over time. All the citations motivate the work and no specific author can be uniquely associated to the cluster. Cluster #68 concerns teaching OSS at university level. All the citations motivate the work.

Table 1 .

 1 Number of articles cited by or citing another article of the OSS conference series.

		2005	2006	2007	2008	2009	2010
	Cited	25	15	16	15	8	-
	Citing	-	12	13	17	11	13
	Total	83	41	51	42	28	40
	Isolated	58	23	33	21	16	27
	% citing/cited	31%	44%	35%	50%	43%	32%
	# articles cited >2	11	3	4	3	0	0

Table 2 .

 2 Major Distributors, Attractors, and Routers

	Distributors and routers
	2005
	#5 pure distributor and empirical paper ([1]), #8 pure distributor and empirical paper ([30]), #17 pure
	distributor and empirical paper ([27]), #44 pure distributor ([31]), #82 pure distributor ([22])
	2006
	#84 pure distributor and empirical paper ([39]), #107 router and empirical paper ([21]), #119 router
	([8]), #121 pure distributor and empirical paper ([32])
	2007
	#127 pure distributor ([38]), #128 pure distributor and empirical paper ([14]), #138 router and
	empirical paper ([20])
	2008
	#180 distributor/attractor and empirical paper ([19])
	2009
	#234 router ([9])
	Attractors
	2008
	#175 Platform for research ([15])
	2009
	#230 Framework ([33]), #233 Extensive background ([18])
	2010
	#305 Includes SLR ([17]), #312 Framework ([35])

International Symposium on Open Source Software initiated in

2005, in Genoa, Italy 2 International Journal of Open Source Software and Processes, http://www.igi-global.com

https://pro.unibz.it/staff/brusso/PapersUsed.html

http://opensource.mit.edu/

http://na11.apachecon.com/

http://www.ooocon.org

Apache PDFBOX library to convert PDF to TXT and DOM4J library to create XML the file

http://www.graphviz.org/

http://ossconf.org/.

Acknowledgements. We would like to thank Muhammad Ali Babar and Klaas Jan Stol and the reviewers for their valuable comments.


Claudia Ruiz
email: cruiz5@gsu.edu

William Robinson
email: wrobinson@gsu.edu

Towards a Unified Definition of Open Source Quality

Keywords: open source, software, quality, measurement, literature review

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Quality is extremely subjective, with as many definitions as there are people with opinions. It is no surprise that studies evaluating the quality of FLOSS and comparing it with traditionally developed software have produced mixed results [START_REF] Stamelos | Code Quality Analysis in Open Source Software Development[END_REF], [START_REF] Paulson | An empirical study of open-source and closed-source software products[END_REF], [START_REF] Kuan | Open Source Software as Lead-User's Make or Buy Decision: A Study of Open and Closed Source Quality[END_REF], [START_REF] Raghunathan | Open source versus closed source: software quality in monopoly and competitive markets[END_REF]. This is probably because of two main reasons: each study has defined quality differently and has evaluated it using different characteristics of different FLOSS projects. Defining quality differently will of course produce mixed results, but even when studies define quality in similar terms, they evaluate it using dissimilar projects and compare different project characteristics.

In order to understand what it is about certain FLOSS projects that lead them to produce high quality software, the antecedents of FLOSS quality must be found. There is, however, no current research on the antecedents of FLOSS quality [START_REF] Crowston | Free/Libre Open Source Software Development: What We Know and What We Do Not Know[END_REF].

This paper takes the first step towards addressing this issue by reviewing the FLOSS literature order to understand how is quality being conceptualized and to propose a unified definition of FLOSS quality.

The rest of the paper is organized as follows: sections two and three provide a brief background on FLOSS and software quality; section four presents the methodology followed; section five describes the findings, and section six discusses the implications of the findings., categorizing the findings according to research approach, and definition of quality.

FLOSS

FLOSS has grown dramatically in the 2000s and is an integral part of the IT industry. It directly supports 29% of the software that is developed in-house in the EU and 43% in the US and could reach a 32% share of all IT services by 2010 [START_REF] Ghosh | Economic Impact of Open Source Software on Innovation and the Competitiveness of the Information and Communication Technologies Sector in the E. U[END_REF].

Linux, Apache, Firefox are commonly found in many computers today and were developed using open source models. Apache is a Web server used by 60% of Websites worldwide [START_REF] Von Hippel | Open Source Software and the "Private-Collective" Innovation Model: Issues for Organization Science[END_REF] and 23.2% of European and 14.5% of North American Web surfers use the Firefox Web browser [START_REF] Hales | Firefox use continues to rise in Europe[END_REF].

This growing popularity begs the question: is FLOSS better than traditionally developed software? Traditionally developed software projects are considered successful if they finish on time, on budget and meet specifications. But the same standards cannot be applied to judge the success of FLOSS projects, since they usually have minimal budgets, are always in a state of development, do not have an official end time, and do not have formal specifications [START_REF] Scacchi | Understanding Requirements for Open Source Software[END_REF].

This lack of objective measures of success has not deterred the adoption of FLOSS products. It even has become a common assumption that FLOSS products are higher quality than traditionally developed software [START_REF] Stewart | The Impact of Ideology on Effectiveness in Open Source Software Development Teams[END_REF], [START_REF] Ajila | Empirical study of the effects of open source adoption on software development economics[END_REF] with firms entering FLOSS projects citing FLOSS's -quality and reliability‖ as one of the main motivating reasons for the endeavor [START_REF] Bonaccorsi | Comparing Motivations of Individual Programmers and Firms to Take Part in the Open Source Movement[END_REF].

This assumption can be traced back to Linus Torvalds, the architect of the Linux kernel, who said that -given enough eyeballs, all bugs are shallow‖ [START_REF] Raymond | The cathedral and the bazaar[END_REF]. Torvalds believed that FLOSS's public peer review and frequent releases lead to fewer bugs because there are more people looking at the software, reporting errors, and fixing those errors. This assumption has a kernel of truth: it has been observed that most problem (bug) reports and solutions in FLOSS projects are contributed by periphery community members and not so many by the core developers [START_REF] Rigby | Open source software peer review practices: a case study of the apache server[END_REF].

A FLOSS project is one that offers its software product's license in accordance to the Open Source Definition [START_REF]The Open Source Initiative[END_REF] providing for free redistribution of the compiled software and the openly accessible source code.

A typical FLOSS project is composed by a community, whose structure has been described as being like an -onion‖ with the most actively contributing members, who are the most invested in the project and have the greatest decision power in the inner part and the least contributing members with the least amount of decision power on the outside. The project leader is at the center and radiating out are the core members, the active developers, the peripheral developers, the bug fixers, the bug reporters, the readers, and the passive users [START_REF] Ye | Toward an Understanding of the Motivation of Open Source Software Developers[END_REF]. These roles are dynamic, changing as the community evolves as the system they are building evolves [START_REF] Ye | Toward an Understanding of the Motivation of Open Source Software Developers[END_REF].

Although each FLOSS project is different and has different development practices and processes, The Apache project can be used as a model of a mature FLOSS development project given its success and well documented and researched development cycle stages. Its stages are below: Identifying work to be done. Core developers look at the bug reporting database and the developer forums for change and enhancement requests. The core developers need to be persuaded of the priority of the request for it to be included in the agenda status list. Assigning and performing development work. Core developers look for volunteers to perform the work. Priority is given to code owners (those who created or have been actively maintaining the particular module). The developer then identifies a solution and gets feedback from the rest of the developers. Prerelease testing. Each developer performs unit testing of his/her own work. There is no integrated or systems testing. Inspections. Each developer then commits his/her changes and the code is then reviewed before it is included in a stable release, while changes to development releases are reviewed after being included in the release. Managing releases. A core team member volunteers to be the release manager and makes the decisions pertaining to the individual release. He or she delineates the scope of the release by making sure that all open requests and problems are resolved and restricts access to the code repository to avoid any more changes [START_REF] Mockus | Two case studies of open source software development: Apache and Mozilla[END_REF].

These development cycle stages draw a parallel to the Scrum Agile development methodology, where a product owner creates a backlog, a prioritized list of functional and non-functional requirements for building into the product. Development is performed in sprints which are 30 day iterations of development activities, which include only the highest priority backlog requirements that can be successfully completed in the allotted time [START_REF] Schwaber | Agile Project Management with Scrum[END_REF].

Most FLOSS development practices are very similar to Scrum Agile development methods but less structured and more ad hoc.

Software Quality

The origins of software quality can be traced back to industrial engineering and operations management and their development of product quality concepts and quality management practices. For these fields, quality is adherence to process specification [START_REF] Deming | Quality, Productivity, and Competitive Position[END_REF], [START_REF] Deming | Out of the Crisis[END_REF] in order to produce a product that meets customer requirements with zero defects [START_REF] Juran | Planning for Quality[END_REF], [START_REF] Crosby | Quality is Free: The Art of Making Quality Certain[END_REF]. In order to achieve this goal, approaches such as TQM (total quality management) [START_REF] Feigenbaum | Total quality control: engineering and management: the technical and managerial field for improving product quality, including its reliability, and for reducing operating costs and losses[END_REF], [START_REF] Ishikawa | What is total quality control? The Japanese way[END_REF] were developed to integrate quality into all company activities and Six Sigma to measure for quality [START_REF] Tennant | Six Sigma: SPC and TQM in manufacturing and services[END_REF].

Industrial engineering and operations management's view of quality can be categorized as the manufacturing, user, and product approaches to quality as described by [START_REF] Garvin | What does 'Product Quality' really mean?[END_REF]. Table 1 summarizes these definitions of quality as well as others (transcendent and value). As well as categorizing definitions of quality, Garvin also categorized the eight dimensions of product quality (Table 2) [START_REF] Garvin | What does 'Product Quality' really mean?[END_REF]. [START_REF] Cavano | A Framework for the Measurement of Software Quality[END_REF], [START_REF] Mccall | Factors in Software Quality[END_REF] which are all listed in Table 2.

Both of these frameworks of factors left out the measure (actual thing that is counted) for each factor. Each implementer and developer was left to define his or her own metrics and criteria for each factor. The ISO 9126 [30] Information technology -Software product evaluation: quality characteristics and guidelines for their use, which is part of the ISO 9000 set of standards by the ISO (The International Organization for Standardization) for quality management, was an attempt to standardize the quality factors to six main factors with three sub-factors under each one. While the quality factor frameworks were used to assess the software product, process frameworks were developed to assess the quality of the process producing the software and to accommodate the manufacturing definition of quality [START_REF] Garvin | What does 'Product Quality' really mean?[END_REF]. One such framework is the CMMI (Capability Maturity Model Integration), which is a process improvement framework that can be used to drive organizational change and to judge the process maturity of another organization. CMMI has five levels, with one being the lowest. A level five organization is one where processes are defined (level 3), quantitatively managed (level 4), and are continually being optimized (level 5). This maturity model lists the processes that an organization should have in order to be considered at a certain CMMI level but it leaves the details of how to put them into place up to the organization.

Quality factor frameworks come close to Garvin's product definition of quality because they distill quality into a set of measurable characteristics while the process maturity models most closely resemble Garvin's manufacturing definition of quality because they define quality by evaluating how close an organization's processes meet a predetermined specification. Garvin's user definition of quality, on the other hand, is hard to implement for commercially offered software because a user's satisfaction or rather dissatisfaction with a software's features or performance cannot be immediately addressed. Rather, user satisfaction must be bundled with the problem resolutions and new feature requests of all other users into a new release, patch, or service pack, which are infrequently issued due to their cost. Because of this limitation, software quality has become Garvin's value definition of quality: conformance to specification at an acceptable cost.

In contrast, FLOSS software quality most closely fits Garvin's user definition of quality. Users can directly log problem reports and new functionality requests directly into the software project's issue tracking system that is used by developers. Because FLOSS has frequent releases, those requests can become part of the software much more quickly than commercially offered software, thus better satisfying users.

Methodology

In order to answer the research question, how is quality defined in the FLOSS literature, we performed a literature review. Since the definition of quality is very subjective, we adopted an interpretive approach [START_REF] Walsham | The Emergence of Interpretivism in IS Research[END_REF] to this review by applying a grounded theory methodology [START_REF] Strauss | Basics of Qualitative Research: Grounded Theory Procedures and Techniques[END_REF].

We used the Straussian type of grounded theory in order to allow previous theories and our own interpretations of quality to guide the data collection and analysis [START_REF] Strauss | Grounded Theory Methodology -An Overview[END_REF].

Data Collection

In order to comply with the theoretical sampling necessary in grounded theory, we searched Google Scholar for journal articles and conference papers containing the terms -open source‖ and -quality‖. We retained papers that met the following criteria: explicit definition of quality and empirical validation of the quality definition. We decided these criteria would provide a relevant sample because the authors of these papers would have to explicitly define quality and operationalize it in order to empirically validate it.

This process left us with 24 papers, to which we then added 16 from the quality and defect-fixing categories in [START_REF] Aksulu | A Comprehensive Review and Synthesis of Open Source Research[END_REF] that met the above stated criteria. This left us with 40 papers that defined quality and performed some form of empirical validation of that definition.

Analysis

The papers of this literature review were analyzed using open, axial, and selective coding [START_REF] Strauss | Basics of Qualitative Research: Grounded Theory Procedures and Techniques[END_REF]. As the papers were read, they were coded using open coding Text segments from each of the papers were highlighted and labeled with a code to categorize and conceptualize the data.

The open coding phase produced 75 codes, which were used to label 637 text segments from the 40 papers gathered. The codes reflected how the authors defined quality, the measures used to operationalize it, the research methods used to analyze it, and the characteristics considered in the FLOSS projects that were used to validate their definition of quality.

The axial coding phase produced five codes which were categories containing the codes from the open coding phase. From the 75 codes from the open coding phase, four were discarded because they labeled few text segments and did not help explain how quality is interpreted in FLOSS research.

Table 3 show the categories produced from the axial coding phase. The categories were based on how the authors approached the research, how they analyzed the data, and how they defined and operationalized quality, with the two main categories being quality as a process and quality as a product. The final category deals with the type of data sampled by the authors to validate their models, in this case, the characteristics of the FLOSS projects they examined. These categories were chosen because they follow the research process: an approach must be chosen along with an analysis method; the phenomenon of interest must be operationalized and finally, the data sample must be chosen.

The final phase, the selective coding phase, produced and integrated category that narrated the conceptualization of quality by FLOSS researchers. This phase was complete when theoretical saturation was reached, meaning, not new conceptualizations could be obtained from the data.

Findings

In this section, we describe how researchers interpret quality in FLOSS publications.

Quality as a product

These studies defined quality as structural code quality [START_REF] Stamelos | Code Quality Analysis in Open Source Software Development[END_REF], [START_REF] Spinellis | A Tale of Four Kernels[END_REF], [START_REF] Capra | An Empirical Study on the Relationship among Software Design Quality, Development Effort, and Governance in Open Source Projects[END_REF], [START_REF] Conley | Design for quality: The case of Open Source Software Development[END_REF], [START_REF] Koru | Comparing high-change modules and modules with the highest measurement values in two large-scale open-source products[END_REF], [START_REF] Gyimothy | Empirical validation of object-oriented metrics on open source software for fault prediction[END_REF], [START_REF] Koru | Identifying and characterizing change-prone classes in two large-scale open-source products[END_REF]. Metrics that are used to measure structural quality are number of statements [START_REF] Stamelos | Code Quality Analysis in Open Source Software Development[END_REF], [START_REF] Spinellis | A Tale of Four Kernels[END_REF], cyclomatic complexity [START_REF] Stamelos | Code Quality Analysis in Open Source Software Development[END_REF], [START_REF] Spinellis | A Tale of Four Kernels[END_REF], [START_REF] Capra | An Empirical Study on the Relationship among Software Design Quality, Development Effort, and Governance in Open Source Projects[END_REF], [START_REF] Koru | Comparing high-change modules and modules with the highest measurement values in two large-scale open-source products[END_REF], [START_REF] Gyimothy | Empirical validation of object-oriented metrics on open source software for fault prediction[END_REF], [START_REF] Koch | Exploring the Effects of Process Characteristics on Product Quality in Open Source Software Development[END_REF], [START_REF] Yu | Maintainability of the kernels of open-source operating systems: A comparison of Linux with FreeBSD, NetBSD, and OpenBSD[END_REF], number of nesting levels [START_REF] Stamelos | Code Quality Analysis in Open Source Software Development[END_REF], [START_REF] Spinellis | A Tale of Four Kernels[END_REF], [START_REF] Koru | Comparing high-change modules and modules with the highest measurement values in two large-scale open-source products[END_REF], [START_REF] Barbagallo | The Impact of Social Networking on Software Design Quality and Development Effort in Open Source Projects[END_REF], Halstead volume [START_REF] Stamelos | Code Quality Analysis in Open Source Software Development[END_REF], [START_REF] Spinellis | A Tale of Four Kernels[END_REF], [START_REF] Yu | Maintainability of the kernels of open-source operating systems: A comparison of Linux with FreeBSD, NetBSD, and OpenBSD[END_REF], coupling [START_REF] Spinellis | A Tale of Four Kernels[END_REF], [START_REF] Capra | An Empirical Study on the Relationship among Software Design Quality, Development Effort, and Governance in Open Source Projects[END_REF], [START_REF] Conley | Design for quality: The case of Open Source Software Development[END_REF], [START_REF] Koch | Exploring the Effects of Process Characteristics on Product Quality in Open Source Software Development[END_REF], [START_REF] Barbagallo | The Impact of Social Networking on Software Design Quality and Development Effort in Open Source Projects[END_REF], coding style [START_REF] Spinellis | A Tale of Four Kernels[END_REF], statements per function, files per directory, percentage of numeric constants in operands [START_REF] Spinellis | A Tale of Four Kernels[END_REF], growth of LOC (lines of code) [START_REF] Koru | Comparing high-change modules and modules with the highest measurement values in two large-scale open-source products[END_REF], modularity [START_REF] Paulson | An empirical study of open-source and closed-source software products[END_REF], average coupling between objects, cohesion, number of children, depth of inheritance tree, methods inheritance factor and other internal software structure metrics [START_REF] Stamelos | Code Quality Analysis in Open Source Software Development[END_REF], [START_REF] Spinellis | A Tale of Four Kernels[END_REF], [START_REF] Capra | An Empirical Study on the Relationship among Software Design Quality, Development Effort, and Governance in Open Source Projects[END_REF], [START_REF] Conley | Design for quality: The case of Open Source Software Development[END_REF], [START_REF] Koru | Comparing high-change modules and modules with the highest measurement values in two large-scale open-source products[END_REF], [START_REF] Gyimothy | Empirical validation of object-oriented metrics on open source software for fault prediction[END_REF], [START_REF] Yu | Maintainability of the kernels of open-source operating systems: A comparison of Linux with FreeBSD, NetBSD, and OpenBSD[END_REF].

The idea behind measuring software code structure is that well-designed software is less complex, less likely to contain faults, and easier to maintain [START_REF] Yu | Maintainability of the kernels of open-source operating systems: A comparison of Linux with FreeBSD, NetBSD, and OpenBSD[END_REF].

Measuring code structure left the researchers with more questions than answers. The most successful projects in terms of number of downloads and popularity were not the ones with the highest structural quality [START_REF] Yu | Maintainability of the kernels of open-source operating systems: A comparison of Linux with FreeBSD, NetBSD, and OpenBSD[END_REF], [START_REF] Barbagallo | The Impact of Social Networking on Software Design Quality and Development Effort in Open Source Projects[END_REF]. Another study found that the software modules with the highest rate of change were not the ones with the highest structural complexity [START_REF] Koru | Comparing high-change modules and modules with the highest measurement values in two large-scale open-source products[END_REF]. Even using machine learning algorithms with structural quality measures in order to predict faults did not produce clear results [START_REF] Gyimothy | Empirical validation of object-oriented metrics on open source software for fault prediction[END_REF].

Comparing structural quality between open and closed projects produced mixed results with some studies finding that FLOSS projects had quality comparable to closed projects [START_REF] Samoladas | Open source software development should strive for even greater code maintainability[END_REF] while others found that open source software did not prove to have structural code quality higher than commercial software [START_REF] Stamelos | Code Quality Analysis in Open Source Software Development[END_REF], [START_REF] Spinellis | A Tale of Four Kernels[END_REF].

Using structural quality to define, measure, and compare FLOSS quality has not proven effective with different researchers achieving different results even when using the same metrics.

Quality as a process

Defect Fixing

Defect fixing [START_REF] Paulson | An empirical study of open-source and closed-source software products[END_REF], [START_REF] Kuan | Open Source Software as Lead-User's Make or Buy Decision: A Study of Open and Closed Source Quality[END_REF], [START_REF] Koru | Comparing high-change modules and modules with the highest measurement values in two large-scale open-source products[END_REF], [START_REF] Gyimothy | Empirical validation of object-oriented metrics on open source software for fault prediction[END_REF], [START_REF] Ghapanchi | Measuring the Effectiveness of the Defect-Fixing Process in Open Source Software Projects[END_REF], [START_REF] Kidane | Correlating temporal communication patterns of the Eclipse open source community with performance and creativity[END_REF], [START_REF] Au | Virtual organizational learning in open source software development projects[END_REF], [START_REF] Crowston | Bug fixing practices within free/libre open source software development teams[END_REF], [START_REF] Koru | Defect handling in medium and large open source projects[END_REF], [START_REF] Glance | Release Criteria for the Linux Kernel[END_REF], [START_REF] Huntley | Organizational learning in open-source software projects: an analysis of debugging data[END_REF], [START_REF] Sohn | A strategic analysis for successful open source software utilization based on a structural equation model[END_REF], [START_REF] Zhou | Open source software reliability model: an empirical approach[END_REF], [START_REF] Zhao | Quality assurance under the open source development model[END_REF], [START_REF] Aberdour | Achieving Quality in Open Source Software[END_REF], [START_REF] Halloran | High Quality and Open Source Software Practices[END_REF], [START_REF] Michlmayr | Quality Practices and Problems in Free Software Projects[END_REF], [START_REF] Mockus | A Case Study of Open Source Software Development: The Apache Server[END_REF], [START_REF] Samoladas | The SQO-OSS Quality Model: Measurement Based Open Source Software Evaluation[END_REF], [START_REF] Deprez | Comparing Assessment Methodologies for Free/Open Source Software: OpenBRR and QSOS[END_REF], [START_REF] Crowston | Information Systems Success in Free and Open Source Software Development: Theory and Measures[END_REF] is by far the most popular definition of quality as a process in the FLOSS quality literature. Authors have done studies defining the what constitutes the process itself in order to determine how it works [START_REF] Koru | Comparing high-change modules and modules with the highest measurement values in two large-scale open-source products[END_REF], [START_REF] Crowston | Bug fixing practices within free/libre open source software development teams[END_REF], [START_REF] Koru | Defect handling in medium and large open source projects[END_REF], [START_REF] Glance | Release Criteria for the Linux Kernel[END_REF], [START_REF] Huntley | Organizational learning in open-source software projects: an analysis of debugging data[END_REF], [START_REF] Sohn | A strategic analysis for successful open source software utilization based on a structural equation model[END_REF], [START_REF] Zhou | Open source software reliability model: an empirical approach[END_REF], [START_REF] Zhao | Quality assurance under the open source development model[END_REF], [START_REF] Aberdour | Achieving Quality in Open Source Software[END_REF], [START_REF] Crowston | Information Systems Success in Free and Open Source Software Development: Theory and Measures[END_REF] and developed models to test its effectiveness [START_REF] Ghapanchi | Measuring the Effectiveness of the Defect-Fixing Process in Open Source Software Projects[END_REF], [START_REF] Wray | Evaluating the performance of open source software projects using data envelopment analysis[END_REF].

They have approach it in terms of total bugs fixed [START_REF] Conley | Design for quality: The case of Open Source Software Development[END_REF], [START_REF] Crowston | Bug fixing practices within free/libre open source software development teams[END_REF], [START_REF] Koru | Defect handling in medium and large open source projects[END_REF], [START_REF] Mockus | A Case Study of Open Source Software Development: The Apache Server[END_REF], [START_REF] Samoladas | The SQO-OSS Quality Model: Measurement Based Open Source Software Evaluation[END_REF], [START_REF] Crowston | Information Systems Success in Free and Open Source Software Development: Theory and Measures[END_REF] and speed of bug resolution [START_REF] Kuan | Open Source Software as Lead-User's Make or Buy Decision: A Study of Open and Closed Source Quality[END_REF], [START_REF] Kidane | Correlating temporal communication patterns of the Eclipse open source community with performance and creativity[END_REF], [START_REF] Au | Virtual organizational learning in open source software development projects[END_REF], [START_REF] Huntley | Organizational learning in open-source software projects: an analysis of debugging data[END_REF], [START_REF] Crowston | Information Systems Success in Free and Open Source Software Development: Theory and Measures[END_REF]. These approaches take into account the evolving nature of FLOSS, which is never truly finished, but rather, remains in a permanently emerging state. It also considers that FLOSS testing and defect reporting and fixing is a community activity where developers, users, and periphery members collaborate to create the software.

However, results using this approach have been mixed. Some open source projects resolved service requests more quickly than their closed counterparts, others did not [START_REF] Kuan | Open Source Software as Lead-User's Make or Buy Decision: A Study of Open and Closed Source Quality[END_REF]. For other studies, software type (database, financial, game, networking) made more of a difference in determining defect resolution speed along with number of developers (groups with less than 15 developers were the most efficient) [START_REF] Au | Virtual organizational learning in open source software development projects[END_REF]. The main difference with closed software is that in most FLOSS projects, bugs are only addressed after feedback is received from users. There is no way to measure the quality of a release pre hoc, only ad hoc [START_REF] Glance | Release Criteria for the Linux Kernel[END_REF]. However, this attitude is changing with projects such as GNOME, Debian, and KDE forming their own Quality Assurance teams and enforcing quality assurance tasks [START_REF] Michlmayr | Quality Practices and Problems in Free Software Projects[END_REF].

Even though the defect fixing approach to measure FLOSS quality considers the evolving nature of its quality, research using it has not operationalize it in an evolutionary manner. Most studies have looked at the bug databases of FLOSS projects cumulatively after a certain amount of time (i.e. after six months of activity) rather than looking at defect resolution rates per release (except for [START_REF] Gyimothy | Empirical validation of object-oriented metrics on open source software for fault prediction[END_REF], [START_REF] Zhou | Open source software reliability model: an empirical approach[END_REF], [START_REF] Mockus | A Case Study of Open Source Software Development: The Apache Server[END_REF] which did compare product releases), which is the evolutionary cycle of FLOSS software. The studies that looked at defects per release found that FLOSS has lower post-feature test defects than commercial software but higher post-release defects than commercial software [START_REF] Mockus | A Case Study of Open Source Software Development: The Apache Server[END_REF] and that release software quality is cyclical, with the Mozilla 1.2 showing a major decrease in quality, which was improved in later releases [START_REF] Gyimothy | Empirical validation of object-oriented metrics on open source software for fault prediction[END_REF].

The cyclical nature of FLOSS quality is illustrated in a study that showed that bug arrival rates follow a bell curve through time between releases. Whenever drastic changes were introduced to the software, the rate would also drastically change [START_REF] Zhou | Open source software reliability model: an empirical approach[END_REF]. This would suggest that defect rates and thus defect resolution rates vary across releases depending on the changes being introduced. If the release introduces new features, or makes major changes to the architecture of the software, many defects will be introduced, while those that simply introduce defect fixes and enhancements, will introduce less.

Another issue stems from the definition of bug. Bug reports in bug databases in FLOSS project management Web sites could include anything from -failures, faults, changes, new requirements, new functionalities, ideas, and tasks‖ [START_REF] Koru | Defect handling in medium and large open source projects[END_REF]. Not to mention duplicate bug reports, poorly defined ones, and those that are out of scope with the product [START_REF] Halloran | High Quality and Open Source Software Practices[END_REF]. This happens because bug reporting systems are usually open to the public, and users without enough technical skills will make mistakes writing the bug reports [START_REF] Michlmayr | Quality Practices and Problems in Free Software Projects[END_REF].

Measuring defect-fixing effectiveness in FLOSS projects has provided mixed results because different studies have defined and thus measured defects or bugs differently. They have also calculated defect-fixing rates by looking at bug tracking databases in the cumulative, without considering that defects are introduced and fixed cyclically in FLOSS, per release.

Other Processes

FLOSS projects tend to rely on tools to enforce policies and standards [START_REF] Halloran | High Quality and Open Source Software Practices[END_REF]. Such tools include defect tracking systems, version control, mailing lists, automatic builds, etc. [START_REF] Aberdour | Achieving Quality in Open Source Software[END_REF], [START_REF] Halloran | High Quality and Open Source Software Practices[END_REF], [START_REF] Michlmayr | Quality Practices and Problems in Free Software Projects[END_REF].

One quality assurance activity that is performed in traditional software development, peer review, is done differently in FLOSS projects with successful results. Peer reviews in open source were more efficient because there is no time wasted scheduling meeting since people work asynchronously and have more detailed discussions [START_REF] Rigby | Open source software peer review practices: a case study of the apache server[END_REF]. An example from the Apache project shows that it has three types of peer review procedures, depending on the experience and trustworthiness of the developer [START_REF] Rigby | Open source software peer review practices: a case study of the apache server[END_REF].

Despite the successful inclusion of peer reviews into the FLOSS development process, testing procedures have not managed to make the necessary cross over into FLOSS. Most projects do not have a baseline test suite to support testing, this means no regression testing can be performed [START_REF] Zhao | Quality assurance under the open source development model[END_REF].

Developers perform their unit testing and do sometimes better than commercial software [START_REF] Mockus | A Case Study of Open Source Software Development: The Apache Server[END_REF], but it is up to the users to discover bugs and defects which could be eliminated with system or integrated testing.

Process Maturity Models

Maturity assessment models have been formulated to help users and integrators evaluate the quality of a FLOSS project versus another [START_REF] Samoladas | The SQO-OSS Quality Model: Measurement Based Open Source Software Evaluation[END_REF], [START_REF] Deprez | Comparing Assessment Methodologies for Free/Open Source Software: OpenBRR and QSOS[END_REF], [START_REF] Del Bianco | The QualiSPo approach to OSS product quality evaluation[END_REF], [START_REF] Soto | The QualOSS open source assessment model measuring the performance of open source communities[END_REF], [START_REF] Deprez | Defining Software Evolvability from a Free/Open-Source Software Perspective[END_REF], [START_REF] Glott | Quality Models for Free/Libre Open Source Software--Towards the[END_REF], [START_REF] Groven | Security measurements within the framework of quality assessment models for free/libre open source software[END_REF], [START_REF] Michlmayr | Software Process Maturity and the Success of Free Software Projects[END_REF]. These models provide a set of criteria to evaluate a FLOSS project. Different models concentrate on different criteria, but they all provide a way to quantify and evaluate the quality of a FLOSS product.

Maturity models use organizational trustworthiness as a proxy for product trustworthiness and thus quality-if the product is built correctly, it will then have high quality.

The assessments are mostly for the FLOSS integrator who must assess the risk of adding the FLOSS product to his or her existing architecture. The assessment models are not predictive (they do not evaluate the factors that lead to quality, nor do they provide a construct for quality) they simply provide a set of criteria with different scores that the integrator can then use to make the decision to adopt the FLOSS product.

Modularity as the enabler to FLOSS quality

FLOSS's paradox of having and adding more developers without compromising its productivity (in contrast to Brook's law that says that adding more developers increases coordination costs and decreases performance) is due to its approach to modularity. A FLOSS project is made up of many subprojects where only a few developers work together without ever having to interact with the developers in other subprojects or modules [START_REF] Schweik | Brooks' Versus Linus' Law: An Empirical Test of Open Source Projects[END_REF].

It is believed this is the reason that projects such as Linux and Apache are considered successful. They have been able to scale because of their modularity. Because of modularity, defects in one module, do not affect the rest [START_REF] Mockus | A Case Study of Open Source Software Development: The Apache Server[END_REF].

However, there is no single definition of modularity. The studies that have defined and measured it do so differently. One study used -correlation between functions added and functions modified‖ to measure modularity. It then compared modularity across a set of open and closed projects. The open projects did not prove to be more modular than the closed ones [START_REF] Paulson | An empirical study of open-source and closed-source software products[END_REF].

Another study used average component size, which was measured as program length (sum of the number of unique operands and operators) divided by number of statements. The study found that applications with smaller average component size received better user satisfaction scores [START_REF] Stamelos | Code Quality Analysis in Open Source Software Development[END_REF].

In terms of influencing quality, modularity has produced mixed results. A study found that higher modularity does not lead to higher quality. This study defined modularity as the distance of each package in a release from the main sequence. Because higher modularity is associated with reduced software complexity, it should result in higher structural code quality, but the authors found that the projects with higher modularity contained the greater number of defects [START_REF] Conley | Design for quality: The case of Open Source Software Development[END_REF].

Yet another study contests that it is small component design that leads to low defect density, higher user satisfaction, and easier maintenance and evolution [START_REF] Aberdour | Achieving Quality in Open Source Software[END_REF].

Work distribution is another way of conceptualizing modularity. In another study, the authors found that a lower concentration of developers making changes to a module led to higher quality for the module. The authors speculated that this could explain the FLOSS paradox of many developers and high productivity: at the project level there could be many developers, but within the project, they should be organized into small teams; this would keep the concentration of authors to code low, thus fostering simpler code, higher quality and better maintainability [START_REF] Koch | Exploring the Effects of Process Characteristics on Product Quality in Open Source Software Development[END_REF].

The literature has defined and operationalized modularity differently using either software structure measures (component size, distance from main sequence, etc.) or development organizational measures such as author concentration per class, number of authors per module, etc.

The development organizational measures have proven to be more effective at finding a correlation between quality and modularity, but these measures are still vaguely defined and more research needs to be performed to optimally define them and operationalize them in order to produce a universal measure of modularity.

Characteristics of samples

The researchers seem to have made very arbitrary choices when it came to choosing FLOSS project to make up their samples.

By far, the most popular place to obtain data for FLOSS quality studies is the SourceForge repository maintained by Notre Dame University. But it was not the only place to find data; some case studies concentrated on popular projects that are not hosted by SourceForge such as Apache [START_REF] Rigby | Open source software peer review practices: a case study of the apache server[END_REF], [START_REF] Mockus | A Case Study of Open Source Software Development: The Apache Server[END_REF], Linux [START_REF] Yu | Maintainability of the kernels of open-source operating systems: A comparison of Linux with FreeBSD, NetBSD, and OpenBSD[END_REF], and Eclipse [START_REF] Kidane | Correlating temporal communication patterns of the Eclipse open source community with performance and creativity[END_REF]. Some projects considered software type a defining factor and only looked at projects of the same type [START_REF] Koru | Identifying and characterizing change-prone classes in two large-scale open-source products[END_REF], [START_REF] Koch | Exploring the Effects of Process Characteristics on Product Quality in Open Source Software Development[END_REF], [START_REF] Yu | Maintainability of the kernels of open-source operating systems: A comparison of Linux with FreeBSD, NetBSD, and OpenBSD[END_REF], [START_REF] Wray | Evaluating the performance of open source software projects using data envelopment analysis[END_REF], while the rest did not consider it a factor. However one study did consider it and found that project category affects the bug resolution time [START_REF] Au | Virtual organizational learning in open source software development projects[END_REF].

The development time of the projects examined by the authors was extremely variable. There were studies that examined FLOSS project data that covered development time for one week [START_REF] Crowston | Bug fixing practices within free/libre open source software development teams[END_REF], 105 weeks [START_REF] Huntley | Organizational learning in open-source software projects: an analysis of debugging data[END_REF], four months [START_REF] Halloran | High Quality and Open Source Software Practices[END_REF], six months [START_REF] Kidane | Correlating temporal communication patterns of the Eclipse open source community with performance and creativity[END_REF], etc. with one project capturing data from initial commit until the last commit before the stable version was released [START_REF] Koch | Exploring the Effects of Process Characteristics on Product Quality in Open Source Software Development[END_REF].

Another factor that varied across the studies was the number of projects examined. From four [START_REF] Crowston | Bug fixing practices within free/libre open source software development teams[END_REF], to 52 [START_REF] Koru | Defect handling in medium and large open source projects[END_REF], all the way to 140 [START_REF] Crowston | Information Systems Success in Free and Open Source Software Development: Theory and Measures[END_REF], and beyond.

Another factor used to choose candidate was the success of the project measured in popularity terms such as number of downloads [START_REF] Crowston | Information Systems Success in Free and Open Source Software Development: Theory and Measures[END_REF] and SourceForge rank [START_REF] Barbagallo | The Impact of Social Networking on Software Design Quality and Development Effort in Open Source Projects[END_REF], [START_REF] Michlmayr | Software Process Maturity and the Success of Free Software Projects[END_REF], which uses number of downloads and recommendations. Researchers refrained from including failed projects and only looked at those that high success measures.

The way researchers are choosing their samples is definitely a reason why there are mixed results in the FLOSS quality literature. They are looking at different types of projects, examining them for different amounts of time, and only considering popular projects.

Summary

Quality is very subjective and hard to define absolutely. With this challenge, FLOSS researchers have used many ways to define quality. They have used product and process metrics and have found mixed results. FLOSS software is always evolving and one version might produce more defects than a pervious one because of some major change in the software or the community structure.

Successful projects are those that have adopted a modular organization of their code and their community, allowing them to grow and isolate defects. They have also implemented tools to automate policy enforcement and adapted traditional software development practices to their context.

There is a need to evaluate the quality of FLOSS projects, and maturity assessment models have emerged to meet this need. However, they are hard to automate, and their scores are hard to interpret.

An important reason as to why researchers have obtained mixed results in researching FLOSS quality is that their samples have different characteristics in terms of number of projects examined, software type, time evaluated, and popularity of projects examined.

Discussion

The reviewed papers show that there is a need to define and quantify quality in FLOSS development projects in order to compare them among each other and to traditionally developed software. Identifying projects that produce high quality products will lead to further research into understanding the factors that lead to higher quality and the interaction of those factors in FLOSS development projects.

The development of assessment models to ascertain quality comes from the position in traditionally developed software that established and repeatable processes lead to the development of quality products.

FLOSS Quality as evolving

With each release, the FLOSS software and its community change. Quality is not linear: the tenth release of a software product might not have fewer defects than its first. It all depends on what type of release it is; whether it is adding new features, restructuring the entire product, restructuring the way it is developed, or simply posting defect fixes. Which type of release of the product is a more important determinant of its quality than its software structure, or its number of developers.

This explains the mixed results obtained from research that only used product measures as a measure of quality -the modules with the highest change rate and the highest number of defects were not those with the lowest design quality or complexity [START_REF] Koru | Comparing high-change modules and modules with the highest measurement values in two large-scale open-source products[END_REF].

Quality as Defect Resolution Rate

Number of defects added by a release divided by the number of lines of code added by the release would seem a good measure of software product quality [START_REF] Mockus | A Case Study of Open Source Software Development: The Apache Server[END_REF] that would allow comparison between open and closed software products.

But this assumption is wrong because the release of a commercial closed software product is not the same as the one from an open source project. An open source product release resembles the commercial software after its feature test, since there is neither system nor regression testing in open source projects [START_REF] Mockus | A Case Study of Open Source Software Development: The Apache Server[END_REF].

These measures do not take into account FLOSS projects' community development. That is why a better measure of FLOSS quality is defect resolution rate, in terms of number of bugs resolved and average time of bug resolution. These measures not only show the quality of the code but also the community's effectiveness at achieving quality.

A key issue is to define bugs as defects in the software product. Bug databases, which are used to calculate the defect resolution rates, are riddled with non-bugs, which must not be taken into account when calculating these rates.

Modularity as Driver of Quality

The -many eyeballs‖ looking at the bugs include core developers, periphery developers, sometime contributors, and users, who can easily find their way to the project's publicly available bug tracking system. This group has activity rates, contribution amounts, contributions included per release, problem reports contributed, problem reports resolved, and download statistics. These are all metrics of the community's quality efforts.

But making sure that these community members can work effectively with each other is very necessary. A modular architecture of the code and the community allows a project to grow and attract new developers without having the defects of one group affect another group.

That is why modularity needs to be defined in terms of technical modularity (the coupling of the modules) and organizational modularity (the coupling of the module managers/owners and the core project manager) [START_REF] Tiwana | The Influence of Software Platform Modularity on Platform Abandonment: An Empirical Study of Firefox Extension Developers[END_REF].

However, a downside of modularity is that if a member leaves, his or her module might become orphaned. That is why projects such as Debian are developing their own -quality assurance‖ groups (http://qa.debian.org), where anyone interested can join and help with mass bug filing and transitions, track orphan code, etc.

It seems that too much modularity might be bad for quality in the long run. It is important to do more research to understand what the right amount of modularity looks like.

Process and Product as Drivers of Quality

Product and process requirements, the traditional specific quality requirements [START_REF] Glinz | On Non-Functional Requirements[END_REF] are still relevant in driving quality in FLOSS products: they are universally understood and any project community still needs to reach for them. But because of this, they are sometimes taken for granted; their inclusion needs to be aided by automation tools, such as testing tools included in the automatic build software that calculates and posts correctness and reliability metrics and compares them with benchmark numbers, alerting the community members if their software product falls below the thresholds.

Relying on the -many eyeballs‖ to report and fix defects has helped FLOSS achieve quality, but there is something to be said for automating the process in order to produce a higher quality product before it is released.

FLOSS requires its own Maturity Model for Quality

The development of maturity models such as QualOSS, QSOS, OpenBRR, shows the need for a process evaluation model like CMMI but for FLOSS.

This means that quality could also be defined in terms of this process maturity model, but for this approach to reach maturity (so that one day we might have level 5 FLOSS projects) more research needs to occur to define, if not the ideal, the most effect FLOSS development processes.

Conclusion

Just like in traditionally developed software, there is little consensus in the FLOSS literature when it comes to defining quality.

Linux and Apache are by far the most studied projects in FLOSS literature. All the reviewed papers studied projects that they considered successful: they had released several versions, and had high popularity rating, and download numbers. However, failed projects also need to be studied in order to determine what led to their downfall.

FLOSS communities and their software product are emergent and need to a measure of quality that will reflect their nature. Defect resolution rates (amount of defects resolved, speed of resolution) are the best way to measure a community's commitment to quality, because they recognize that FLOSS is not a static product, but ever evolving. These rates should be calculated per release, and not cumulatively, because the cycle of FLOSS evolution is the release. Researchers should be careful to only include defects and not new feature requests, duplicates, or poorly reported bugs into their calculations.

Modularity is being touted as the main driver of FLOSS quality success, but it needs to be further defined and studied in order to understand how it works.

Table 1 .

 1 Garvin's quality definitions[START_REF] Garvin | What does 'Product Quality' really mean?[END_REF].

	Approach	Definition of Quality
	Transcendent	Innate excellence that cannot be defined, only recognized through experience.
	Product	Discrete and measurable product characteristics.
	User	Subjective consumer satisfaction.
	Manufacturing	Conformance to specification.
	Value	Conformance to specification at an acceptable cost or price.
	With this legacy from industrial engineering and operations management, software
	quality started with the product definition of quality by defining frameworks of
	factors. The most popular were Boehm's model of 23 factors (dimensions of quality)
	[27] and McCall's with 11 factors

Table 2 .

 2 Quality Factors

	Model	Factors
	McCall	Accessibility, Accountability, Accuracy, Augmentability,
		Communicativeness, Completeness, Conciseness, Consistency, Device-
		independence, Efficiency, Human engineering, Legibility, Maintainability,
		Modifiability, Portability, Reliability, Robustness, Self-containedness, Self-
		descriptiveness, Structuredness, Testability, Understandability, Usability
	Boehm	Correctness, Efficiency, Flexibility, Integrity, Interoperability ,
		Maintainability, Portability, Reliability, Reusability, Testability, Usability
	ISO 9126 Efficiency, Functionality, Maintainability, Portability, Reliability, Usability
	Garvin	Aesthetics, Conformance, Durability, Features, Perceived quality,
		Performance, Reliability, Serviceability

Table 3 .

 3 Categories from Axial Coding

	Category	Description	Sample of codes within category
	Research	Approach used to analyze quality	Case study
	approach	in research study	Survey
			Factor model
			Maturity model
	Analysis method Methodology used to analyze data Regression
			Structural equation model
			Machine Learning
			Social Network Analysis
	Quality as a	Quality operationalized as	Defect fixing rate
	process	processes that can be measured.	Defect fixing time
			Definition of bug
			Quality assurance procedures
			Process metrics
	Quality as a	Quality operationalized as	Product metrics
	product	characteristics of final software	Number of post-release defects
		product.	Cyclomatic complexity
			Halstead Volume
			CBO (coupling between objects)