Apostolos Kritikos
email: akritiko@csd.auth.gr

Ioannis Stamelos
email: stamelos@csd.auth.gr

Open Source Software Resilience Framework

Keywords: open source software, software resilience, software engineering, software quality, software metrics

niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Background

Open Source Software (OSS) has been continuously growing and evolving for over two decades now. It started as a revolutionary software engineering approach, producing software that was freely and openly available to be used, edited, copied, even commercially utilized. Since then, it has been fueling the development process of companies with source code, testing and bug fixing.

Moreover it has been one of the main drivers for the birth of healthy, successful companies [START_REF] Weber | The success of open source[END_REF], [START_REF] Thakker | The Money In Open-Source Software[END_REF].

In the beginning, the focus about OSS revolved mainly around its development style. Eric Raymond in [START_REF] Raymond | The cathedral and the bazaar[END_REF] compares closed source development to a Cathedral because of its structured and concrete definition of work and roles. In contrast, open source development resembles to a bazaar, a place with no strict rules, where people can openly and freely contribute. As Raymond specifically states "No quiet, reverent cathedral-building here rather, the Linux community seemed to resemble a great babbling bazaar of differing agendas and approaches (aptly symbolized by the Linux archive sites, who'd take submissions from anyone) out of which a coherent and stable system could seemingly emerge only by a succession of miracles" meaning that in a time where closed (or proprietary) software development was the norm, an open approach of collectively developed software seemed odd and likely to fail.

The fact that OSS succeeded is a result of several factors. A primary and extensively studied factor is structural quality which led to the proposal of several quality models like ISO25010 [START_REF]2011 Systems and Software Engineering-Systems and Software Quality Requirements and Evaluation (SQuaRE)-System and Software Quality Models[END_REF] for evaluating software in general or Open Source specific ones like OpenBRR [START_REF] Wasserman | The business readiness rating model: an evaluation framework for open source[END_REF]. In [START_REF] Midha | Factors affecting the success of open source software[END_REF] Midha and Palvia identify several other factors outside the scope of source code and software structure. The level of permissiveness of the license under which the OSS is published, community aspects such as number of active developers and end users, language translations are some indicative examples. In [START_REF] Mobile | Open governance index-measuring the true openness of open source projects from android to webkit[END_REF] the authors study Open Source Governance models, another important factor for OSS success.

In [START_REF] José | A review of software quality models for the evaluation of software products[END_REF] Miguel presents the evolution of quality models between 1977 and 2013. The models are further categorized to Basic Models with the aim of evaluating the software product as a whole and Tailored Quality Models (2001-2013) which extend to component evaluation. The latter category includes the OSS specific quality models as well. The study compares and contrasts the models based on their characteristics and concludes that generic quality evaluation models cannot easily be applied in specific cases. On the other hand tailored quality models usually cover the needs of very specific domains limiting their applicability.

In [START_REF] Anthony I Wasserman | Osspal: Finding and evaluating open source software[END_REF] Wasserman describes the evolution of Business Readiness Rating (BRR) model to OSSpal. As Wasserman very aptly states, software quality, maturity, stability, documentation community and so forth vary in different OSS projects and continuously change as those projects evolve. Therefore it is important for OSS evaluation models to include, apart from numerical scores and metrics, qualitative criteria as well.

In [START_REF] Teixeira | Lessons learned from applying social network analysis on an industrial free/libre/open source software ecosystem[END_REF] the authors are combining quality assurance methodologies with social network analysis techniques to study competition and collaboration in large OSS ecosystems. Community wise, in [START_REF] Andrade | Principled evaluation of strengths and weaknesses in floss communities: A systematic mixed methods maturity model approach[END_REF] the authors are proposing a reference model for evaluating the maturity of an OSS project's community. They focus on the socio-technical practices of OSS software development and try to combine characteristics of existing software quality models to this end.

The aforementioned diverse set of factors which are responsible for the success of an OSS dictates the need to approach it as an evolving system in order to be able to study it in a holistic way. Moreover, in order for an OSS to be able to succeed and achieve longevity, it is crucial to be resilient to survive potential stresses and crises that might occur. Such stresses or crises could be related with forks of the project that might drive the attention of the original project's community away, migration of lead developers or even part of the development community to other forks or projects, an unsuccessful major release that might hurt the reputation of the project, changes to the license, migration to another forge and so forth. In [START_REF] Gamalielsson | Sustainability of open source software communities beyond a fork: How and why has the libreoffice project evolved[END_REF] Gamalielsson and Lundell study the case of Libre Office, an OSS project that started as a fork of Open Office, but managed to retain the development community and evolve, as of the time of writing, to a successful OSS project.

In this publication we are trying to adapt the concept of City Resilience, a term found in the scientific field of Urban Planning and Architecture, to OSS evolution. The term resilience is defined in [START_REF] Wieland | The influence of relational competencies on supply chain resilience: a relational view[END_REF] as the ability [of a system] to cope with change. In [START_REF] Warren | Investing in software resiliency[END_REF] the author defines a resilient system as one that can take a hit to a critical component and recover and come back for more in a known, bounded and generally acceptable period of times. In [START_REF] Da | City resilience framework[END_REF] the authors of the City Resilience Framework, which inspired this paper, define city resilience as follows: "city resilience describes the capacity of cities to function, so that the people living and working in cities particularly the poor and vulnerable survive and thrive no matter what stresses or shocks they encounter".

From the aforementioned literature it becomes clear that an OSS project can be approached from several perspectives. The structure of its source code, the community that was built around it (developers, end users, testers), business related aspects (license, business models). In order to be able to holistically evaluate an OSS project we need to study it as a continuously evolving system. In this work we are proposing a framework with the aim of studying the resilience of an OSS project trying to combine several of the aforementioned perspectives.

The rest of the paper is organized as follows. In Section 2 we analyze the City Resilience Framework, a framework designed to measure Urban Resilience, which inspired our work. In Section 3 we present the adaptation of the City Resilience Framework to OSS. In Section 4 we demonstrate the application of the Open Source Resilience Framework to two (2) Open Source projects. In Section 5 we discuss possible threats to validity and, finally, in Section 6 we summarize our conclusions and we discuss ideas for future work.

City Resilience Framework

The City Resilience Framework (CRF), as presented in [START_REF] Da | City resilience framework[END_REF], is the result of research undertaken with the aim of establishing an accessible, evidence-based definition of Urban Resilience by Arup and the Rockefeller Foundation. The CRF is, as of the time of writing, used by the 100 Resilient Cities [16] non profit organization to primarily evaluate the Urban Resilience of more than 90 cities around the world and, additionally, to assist the cities on crises with tailored made resilience strategies.

Trying to tackle the fact that every city is unique the City Resilient Index (CRI) was proposed. It is a set of indicators, variables and metrics that allow cities to understand, baseline and subsequently measure local resilience over time. As the authors of [START_REF]City resilience framework[END_REF] state "The CRI will measure relative performance over time rather than comparison between cities. It will not deliver an overall single score for comparing performance between cities, neither will it provide a world ranking of the most resilient cities."

City Resilience Index

City Resilience Index (CRI) suggests that resilience of a city is related to four (4) key dimensions that are further decomposed to twelve [START_REF] Gamalielsson | Sustainability of open source software communities beyond a fork: How and why has the libreoffice project evolved[END_REF] goals that can help a city achieve resilience. These dimensions and goals are the following:

Adaptation of City Resilience Index to Open Source Software

We argue that Open Source Software projects share a conceptual similarity with cities. They are dynamic and continuously evolving systems with their own structural properties, they attract people that form communities around them which, on a second level, may utilize a governance model. Some OSS projects have commercial activity. As it is happening with cities, OSS projects can face stresses and crises (i.e. developers abandoning the project to work on a fork or users massively migrate to a competitive project).

In this section we are presenting our attempt to adapt the City Resilience Index to Open Source Software projects. We aim in utilizing the models and metrics of the extensive literature on OSS quality, metrics and evaluation and provide a framework that will measure the relative performance of an OSS project over time rather than a comparison between projects.

Open Source Software Resilience Framework (OSSRF)

Following the City Resilience Framework paradigm, Open Source Software Resiliency Framework (OSSRF) is also being primarily structured to four (4) dimensions that are then analyzed in twelve [START_REF] Gamalielsson | Sustainability of open source software communities beyond a fork: How and why has the libreoffice project evolved[END_REF] goals and, on a third level, on a set of indicators for each goal.

Dimensions

As in the CRF we propose four key dimensions related to Open Source Software as follows:

1. Source Code: The first dimension of CRF is Health & Well-being and it is related with people. In Open Source Software we consider source code (i.e. classes) to be the structural unit of the project. In this dimension we will take under consideration aspects like the activity and growth rate of an OSS project along with some other related aspects. 2. Business & Legal: The second dimension of CRF is Economy & Society and is related with organization. In Open Source Software the norm is voluntary work but, more mature projects are utilizing Open Source Business Models to offer commercial services (be it pro features or support). For those types of projects licensing plays a key role when it comes to commercialization.

Integration & Reuse:

The third dimension of CRF is related to place.

Open Source Software projects usually reuse components of other OSS projects or are being reused themselves. In this spirit, in the third dimension of the Open Source Software Resilience Framework we will be dealing with the aspects of integration and reuse. 4. Social (Community): Finally the last dimension of CRF is about Leadership & Strategy and is related with utilizing knowledge from the past to become better and more resilient in the future. In Open Source Software both leadership and strategy related processes are usually connected with the community. Moreover most of the knowledge related to an Open Source Software usually comes from its community activity (i.e. feature proposal, bug reports, translations, documentation, testing and so forth).

Goals

The aforementioned dimensions are further decomposed to the following twelve [START_REF] Gamalielsson | Sustainability of open source software communities beyond a fork: How and why has the libreoffice project evolved[END_REF] [START_REF] Munga | The adoption of open source software in business models: a red hat and ibm case study[END_REF]. 0 is considered a non resilient value. 1.3. Paid support: 0 (no) or 1 (yes) based on whether the OSS project offers a paid plan for support. [START_REF] Munga | The adoption of open source software in business models: a red hat and ibm case study[END_REF]. 0 is considered a non resilient value. 2. Flexible Licensing 2.1. Level or permissiveness: 0 (all restrictive -i.e. commercial), 1 (persistent i.e. GPL), 2 (all permissive -i.e. BSD). We base the indicator to the of [START_REF] Välimäki | Evaluation of open source licensing models for a company developing mass market software[END_REF]. The indicator is considered non resilient when it is less than 1. 2.2. Level of persistence: 0 (no) or 1 (yes) based on whether there are parts of the project's code or dependencies published under persistent licenses (i.e. GPL). We base the indicator to the of [START_REF] Välimäki | Evaluation of open source licensing models for a company developing mass market software[END_REF]. 1 is considered a non resilient value. 3. External Organization Support: 0 (no) or 1 (yes) based on whether the OSS project is supported by an external organization (non profit, governmental or corporate). 0 is considered a non resilient value.

Resilience determination mechanism

Since the evaluation of a project regarding its resilience is based on indicators we need a mechanism to determine whether the OSS project under review is resilient and, on a second level, how its resiliency changes as it evolves. Starting to the indicators level we will consider an OSS project successful towards a resilience goal when it is considered resilient at least to 50% of the goals ingredients.

Moving to the dimensions level, an OSS project will be considered successful towards a resilience dimension when it is considered resilient at least to 50% of the goals of the specific dimension. Likewise, on a project level, the OSS project is considered resilient when at least two (2) out of four (4) dimension (50%) are considered resilient.

Open Source Software Resilience Framework Application

In this section we will be demonstrating the application of the OSSRF to two OSS projects, the OKapi and WooCommerce, using the indicators analyzed in the previous sections. Due to space limitations, for our demonstration purposes will be using only the fully analyzed dimension "Business & Legal". Since the OSSRF takes under consideration the evolution of OSS projects we would normally have to apply the framework to all the major releases of the two projects. Due to limitation spaces we chose to use the last major release of each year from the beginning of each project until either the year the project became inactive (i.e. for the case of OKapi) or the last year that ended (i.e. 2017) for the case of WooCommerce. We do not consider major releases, alpha, beta release or release candidates so if some early years of the project have been excluded is because there were no major release back then.

OKapi -A non resilient project

The first project, OKapi is a small framework for building web applications. It's built on PHP and is hosted in Github [START_REF]OKapi Github Repository[END_REF]. It started during 2008 and hasn't been updated since July 2011. We selected this case in order to test our framework to a project that intuitively seems non resilient.

The versions of the releases of OKapi to which we will be applying the OS-SRF, with their corresponding dating (year / month) are shown to the following table: By applying the Business & Legal dimension's indicators to OKapi we are getting the results as shown in Table 1. In order to make the demonstration of OKapi is officially not maintained but even when it was there is no indication that supported donations, commercial features or support. There is no official website or reference in any related document, to our best knowledge, about revenue streams for the project. Therefore we consider the project non resilient regarding the indicators of the first goal. As far as the licensing part is concerned, there is no license defined to the project and OKapi does not seem to reuse other open source projects. Therefore the project is considered non resilient as far as the "Level of permissiveness" is concerned but it is considered resilient towards the "Level of persistence". Therefore regarding the goal "Flexible Licensing" OKapi is considered resilient. Finally, there is no indication that OKapi was ever supported by an external organization therefore we consider the project non resilient towards this goal. The aforementioned results apply to all the versions of the project.

Based on the results two out of three goals are considered non resilient for the OKapi project. Therefore based on OSSRF, the project is considered non resilient.

WooCommerce -A resilient project

The second project is WooCommerce, an open source eCommerce plug-in for WordPress content management system, written also in PHP and hosted in Github [START_REF]WooCommerce Github Repository[END_REF]. It started as an OSS project in 2011 and as of the time of writing is active. We selected this case in order to test our framework to a project that intuitively seems resilient.

The versions of the releases of WooCommerce to which we will be applying the OSSRF, with their corresponding dating (year / month) are shown to Table 3. By applying the Business & Legal dimension's indicators to WooCommerce we are getting the results as shown in the Table 4.

WooCommerce is a free and OSS WordPress plug-in but there are commercial plug-ins created for WooCommerce. There is also paid support offered. To our best knowledge it does not support donations. Additionally, WooCommerce is published under GPL license and some of its dependencies are published under persistent license (i.e. wc-e2e-page-objects). Finally WooCommerce is backed from both the WordPress Foundation and the Automattic Company, both wellknown organizations. Towards "Economic Sustainability" goal the project is considered resilient since 2 out of 3 indicators are considered resilient. Regarding "Flexible Licensing" it is also considered resilient since 1 of 2 indicators is considered resilient. Finally, "External Organization Support" is also considered resilient. The aforementioned findings apply to all versions.

Conclusions & Future work

This is a preliminary work proposing an evaluation approach with the aim of providing a strong theoretical basis for evaluating OSS projects adapting the concept of Urban Resilience to OSS. By doing so, we are trying to benefit form the advantage of resiliency to follow the evolution of a dynamic system (such as a city or an OSS) while at the same time we are providing a framework that can take under consideration concepts such as crises and or stresses that can theoretically impact the survival of an OSS. We applied a single dimension of the OSSRF to an intuitively non resilient OSS project and an intuitively resilient one and the results seem to concur with the initial intuitions.

For future work we intend to thoroughly fine-tune the rest of the indicators by testing it to a variety of OSS projects. This will also allow us to investigate how the OSSRF responds to projects of different age, community size or source code size and complexity. We also intend to investigate whether the software domain of an OSS project affects the results of the application of the OSSRF.

Regarding the framework itself we will experiment with other approached regarding the "Resilience determination mechanism" (i.e. weighted goals).

In addition we will be extending the OSSRF to be able to work with a variety of control version systems (not only git-like but also Mercurial, SVN, CVS). In a similar spirit will would like to experiment with projects of different programming languages (i.e. Java).

Another challenging idea for future work would be to apply OSSRF to OSS projects that are known to have faced specific stresses or crises in order to identify how those crises relate with the resiliency levels of an OSS project.

Finally we intend to attempt and request feedback, in the form of a survey, from key players of the Open Source Software international community (lead developers, stakeholders, academics and so forth) about OSSRF.

1 .

 1 Health & well-being: Related to people, working and living in the city.

	Goals: (1) Minimal human vulnerability, (2) Diverse livelihoods & employ-
	ment (3) Effective safeguards to human health & life.
	2. Economy & society: Related to the organization of cities on a social
	and economical level. Goals: (1) Sustainable economy, (2) Comprehensive
	security & rule of law, (3) Collective identity & community support.
	3. Infrastructure & environment: Related to place, the quality of infras-
	tructure and ecosystems. Goals: (1) Reliable mobility & communications,
	(2) Effective provision of critical services, Reduced exposure & fragility.
	4. Leadership & strategy: Related to knowledge of the past and adapting
	appropriately for the future. Goals: (1) Effective leadership & management,
	(2) Empowered stakeholders, (3) Integrated development planning
	Finally, the aforementioned goals are, on a third level, analyzed in indicators
	in order to identify the critical factors that contribute towards the resilience of
	urban systems.

  goals:

	3.1.3 Indicators
	Finally, the twelve (12) goals are further analyzed to indicators in order to pro-
	vide a more specific description of the goals. For the purposes of this paper we
	will analyze the indicators related to the goals of the Business & Legal dimen-
	sion. Due to space limitations, we provide the full analysis of the indicators to
	the following url: http://users.auth.gr/akritiko/ossrf_indicators.html
	for the intended audience.
	1. Economic Sustainability
	1.1. Donations: 0 (no) or 1 (yes) based on whether the OSS project accepts
	donations. 0 is considered a non resilient value.
	1.2. Commercial features: 0 (no) or 1 (yes) based on whether the OSS project
	offers commercial features or a pro (paid) version. The indicator was
	based to the work of
	Source Code
	1. Continuous Growth
	2. Holistic Documentation
	3. Systematic Testing & Violation Minimization
	Business & Legal
	1. Economic Sustainability
	2. Flexible Licensing
	3. External Organization Support
	Integration & Reuse
	1. Low Dependability
	2. Low Complexity
	3. Ease of Integration
	Social (Community)
	1. Well defined Project Standards
	2. Well Defined Governance
	3. Developer Base Activity

Table 1 .

 1 OKapi releases

	Version	1.1.5	1.2.1	1.2.3
	Date	2008/12	2009/12	2010/12

Table 2 .

 2 OKapi -Business & Legal dimension. Goals & Indicators

	Indicator	v1.1.5	v1.2.1	v1.2.3
			Economic Sustainability
	Donations	0 (F)	0 (F)	0 (F)
	Commercial features	0 (F)	0 (F)	0 (F)
	Paid support	0 (F)	0 (F)	0 (F)
			Flexible Licensing
	Level or permissiveness	0 (F)	0 (F)	0 (F)
	Level of persistence	0 (S)	0 (S)	0 (S)
		External Organization Support
	External Organization Support	0 (F)	0 (F)	0 (F)
	the Resilience Determination Mechanism we proposed at Section 3.1.4. easier,
	we have marked the values of the indicators to Table 2. with (F), if an indicator
	is considered non resilient, and (S) if it is considered resilient.

Table 3 .

 3 WooCommerce releases

	Version	1.3.2	1.6.6	2.0.20 2.0.10 2.0.12 2.6.11 3.2.6
	Date	2011/12 2012/12 2013/12 2014/12 2015/12 2016/12 2017/12

Therefore based on OSSRF, the project is considered resilient.

Threats to validity

We should note that OSSRF should be applied to project of a relative maturity in terms of community and age (we would intuitively suggest at least 10 contributors and a maturity of more than a year). Applying it in solo maintained OSS projects, or projects that not yet have reached the proposed maturity may lead to misleading results.

OSSRF is an adaptation of the City Resilience framework to Open Source Software. Although the structure of the original framework was retained, despite the conceptual similarities that we have already mentioned earlier, the mapping of dimensions, goals and indicators is a product of the subjective lens of the authors.

In the application of the OSSRF to the two projects we have applied the indicators of one of the four available dimensions of the OSSRF, "Business & Legal" dimension.

Regarding the goals and indicators, some of them are based on metrics available for object oriented source code. Additionally, as far as control version systems are concerned, for the needs of this publication we selected projects that are hosted in Github.

As far as the scales and their aggregation is concerned, in this preliminary approach we considered each criterion equally important and the threshold for defining a project as resilient or non resilient is 50%.

Finally, both of the projects analyzed in this paper, were developed in PHP and their domains are close (OKapi was a framework for web applications and WooCommerce is a plugin for a WordPress which is also considered by some a kind of web framework).


Athos Ribeiro
email: athoscr@ime.usp.br

Paulo Meirelles

Nelson Lago
email: lago@ime.usp.br

Fabio Kon
email: fabio.kon@ime.usp.br

Ranking Source Code Static Analysis Warnings for Continuous Monitoring of FLOSS Repositories

Keywords: Static analysis, Software quality, False positives, Free Software, Open Source Software

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Source code static analysis is a valuable technique to support software assurance. In theory, it can explore abstractions of all possible program behaviors, which is not feasible with software testing [START_REF] Hovemeyer | Finding Bugs is Easy[END_REF]. Thus, it can find software bugs using a complementary approach to automated tests.

However, the fundamental problems of static analysis are undecidable [START_REF] Landi | Undecidability of static analysis[END_REF], so approximations must be made, leading static analyzers to generate false alarms or to miss occurrences of software flaws. False positives are produced when a static analyzer processes bug-free code and reports it as buggy code. The tool may also miss actual bugs when it processes buggy code and report it as a bug-free code [START_REF] Black | Static analyzers in software engineering[END_REF] (false negatives). Since static analysis enumerates many execution paths, static analyzer reports frequently contain an excessive amount of information, which often includes a substantial amount of false positives.

The high amount of information generated combined with a significant false alarm rate hinder the inclusion of static analyzers in the software development cycle. Moreover, false positives require manual inspection, which increases the effort of analyzing tool reports [START_REF] Heckman | Adaptively ranking alerts generated from automated static analysis[END_REF][START_REF] Muske | Review efforts reduction by partitioning of static analysis warnings[END_REF] and may even cause static analyzers to be discarded as irrelevant [START_REF] Kremenek | Z-ranking: Using statistical analysis to counter the impact of static analysis approximations[END_REF]. Literature suggests that using multiple static analyzers improves static analysis coverage [START_REF] Black | Static analyzers in software engineering[END_REF] since some tools perform better in specific tasks due to different analysis methods. This practice may decrease the number of false negatives but is likely to generate more false positives as the number of tools used increases.

In this study, we present kiskadee, a system designed to support continuous static analysis in software repositories using multiple static analyzers to generate reports using a common output language. By running multiple static analyzers on the same code base, kiskadee reduces the number of false negatives in the analysis. To address false positives, kiskadee ranks warnings in the static analysis reports using the AdaBoost algorithm's [START_REF] Freund | A short introduction to boosting[END_REF] classification probabilities. Warnings with the highest rank are more likely to indicate real and more critical software flaws and warnings with the lowest rank are more likely to be false positives. In this context, a warning is a single issue produced by a static analyzer. Finally, kiskadee stores the ranked static analysis reports in a database. The ranked reports in the database are made available to kiskadee users, providing them with more accurate data and favoring the use of static analysis.

Related Work

Muske and Serebrenik [START_REF] Muske | Survey of approaches for handling static analysis alarms[END_REF] provided an overview of different techniques on how to handle static analysis alarms. In this survey, the authors classified part of the techniques as the automatic post-processing of the alarms, which includes ranking or classification of alarms. We assessed the studies and techniques under the aforementioned classification to better position the present work.

Previous studies show that the most relevant features for training accurate machine learning models to arbitrate about the positiveness of static analysis alarms are extracted from properties intrinsic to the analyzed project, namely the project change history, function and file names, and even the name of the programmer who introduced the change that triggered the alarm [START_REF] Heckman | A model building process for identifying actionable static analysis alerts[END_REF][START_REF] Jung | Taming false alarms from a domain-unaware c analyzer by a bayesian statistical post analysis[END_REF][START_REF] Kremenek | Correlation exploitation in error ranking[END_REF][START_REF] Ruthruff | Predicting Accurate and Actionable Static Analysis Warnings: An Experimental Approach[END_REF][START_REF] Yoon | Reducing false alarms from an industrial-strength static analyzer by svm[END_REF]. Since these project-specific features are in great part responsible for the high accuracy of the models proposed up to now, a model trained on such features cannot be readily used to query about alarms generated for other projects, hampering the general availability of the model in automated post-analysis tools.

Ruthruff et al. [START_REF] Ruthruff | Predicting Accurate and Actionable Static Analysis Warnings: An Experimental Approach[END_REF] propose a method to predict if a warning is an actionable fault, i.e., if it is not a false positive and if a programmer should fix it. It uses a screening approach for model building that discards metrics with low predictive power. Among the factors used to predict false positives, which happened 85% of the times in the authors study, are the priority given by the static analyzer, the file length, and code indentation, suggesting that the authors performed additional code analysis to extract factors from the code.

As discussed, related works emphasize that the most important characteristics to arbitrate on static analysis warnings positiveness are internal to the analyzed project. Our study differs from them by assessing static analysis warnings only with the information present in the warnings themselves, therefore, the approach in itself usually produces poor results. To compensate for this, we use multiple static analyzers with kiskadee to generate more information and correlate the information provided by them to assess the correctness of a given warning better. Since this strategy still might result in a low-quality classifier, we turn to ensemble techniques to generate and combine multiple weak classifiers generated this way into a stronger one [START_REF] Russell | Artificial Intelligence: A Modern Approach[END_REF].

Continuous Static Analysis with kiskadee

In Free/Libre and Open Source Software (FLOSS) projects, a common source of bug reports are the GNU/Linux distributions. These distributions ship thousands of software projects, which they call packages. Distribution developers refer to the projects that maintain the software they ship in the distribution as upstream. It is not unusual for distribution developers to report bugs in upstream projects or to send patches to fix bugs found by their distribution users or during the packaging process.

By continuously running multiple static analyzers in several of these packages, i.e., once for each version of the package, we can create a database of static analysis reports on software projects of different sizes and application domains. Developers can then use the information in this database to find and act on software flaws. We chose to use GNU/Linux distributions due to the high amount of software packages available and the well-defined and documented interfaces they provide to download the latest versions of these packages. It is also an advantage that the cultural norm for GNU/Linux distribution developers is to report (and often propose fixes for) bugs. Therefore, using their repositories for this work may provide a broader user base for the tools and techniques developed in this research.

To continuously run static analysis on software packages and handle the false alarms generated by these analyses, we developed kiskadee. Figure 1 represents kiskadee's architecture overview, where the numbers denote its execution flow. In steps ( 1) and ( 2), kiskadee monitors software repositories for new releases. In step (3), kiskadee downloads the source code of each new software version in a repository it monitors and runs a set of predefined static analyzers on it in step (4). In step [START_REF] Black | Static analyzers in software engineering[END_REF], kiskadee translates each static analyzer report to a common warnings report format. This common format is needed because each static analyzer defines its unique format to report warnings. In step [START_REF] Boland | Juliet 1.1 c/c++ and java test suite[END_REF], kiskadee ranks the warnings based on their probability of being real bugs, where warnings on the top have a higher probability of being real bugs and warnings on the bottom are more likely to be false positives. This ranking step is performed with a classification model, described in Section 4. The ranked warnings are then saved in a database in step [START_REF] Drucker | Boosting decision trees[END_REF], using kiskadee's common warning report format. Finally, in step ( 8), kiskadee provides an API consumed by a visualization tool to display the ranked warnings filtered by package versions. The information provided can be used either by distribution developers to evaluate and report possible bugs upstream or by upstream developers themselves. Researchers can also use kiskadee's database in different contexts.

FLOSS development communities have been discussing a common static analysis report output format. The Fedora Project Static Analysis Special Interest Group [START_REF]Fedora project static analysis special interest group[END_REF] designed a tool to run static analyzers during the package build process [START_REF]Fedora mock-with-analysis project[END_REF]. Although the tool itself is in its early development stages and not ready for usage in production, the developers discussed [START_REF]Firehose mailing list[END_REF] a common report format for the static analyzers in their mailing lists. After a few iterations, with Debian Project developers collaboration, they created Firehose, a complete definition of a common warnings report format for static analyzers and a set of tools to generate, parse, and verify this format. We use Firehose as kiskadee's common warning report format, eliminating the need to design a new format and to develop the tools to handle it, like parsers and generators.

kiskadee was run with three static analysis tools to generate the warnings data set for our experiments. The Criteria to select the tools were: (1) The tool must be able to examine C/C++ code for security flaws (e.g., buffer overflows, null pointer dereferences); and (2) the tool source code must be released under an FLOSS license.

Criterion 1 ensures the tool can analyze a subset of the test cases in our data set, introduced in Section 4, whereas criterion 2 preserves us from disputes by tool vendors regarding the analysis of the results (such as allegations of sub-optimal tool calibration or detrimental calculation methodology). FLOSS tools also simplify the process of retrieving string constructs for static analyzer warning messages and categories when necessary, since we can verify their source code. Following the criteria, the static analyzers selected were Clang Static Analyzer (version 3.9.1), Cppcheck (version 1.79), and Frama-C (version 1.14 with the value analysis plugin activated).

Fedora Project uses an external system to monitor the software projects they distribute. Anitya [START_REF]Anitya project website[END_REF] maps upstream projects to distribution package names. Whenever a new version of an upstream project is released, Anitya publishes the new release in a Fedora infrastructure publish-subscribe system where other systems in the distribution infrastructure can handle it. New software versions are published by Anitya as soon as they are released upstream. Analyzing these new software versions with kiskadee as early as they are released allows the distribution developers to address potential bugs found by kiskadee before the software is shipped to final users. Therefore, kiskadee monitors packages by reading information published by Anitya in the Fedora infrastructure publish-subscribe system.

kiskadee can point to other software repositories as well through its plugin architecture (kiskadee's fetchers). Each fetcher must implement functions that define which repository to monitor, how to monitor it, and which static analyzers to run for that repository. Hence, we can extend kiskadee to run different static analyzers for different software repositories or GNU/Linux distributions.

Ranking Warnings

To create and train a predictive model, before analyzing project repositories, we ran kiskadee on Juliet [START_REF] Boland | Juliet 1.1 c/c++ and java test suite[END_REF], a publicly available test suite composed of a collection of source code snippets with specific flaws injected in known locations, which facilitates the assessment of static analysis tools. By running kiskadee on Juliet (without the ranking step), we obtained the analysis reports (a set of warnings) in the Firehose format, easing further processing. Then, we checked whether each single warning matched one of the injected flaws in the test suite, labeling them either as true positive or false positive. After the labeling step, we extracted the characteristics used to train our model. These did not include any characteristics from the analyzed source code and project history, which tend to be the most relevant ones when predicting warnings positiveness, as shown in previous works [START_REF] Jung | Taming false alarms from a domain-unaware c analyzer by a bayesian statistical post analysis[END_REF][START_REF] Yoon | Reducing false alarms from an industrial-strength static analyzer by svm[END_REF]. By not using the characteristics aforementioned, we can produce a general model that can be used with any project without prior knowledge about it, eliminating the need to perform the expensive model training step for each project one may want to analyze. This means that the model obtained during the preparation of this paper can be used directly by kiskadee in production to rank warnings in any given project.

To compensate for not using the most relevant characteristics pointed by previous works, we used an ensemble learning method [START_REF] Russell | Artificial Intelligence: A Modern Approach[END_REF] to train several weak classifiers, which were then able to vote about the positiveness of new examples.

Finally, instead of just classifying new examples as true and false positives, we ranked the warnings based on their probabilities of being real flaws, where the top entries were more likely to be of interest, and the bottom entries were more likely to be false positives.

The results obtained were auspicious: using three static analyzers with a false positive rate of 0.61 when aggregating all three tool warnings in a single report, we achieved an accuracy of 0.8 over the test set. This accuracy was not too distant from the state of the art (0.85 [START_REF] Ruthruff | Predicting Accurate and Actionable Static Analysis Warnings: An Experimental Approach[END_REF]), which depends on characteristics specific to the project being analyzed to train their model. Since our model does not depend on project-specific features, it may receive any project as input without the need to train a different model for each analyzed project.

The following steps describe the techniques and tools used to collect and prepare the data to train our model, as well as the methodology used to train the model and rank the static analysis warnings.

Step 1: choosing data sources. A data set of labeled static analysis warnings may be obtained by running static analyzers on previously selected source code and matching the triggered warnings with actual software defects, labeling the warnings as true or false positives. The source code used for extracting the data set may consist of real-world software or synthetic test cases, i.e., programs written with intentional defects.

Juliet [START_REF] Boland | Juliet 1.1 c/c++ and java test suite[END_REF] is a synthetic C/C++ test suite created by the United States National Security Agency (NSA) and distributed by the National Institute of Standards and Technology (NIST) under the public domain. It is composed of 61,387 test cases covering 118 different software flaw categories. Each test case is a code section with an instance of a specific flaw (to capture true positives) and an additional section with a correct, fixed instance of that previous flawed section (to capture false positives). Juliet also includes a user guide with instructions on how to assess and label static analysis tool warnings generated over its test cases. We use Juliet version 1.2 to generate our data set.

Before examining Juliet with kiskadee static analysis tools, we pruned the test suite to prevent analysis of test cases that depend on constructs of specific operating systems or external libraries. Table 1 shows the total number of test cases in Juliet before pruning and the number of test cases after the pruning step for both C and C++. The latter are the tests examined by the static analyzers for alarms generation, consisting of 39,100 C/C++ test cases.

Step 2: collecting labeled warnings from multiple static analyzers. Based on Juliet documentation, we process each file in the remaining test cases to produce a list (L) with information on whether a static analysis warning for a given location should be labeled as true positive or false positive. Then, we run each static analyzer on the pruned test suite to generate the static analyses reports. Table 2 shows the total number of warnings generated, before discarding warnings whose labeling step cannot be automated based on Juliet documentation. By matching each warning produced by the static analyzers with the corresponding flaw categories covered by Juliet, we can use the list L to produce labels for each warning. This set of labeled warnings can finally be examined to extract a training set from it, as discussed next. Table 3 summarizes the findings of the static analyzers on the test cases, including the number of true and false positives generated (TP and FP, respectively), the false positive rates, and the precision for each tool and for the whole kiskadee report, i.e., the aggregated reports composed of the warnings of all tools.

Step 3: extracting features from labeled warnings. We obtain the features used to train our classifier from the set of labeled warnings. Here, a feature is any characteristic that can be attributed to a warning in the data set. To select relevant features for false and positive alarm classification, we refer to previous studies that also rely on characteristics extracted from alarms and source code to classify alarms [START_REF] Heckman | A model building process for identifying actionable static analysis alerts[END_REF][START_REF] Jung | Taming false alarms from a domain-unaware c analyzer by a bayesian statistical post analysis[END_REF][START_REF] Kremenek | Correlation exploitation in error ranking[END_REF][START_REF] Ruthruff | Predicting Accurate and Actionable Static Analysis Warnings: An Experimental Approach[END_REF]. For instance, Kremenek et al. [START_REF] Kremenek | Correlation exploitation in error ranking[END_REF] demonstrate that the tool warning positiveness is highly correlated to code locality.

We extract our set of features by processing the aggregated report of labeled warnings. While we can infer the name of the tool that triggered a warning, the programming language analyzed, and the severity of the warning by looking at a single warning at a time, other features require processing the whole aggregated report to be extracted. Namely, these are (1) the number of times the same location was pointed as flawed in the report, (2) the number of warnings triggered around the location of a given warning (e.g., warnings for locations at most 3 lines away from the current warning), (3) the category of the software flaw suggested by the warning, (4) which other static analyzers generated warnings for the same location, and (5) the number of warnings generated for the same file the current warning is pointing to.

Step 4: training decision trees with AdaBoost. Given the data collected, we build a prediction model to classify each triggered warning as being a true positive or a false positive. The classification results may also be used to rank the warnings, as we describe in Step 5.

Since we do not post-analyze the source code nor inspect the project history of the analyzed software projects, which are shown to be the best places to look for features to arbitrate on source code static analysis warnings positiveness, we turn to ensemble learning methods to train several weak classifiers with our feature set. These weak classifiers combined can then arbitrate on new examples together, composing a stronger classifier with lower error [START_REF] Russell | Artificial Intelligence: A Modern Approach[END_REF].

One widely used ensemble learning method is boosting [START_REF] Russell | Artificial Intelligence: A Modern Approach[END_REF], whose main idea is to run a weak learning algorithm several times in different distributions of the training set to generate and combine various weak classifiers into a stronger one. For this study, we use the AdaBoost algorithm [START_REF] Freund | A short introduction to boosting[END_REF], a more general version of the original boosting algorithm [START_REF] Polikar | Ensemble based systems in decision making[END_REF].

The AdaBoost algorithm works with any given base learner. We use a decision tree learning algorithm as our base learner because we have both categorical and non-categorical features in our data set, and decision trees can work with both, without the need to pre-process the data set. Furthermore, as shown in the literature, decision trees perform well with AdaBoost [START_REF] Drucker | Boosting decision trees[END_REF].

We divide our data set into a training set and a test set. The training set is built by randomly selecting 75% of the examples labeled as true positives and 75% of the examples labeled as false positives from the features data set. We then proceed to train our predictive model using 10-fold cross-validation with the training set. We perform the 10-fold cross-validation technique with different values for T (number of weak classifiers trained) in AdaBoost. We then compare the average performance of the classifiers obtained for each distinct value of T validated in this manner and use the best model trained during the cross-validation for that T to classify the test set.

Step 5: ranking static analysis warnings. We use the model trained in Step 4 to rank the warnings in a static analysis report based on the model classification probabilities. We reorder the warnings in a list according to the probability of the warning being a true positive, where warnings with higher probabilities are ranked in the top of the list and warnings with lower probabilities of being true positives are arranged in the bottom of the list. This way, a programmer inspecting the ranked static analysis report may examine only the top warnings in the list up to a given threshold, assuming a certain risk of missing true positives. Alternatively, he/she may stop inspecting warnings when false positives start to abound.

Results and Discussion

Although we use a binary classification algorithm to train our model, we do not need to limit ourselves to a direct binary classification; it is also possible to use the trained predictive model to rank warnings according to their expected relevance. Next, we present and discuss the results obtained with kiskadee's ranking approach. While comparing our results with other ranking approaches or, at least, with the ranking order of each tool would be ideal, these would not be feasible. In the first case, we would have to replicate other works with our data set; in the second, it would not make sense to compare the results of a single tool to the aggregate results. Therefore, we chose to compare kiskadee with a random ranking algorithm.

To evaluate our ranking performance over the test set, we refer to the methodology presented by Kremenek et al. [START_REF] Kremenek | Correlation exploitation in error ranking[END_REF], which we describe below.

We define S(R) to be the sum of F P j , the cumulative number of false positive warnings found before reaching the j th true positive warning when navigating a ranked list (starting from the first entry) ordered by a ranking algorithm R.

S (R) = N tp j=1 FP j (1)
It is worth observing that S (R) = 0 for an optimal ranking algorithm and S (R) = N tp × N f p for the worst ranking algorithm, where N tp and N f p are the total number of true positive warnings and false positive warnings in the list, respectively.

We then define the average of the cumulative number of false positive warnings found before reaching each true positive warning, FP avg (Eq. 2).

FP avg = S (R) Ntp (2)
Finally, we measure the performance ratio of our ranking algorithm against a random ranking algorithm, which shuffles the list of warnings, with Eq. ( 3).

In a perfect ranking situation, the first false positive occurrence would be positioned after the last true positive occurrence, therefore, FP avg = 0. For our test set, in the worst case scenario, one would hit all the false positives before finding the first true positive. Leading to FP avg = 3942. When applying a random ranking algorithm, we found FP avg = 1992, while, for kiskadee's ranking approach, FP avg = 380 over our test set.

The median kiskadee model performance over random, as proposed by Kremenek et al. [START_REF] Kremenek | Correlation exploitation in error ranking[END_REF], was 5.2, which indicates that, on average, one hits 5.2 times more false positives before each true positive with a random ranked warning list than one would if using the proposed ranking. Figure 2 shows the number of real flaws found in a ranked list per inspected entries (warnings) for different ranking models applied to the test set: optimal, where all the real flaws are in the top of the list; worst, where all the false positives are in the top of the list; random, where the entries are randomly shuffled in a list; and model, which represents the ranking model proposed for kiskadee.

As Figure 2 shows, kiskadee's model outperforms the random ranking algorithm by presenting all software flaws in the test set after 3990 inspections, while the random ranking algorithm presents software flaws in a linear relation with the number of inspections, where the last few real software flaws in the test set are only presented in the end of the list, after 6486 inspections.

Conclusion

Different from related works, kiskadee's ranking approach does not use features based on the analyzed project intrinsic properties for model training, namely, source code change history and code metrics. Consequently, by smoothly decreasing the classification accuracy, the model obtained can be used successfully with any given software project. This is a compelling trade-off to enable kiskadee to analyze and rank any project given as input, allowing the continuous monitoring and analysis of different software repositories, such as the ones provided by GNU/Linux distributions. kiskadee can be used to reduce the cost of inspecting false alarms by setting a minimum value for the rate in which real flaws are found per inspection in a ranked list (i.e., a confidence level). When the rate of real flaws per inspections drops below that level, one could stop the inspection for that warning list. Interesting future works include studying confidence levels and the trade-off between loss of information and the cost of inspecting a larger number of false alarms, improving the classification model by collecting users feedback, and investigating other algorithms besides AdaBoost with decision trees as classifiers.

kiskadee is licensed under the GNU Affero General Public License. Its development repository, the complete project documentation (including UI screenshots), and the data set used for the ranking experiments here presented are available at pagure.io/kiskadee.

Fig. 1 :

 1 Fig. 1: kiskadee design overview.

  Performance =FP avg (random) FP avg (AdaBoost ranking)

Fig. 2 :

 2 Fig. 2: Number of actual flaws found in top-down inspection of lists ranked with different approaches.

Table 1 :

 1 Number of Juliet test cases.

	C C++ Total	Before Pruning After Pruning 36,078 22,459 25,309 16,641 61,387 39,100	Tool Clang Static Analyzer Cppcheck Frama-C Total	Warnings 37,229 124,025 120,573 281,827

Table 2 :

 2 Warnings generated per tool.

	Tool	Warnings	TP	FP	FP Rate	Precision
	Clang Analyzer	6207	984	5223	0.84	0.16
	Cppcheck	4035	314	3721	0.92	0.08
	Frama-C	15717	8892	6825	0.43	0.57
	Aggregated tools	25959	10190	15769	0.61	0.39

Table 3 :

 3 Labeled warnings per tool.