Annemarie Harzl
email: aharzl@ist.tugraz.at

Combining FOSS and Kanban: An Action Research

Keywords: Free Open Source Software, Agile Software Development, Lean, Kanban, Action Research

Even though Free and Open Source Software (FOSS) and Agile Software Development (ASD) have been recognized as important ways to develop software, share some similarities, and have many success stories, there is a lack of research regarding the comprehensive integration of both practices. This study attempts to consolidate these methods and to answer if FOSS and ASD can be combined successfully. Action Reseach (AR) is conducted with one sub-team of a large FOSS project. We performed two action research cycles based on the Kanban method. This paper has two main contributions; first, it describes a real world situation, where Kanban is applied to a FOSS project, and second, it suggests two new Kanban practices. These two methods are targeted specifically at FOSS projects and their characteristics.

Introduction

This paper examines the use of Agile Software Development (ASD), namely the Kanban method [START_REF] Anderson | Kanban -Successful Evolutionary Change for Your Technology Business[END_REF][START_REF] Baumeister | Kanban in der IT -Eine Kultur der kontinuierlichen Verbesserung schaffen[END_REF], in the context of Free and Open Source Software (FOSS) development.

Since the publication of the agile manifesto in 2001 a large body of research has been published on agile methods [START_REF] Dingsøyr | A decade of agile methodologies: Towards explaining agile software development[END_REF], e.g. on adopting agile methods [START_REF] Boehm | Get ready for agile methods, with care[END_REF] and implementing agile methods in distributed settings [START_REF] Boland | Transitioning from a co-located to a globally-distributed software development team: A case study at analog devices, inc[END_REF][START_REF] Ramesh | Can distributed software development be agile?[END_REF]. FOSS and ASD have become common software development processes over the last fifteen years [START_REF] Dingsøyr | A decade of agile methodologies: Towards explaining agile software development[END_REF][START_REF] Crowston | Free/libre open-source software development: What we know and what we do not know[END_REF] and some studies have been done about combining FOSS and ASD [START_REF] Gandomani | A systematic literature review on relationship between agile methods and open source software development methodology[END_REF].

Although Warsta and Abrahamsson [26] and Koch [START_REF] Koch | Agile principles and open source software development: A theoretical and empirical discussion[END_REF] already showed in 2003 and 2004 that FOSS development and the definition of ASD methods are rather close, research about agile development in the context of open source software was still identified as a future research area in 2009 [START_REF] Ågerfalk | Introduction to the special issueflexible and distributed information systems development: State of the art and research challenges[END_REF]. In 2013 Gandomani et al. [START_REF] Gandomani | A systematic literature review on relationship between agile methods and open source software development methodology[END_REF] conducted a systematic literature review on relationships between ASD and FOSS development. They concluded that the examined studies indicated that ASD can support FOSS development, mainly because they share several concepts and principles. However, the authors did not find a case study successfully integrating both methodologies. This research tries to comprehensively combine these two for a specific real world case. By doing this we attempt to answer the following research question: "Can FOSS and ASD be comprehensively combined?". This paper describes two action research cycles and proposes two FOSS-related additions to the Kanban practices from the perspective of a real project, therefore making it scientifically and practically relevant.

Related Work

Even though the research fields of FOSS and ASD have been of interest for over a decade with a large number of publications, there is to the best of our knowledge no work on comprehensively integrating the two methods. This is supported by the findings of Gandomani et. al. [START_REF] Gandomani | A systematic literature review on relationship between agile methods and open source software development methodology[END_REF]. Most studies show only collaboration between the two and most of the time only specific practices are applied to a FOSS project [START_REF] Deshpande | Continuous integration in open source software development[END_REF][START_REF] Düring | Sprint driven development: Agile methodologies in a distributed open source project (pypy)[END_REF]. Other studies use Kanban or FOSS for teaching computer science classes [START_REF] Ahmad | Student perceptions and attitudes towards the software factory as a learning environment[END_REF][START_REF] Mackellar | Bridging the academia-industry gap in software engineering: A client-oriented open source software projects course[END_REF], but never together. Koch [START_REF] Koch | Agile principles and open source software development: A theoretical and empirical discussion[END_REF] compares the methods based on some criteria, but the paper does not include a practical implementation.

Background

This section describes some details about the FOSS project under study, my role as a researcher within the project, and reasons for integrating Kanban into a FOSS project.

The FOSS Project

The project under study is a hybrid student FOSS project. Within the scope of this umbrella project, various teams develop mobile applications for different platforms and purposes. The software development method follows an agile approach with elements of eXtreme Programming (XP) and Kanban. Most of the contributors are participating in the project in the context of their studies (e.g. Bachelor thesis or Master project) and stay with the project between six months and approximately two years (with breaks between the Bachelor thesis and Master project). Unlike usual FOSS projects, where a small number of people develops the majority of the code [START_REF] Kagdi | Who can help me with this source code change?[END_REF], contributions to this project are more evenly spread. Developers in this project are constantly changing and there are no core developers, who stay with the project for multiple years. Students are often working off-campus and our international contributors are of course working all over the world. Individuals are not assigned to a team, everyone self-selects the team and topic he or she is willing to work on. There are only three distinct roles: contributors, seniors, who are allowed to accept code of others into the main repository and the team coordinator, who is the main contact person for other teams and has a good overview of the team's status. The team coordinator volunteers for this position and the team decides, who is suitable for this position. There exists no central management, only the project head, who is mainly responsible for the overall orientation of the umbrella project and the sub-projects, and one person, who is responsible for organizational activities, e.g. managing accounts and infrastructure. Software development skills of the contributors range from beginner to intermediate and knowledge about agile methods ranges from very little to moderate.

Interesting to note is that this project was not started by "scratching a developer's personal itch" [START_REF] Raymond | The Cathedral and the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary[END_REF], but by a university professor with an idea. The developers of the software are not the target group of the developed software. To foster the understanding of user needs a Usability and User Experience (UX) team works alongside the development teams.

My Role as a Researcher within the Project

The study is designed as insider in collaboration with other insiders, but power relations may play a part. In the FOSS project under study I am responsible for organizational and supporting processes, for example creating user accounts and giving a short introduction into the overall project. I am neither programming with other contributors nor am I leading one of the programming teams. However, I am on good terms with the project head and founder, whose word has a lot of bearing in the whole community. He is also the professor grading the students, who do their Bachelor thesis or Master project in the FOSS project. Thus, although I am not in an official hierarchical position, contributors probably see me as someone with informal power within the organization.

Justification of Kanban

The Kanban method [START_REF] Anderson | Kanban -Successful Evolutionary Change for Your Technology Business[END_REF][START_REF] Baumeister | Kanban in der IT -Eine Kultur der kontinuierlichen Verbesserung schaffen[END_REF] is about evolutionary change and strives to establish a culture of continuous improvement (kaizen). It consists of four principles: start where you are; pursue incremental, evolutionary change; respect the current processes, roles, responsibilities and titles; promote leadership at all levels and six practices: visualize the workflow; make policies explicit; manage flow; limit Work In Progress (WIP); implement feedback loops; improve collaboratively, evolve experimentally (using models and the scientific method). It was mainly chosen for the following three reasons.

First of all, in 2014 Ahmad et al. [START_REF] Ahmad | Student perceptions and attitudes towards the software factory as a learning environment[END_REF] concluded that Kanban appears to be a good pedagogical tool and useful for teaching inexperienced software developers about software engineering. It appears to have a short learning curve and a low adoption treshold. It further helps students to improve their team work skills, for example communication and collaboration. The project under study is not only a FOSS project, with many inexperienced developers, it serves teaching purposes as well, making Kanban an appropriate and light-weight approach for teaching and FOSS development. Kanban fosters collaboration and keeps the entrance barrier of the project as low as possible, so potential contributors are not scared off.

Another reason was that Kanban is the most adaptive method [START_REF] Kniberg | Kanban and Scrum -making the most of both[END_REF]. It allows for small evolutionary changes, does not require week-long expensive trainings and job titles and responsibilities do not have to be altered. Moreover, small incremental changes do not require positional power, which is not available in a FOSS community anyway. People affected by these changes are not enduring them, but are involved in the process and their participation is an integral part. People are allowed and encouraged to use their own mind. Thus, Kanban is also a good fit for Action Reseach (AR) as research methodology, which is discussed in more detail in Section 4.

Last but not least, projects within the umbrella project already used a Kanban-like board. By choosing the Kanban method, the team would be able to keep existing tools and the initial alterations would not be overwhelming.

Research Methodology

Justification of Action Research

To accomplish not only scientific but also practical outcomes was an important part of the motivation to conduct this study. According to Dick [START_REF] Dick | A beginner's guide to action research[END_REF] AR is well suited for this type of goal, thus, we chose AR as the research method. Researchers and practitioners collaborate to solve real world problems through theoretically informed actions [START_REF] Greenwood | Introduction to Action Research: Social Research for Social Change[END_REF]. To achieve outcomes, people affected by those actions have to commit to them. One way to ensure this commitment is through involvement [START_REF] Dick | A beginner's guide to action research[END_REF], for which AR offers various participatory methods [START_REF] Herr | The Action Research Dissertation -A Guide for Students and Faculty 2nd Edition[END_REF]. Another reason for choosing this methodology was that only little research has been done on the integration of ASD and FOSS development [START_REF] Gandomani | A systematic literature review on relationship between agile methods and open source software development methodology[END_REF], suggesting that the theory about this matter is not fully developed. Therefore, according to Edmondson, McManus and Kampenes et.al. [START_REF] Edmondson | Methodological fit in management field research[END_REF][START_REF] Kampenes | Flexibility in research designs in empirical software engineering[END_REF], a flexible approach, like AR, would be appropriate.

Case Selection

The project in question was selected due to the following reasons:

-Personal contact: Direct personal contact to the people involved allows for more detailed observations on group interaction than analysis of e.g. mailing lists. Trust, which is needed to change a work process, is easier established through personal than written contact. Furthermore, it is easier to receive feedback on multiple levels and to refine the research methodology and researcher skills through personal contact. -Experiments and evaluation: Although the setting with mainly student contributors is rather unusual, students work on many FOSS projects and are not atypical FOSS contributors. Moreover students are often used to research, thus more used to experimenting with different approaches and willing to evaluate them. Other contributors may be more reluctant to do so.

-Time and access: A basic trusting relationship to project members was already established, so the bonding period with the community, which could take a very long time, could be minimized and allowed to conduct the AR within a reasonable time frame. Topics can be discussed in a shorter period of time and one has access to various artefacts, e.g. whiteboards or flipcharts.

While this project should of course not remain the only case studied, in our opinion it is a good starting point to explore Kanban in the context of FOSS projects. We will elaborate on the limitations in Section 6.2.

The sub-project was co-selected by the participants of the AR. My supporting role (as described in Section 3.2) within the project may have led one team coordinator to ask me for help. The team experienced problems with motivation and their workflow and the team members did not know how to overcome these issues on their own. Therefore, one goal of this research is to achieve practical outcomes, which improve the working situation of the team. This bias for action contributed to the selection of the research methodology.

Asking for help shows some commitment, which is usually needed to achieve action outcomes. Knowing this, we decided to conduct the study with this team. Other factors for the decision were: the team (six to eight people, varying over time) has roughly the average size of teams in this FOSS project (six to twelve team members at the same time), it uses the same agile workflow as the other teams and direct personal contact with the members of the team is possible.

The sub-project develops a mobile application targeted at teenagers, which should enable its users to create small projects without prior domain knowledge. The application has not been released to the public, it is only tested by members of the FOSS project. From the beginning the team used elements of XP and a Kanban board, like all other sub-projects. The applied XP practices included automated unit tests, pair programming, refactoring, release planning (occurs in irregular intervals), short releases, continuous integration, coding standards, collective code ownership simple design and regular meetings (weekly). Visualize the workflow was the only Kanban practice applied, in the form of an agile board, but members did not know, that this was a Kanban practice.

Action Research Cycles

A modified version of Susman and Evered's approach [START_REF] Susman | An Assessment of the Scientific Merits of Action Research[END_REF] was used as a research method. The cyclical model contains the five stages diagnosing, action planning, action taking, evaluating, and specified learning. One diagnosing phase and two regular AR cycles consisting of action planning, action taking, evaluation and specified learning were realized. An additional cycle zero was added after an observation phase at the beginning of the study. A participatory AR approach was used, all steps were discussed with the study participants and decided jointly.

As already described in Subsection 3.1 the sub-project uses elements of XP and Kanban. However, an initial questionnaire showed, that team members assess their knowledge about both methods quite differently. While all members think that they have average to very good knowledge about XP only 17% think that they have very good knowledge about Kanban. The other 83% think that they possess little to no knowledge about Kanban.

This supported my decision to conduct a Kanban coaching session at the beginning of cycle zero and the second cycle. In these sessions I talked about Kanban in general, its principles and practices, and explained terms like flow and kaizen. These sessions were based on two books [START_REF] Anderson | Kanban -Successful Evolutionary Change for Your Technology Business[END_REF][START_REF] Baumeister | Kanban in der IT -Eine Kultur der kontinuierlichen Verbesserung schaffen[END_REF] and one video1 . It was necessary to provide the participants with some theoretical background about Kanban so they could understand its principles and practices. Furthermore, it was also important to enable them to decide how to integrate Kanban practices into their workflow. Due to observations made during the diagnosing phase, I conducted a cycle zero, including a user analysis and a stakeholder analysis. The first AR cycle was designed to introduce two practices into the team, namely visualize (the workflow) and make policies explicit. The second AR cycle should then familiarize the team with the principles limit WIP and manage flow.

Data Sources

Various types of data sources were used as empirical basis: a questionnaire about the participants knowledge on agile practices, weekly notes from team meetings written by the participants, my notes taken during meetings and discussions, the artefacts produced as part of the user and stakeholder analysis, and the team's Kanban board. Six to eight people participated in the study. Two people joined the team after cycle zero and one person left the team after the second cycle was completed. I had no position in the team but as already discussed in Subsection 3.2 my role within the umbrella project may result in research bias. I made some suggestions and most of the time the research participants accepted them.

Data Analysis

Firstly, the questionnaire regarding the agile knowledge of the team was statistically analyzed. Secondly, all researcher notes taken during meetings and discussions, were examined for issues of interest to the research and recurring topics or problems. For this purpose statements were "coded" and grouped together, if they had a theme or problem in common. Thirdly, the results of the team's user analysis and information about the sub-project's target group, retrieved from the project head, were compared. For this purpose statements from the team and the project head were compared one by one and discrepancies identified. Finally, to accomplish a better visualization of the team's workflow, the Kanban board was analyzed for its adherence to Kanban principles and practices, e.g. pull instead of push and limiting work in progress.

Action Research

This section explains cycle zero, the two AR cycles and their phases as described in Subsection 4.3.

Diagnosing

The sub-project of the FOSS project was inspired by a programming exercise done during a university programming course in 2012. The results showed some promising ideas for a new application, which would fit nicely into the portfolio of the umbrella project. However, the code was neither finished (many functions were only rudimentary implemented) nor was it as structured and neat as it should be, because the course was part of a Bachelor study program and lasted only one semester. Thus, only the ideas remained and the code had to be rewritten. Some of the students of the programming course decided to do their Bachelor thesis within the scope of the FOSS project. Together with some other students they started to develop the software anew. After a while it became apparent, that the team had been too ambitious and had ignored agile principles, such as working in small iterations and implementing the simplest thing that could possibly work. The team tried to implement too many features in parallel and was overwhelmed by the amount of work necessary to finish it. This situation became even worse when some individuals decided not to use the XP practice pair programming and accomplished most of their work alone, leaving the rest of the team clueless about their contribution to the code. As a result the team ended up with a heap of unfinished code and after around a year the sub-project came to a halt. The team decided to start over again.

The second attempt to restart was not successful either. The application did not meet its usability goals and was abandoned again.

By this time the team members' motivation was understandably very low. They struggled with the code, their workflow and their team spirit. Disillusioned by the failed attempts the team coordinator of this sub-project thought that the team could not solve their problems all by themselves and asked me for help.

The main problems of this team seemed to be: underestimation of the tasks at hand, insufficient adoption of agile practices and the overall workflow resulting in frustration and lack of motivation.

Cycle Zero: Get to know Kanban, your users and your stakeholders

Before starting the AR cycles I regularly attended the weekly team meetings. I observed the interactions in the team and of course got to know the team members. By attending these meetings I got the impression that the team was targeting a different main user group than the project head envisioned. I had of course previously talked with him about the sub-project and its target group. As far as I knew the project head wanted to target beginners in the domain and it seemed to me that the team was targeting people with at least intermediate knowledge in the domain. This assumption led me to add a cycle zero to the study. I wanted to clarify, if there was indeed a misunderstanding between the team and the project head, therefore I moderated a user analysis with the team. This user analysis yielded some unforeseen results and led to a repositioning of the sub-project within the umbrella project with some major changes for the team and its interaction with other teams. Therefore, I performed a stakeholder analysis with the team as well.

Action Planning and Action Taking The user analysis was done as a simple brainstorming exercise, where we collected all possible user groups team members could think of on a whiteboard. Afterwards, the team chose the main user groups for which they were developing the application.

The stakeholder analysis was conducted according to Leopold and Kaltenecker [START_REF] Baumeister | Kanban in der IT -Eine Kultur der kontinuierlichen Verbesserung schaffen[END_REF]. First, the team determined all stakeholders and listed them on individual pieces of paper. Each paper was sized differently, reflecting the importance of the stakeholder for the team's long-term success. Then, the pieces were put on a table and arranged around the team's mission, which is at the center of the analysis. Stakeholders, who are affected more by the team's day to day work and possible changes, are placed nearer to the center. Stakeholders, who are affected less, are placed further away. Afterwards, the frequency of relationships between all stakeholders was determined. The stronger the relationship, the more lines were drawn between two stakeholders. At last the quality of these relationships was determined as friendly, adversarial, love-hate or unknown. For an exemplary stakeholder analysis see Figure 1. Evaluating and Specified Learning The user analysis showed some discrepancy between the main user group the team was targeting and the main user group the project head envisioned. I articulated my impression to the team and we concluded that the team had to talk to the project head. Luckily for us he was available at the time of the user analysis and we asked him to join the discussion. And indeed there was a misunderstanding. He explained to the team which user group they should target and why and it became obvious, that the team had been developing their application for a different target group. This was a very unexpected result for the team and the whole software development came to a halt. The team had to redefine their goals and to rethink their application. Two meetings with the project head followed and the UX team was contacted as well. The project head and the team discussed the goals for their sub-project. The UX team did a user inquiry with four members of the targeted user group. The results of the user inquiry were discussed with the developing team. Based on the input of the four possible users, the developing team and the project head, the UX team designed a digital mockup of the application. This mockup showed an absolutely different Graphical User Interface (GUI) than the current application. It was more intuitive for novice users, but more elaborate to develop. Thus, the team had to reimplement the whole GUI. More importantly the former independent stand-alone application was integrated into another application of the umbrella project. This decision was made jointly by the sub-team, the super-team and the project head. The main reasons for this decision were: the sub-project will extend the functionality of the super-application with some very important features. The sub-project will reach more users as an extension than as a stand-alone application because the super-application is already publicly available and has a steadily growing user base.

As a result of this repositioning from stand-alone application to extension and because it is recommended in Leopold and Kaltenecker [START_REF] Baumeister | Kanban in der IT -Eine Kultur der kontinuierlichen Verbesserung schaffen[END_REF], I performed a stakeholder analysis with the team. The analysis showed that the intensity and quality of some relationships between stakeholders were unknown to the team and that the team had to work on intensifying some relationships, especially the one with the new super-team.

First Cycle

Action Planning and Action Taking The team already used a Jira Kanban board to visualize the workflow. The board contained the following columns: backlog, in development, done, done and accepted.

Team policies were determined in an open discussion. Team members collected all policies on a white board and discussed their meaning and importance jointly. After agreeing on a set of policies, they were transferred to the project wiki.

Evaluating and Specified Learning The main focus in this cycle was making policies explicit. We regarded visualize the workflow as already finished, because the team already used a board. Additionally, as a consequence of cycle zero the software development was put on hold and the team focused on redefining their goals, hence there was no activity on the board at the Making policies explicit yielded some interesting effects: Team members discussed their unspoken policies for the first time and discrepancies showed. Some were of semantic nature others reflected a different view of processes. During the discussion some problematic habits were identified and immediately discussed. To prevent these habits from reoccuring, new team policies were stipulated jointly and team members agreed to honor those policies.

Second Cycle

Action Planning and Action Taking Limiting work in process and managing flow were introduced to the team during the second Kanban coaching session. Different visualization possibilities were shown and the importance of limiting WIP and its on transition time, based on Little's Law [START_REF] Little | Little's law[END_REF], were explained. It was also discussed how WIP limits could be used to make problems visible and improve flow.

Evaluating and Specified Learning We revisited the Jira board to limit WIP. In the course of doing so we discovered something we already should have uncovered during the first cycle. The current board did not model a pull system. It was rather a push system. While team members pulled tasks from the backlog into development, the transition from development to code acceptance was a push process. Developers pushed the task from development to done where senior developers had to take them and move them to done and accepted. There was not even a state for being in acceptance. It was not possible to determine whether a task was already in the process of being accepted, or if it was still simply marked as done. I think this oversight can be ascribed to the focus on redefining the goal, which was still in progress during the first cycle.

To repair this, a new state was introduced into the workflow and the column in acceptance was added to the board, now reflecting a real pull system. The columns in development and done were merged into one column. Initially, the team wanted to create two subcolumns for in development, but Jira does not offer this functionality. Therefore we experimented with different possibilities of visualization. Figure 2 depicts on the one hand the desired visualization and on the other hand the current visualization, which is realizable within the constraints of Jira.

Afterwards, WIP limits were set and monitored. They soon revealed a bottleneck at the acceptance state. The root cause was quickly identified and team members were working hard to resolve this bottleneck. Due to the merge with the super-project the team now depended on the senior developers of the superproject to accept the sub-team's code. The senior members of the sub-team were and still are working hard to familiarize themselves with the slightly different acceptance process of the super-team and thus dissolving the bottleneck as soon as possible and improving flow. Another positive effect of the focus on the board is, the team now uses their Kanban board during each meeting, which it did not do prior to the AR cycles. While this study can not conclusively answer the research question "Can FOSS and ASD be comprehensively combined?" it shows some promising outcomes. Integrating Kanban and FOSS has so far been successful and beneficial the research participants. New insights, e.g. on the target user group, have been gained and the team's workflow has become more effective. From my point of view, motivation has improved but this remains to be verified in the future.

Extending the Kanban practices

Based on the experiences described in Section 5, we propose two additional Kanban practices for FOSS projects.

-Conduct regular user interviews or feedback sessions with your users -Review your assumptions about your current development practices These recommendations are of course based on a hybrid student FOSS project, but usual FOSS projects could benefit from these additional practices as well. While most companies applying agile and lean practices have a marketing and sales team or even a user focus group, FOSS projects tend not to have this kind of resources. Nowadays, many FOSS solutions are employed by a large number of people, who do not contribute to the code, e.g. Mozilla Firefox or Linux. Thus, FOSS developers are not simply "scratching their own itch" anymore, they serve many people, who must not share the developer's requirements and domain knowledge, all over the world. Therefore, it could be beneficial for FOSS projects to investigate their users' needs. As for the second recommendation: Although, we agree with the Kanban principle "start where you are", we assume it could be beneficial for FOSS projects to review their current development practices before embarking on the endeavor of integrating Kanban, or any other agile or lean method, into their development process. FOSS projects usually do not have SCRUM masters, process experts or in general someone, who controls whether software development practices are exercised correctly. An honest and critical reflection about the current practices can clear some misconceptions, can further a joint understanding of the current situation and it is a first step into the direction of kaizen.

Limitations

This study is limited to a single case of a hybrid student FOSS project, therefore it has only very limited external validity. Only one team is part of this study therefore the results are not generalizable to other teams or projects without further research.

Another limitation may be my positionality [START_REF] Herr | The Action Research Dissertation -A Guide for Students and Faculty 2nd Edition[END_REF] in the setting. Herr and Anderson [START_REF] Herr | The Action Research Dissertation -A Guide for Students and Faculty 2nd Edition[END_REF] describe positionality as asking the question, "Who am I in relation to my participants and my setting?". Subsection 3.2 already identified power distance as a possible limitation. The informal power position may result in research bias, since suggestions I make could be accepted due to this perceived power distance and not only because team members agree with these suggestions. In an attempt to counterbalance this bias a bit, whenever possible at least two alternatives were proposed and the final decision was made by the team.

The setting of a hybrid student FOSS project might also be seen as a limiting factor, because some characteristics differ. Teams members contribute to receive course credits and not (only) to earn a reputation among other developers or out of altruistic motives. Student contributors change regularly and there exists no group of core or chief developers, which usually consists of 10% to 20% of a team, and which creates around 80% of the source code [START_REF] Koch | Agile principles and open source software development: A theoretical and empirical discussion[END_REF]. Future research could determine if or how these diverging characteristics impact a FOSS project. Students as main contributors to this project may also be considered a limitation, because they have not finished their studies. But people from different backgrounds contribute to FOSS projects regardless of their formal education and IT students work as normal developers on many FOSS projects. Many FOSS contributors have no formal software engineering education at all. Salman et al. [START_REF] Salman | Are students representatives of professionals in software engineering experiments[END_REF] even observed that students and professionals show similar performances in carefully scoped software engineering experiments, when the development approach used is new to both groups.

Conclusion

This paper describes a practical integration of Kanban through AR in the context of a hybrid student FOSS project. Based on the findings of this work we proposed additional Kanban practices for FOSS projects. There is a lack of research regarding integration of ASD in the FOSS development context [START_REF] Gandomani | A systematic literature review on relationship between agile methods and open source software development methodology[END_REF]. Thus, this paper contributes by offering some insights on the matter. As future work we plan on conducting more AR cycles with the same team to exercise the practices from the first and second cycle and further introduce the remaining Kanban practices into the workflow. To assure validity of the proposed additions to the Kanban approach, the cycles described in this paper should be conducted with other teams and within other FOSS settings.

Fig. 1 .

 1 Fig.1. Exemplary stakeholder analysis, adapted from Leopold and Kaltenecker[START_REF] Baumeister | Kanban in der IT -Eine Kultur der kontinuierlichen Verbesserung schaffen[END_REF]

Fig. 2 .

 2 Fig. 2. The desired visualization on the left side and the actual visualization, which is possible in Jira, on the right side

https://youtu.be/6nOUa6E0250


Toshiki Hirao
email: hirao.toshiki.ho7@is.naist.jp

Akinori Ihara
email: akinori-i@is.naist.jp

Yuki Ueda

Passakorn Phannachitta
email: phannachitta-p@is.naist.jp

Ken-Ichi Matsumoto
email: matumoto@is.naist.jp

The Impact of a Low Level of Agreement among Reviewers in a Code Review Process

Keywords: Modern Code Review, Software Development, Agreement

Software code review systems are commonly used in software development. In these systems, many patches are submitted to improve the quality. To verify the quality, voting is commonly used by contributors; however, there still exists a major problem, namely, that reviewers do not always simply reach a broad agreement. In our previous study, we found that consensus is not usually reached, implying that an individual reviewer's final decision usually differs from that of the majority of the other reviewers. In this study, we further investigate the reasons why such situations often occur, and provide suggestions for better handling of these problems. Our analysis of the Qt and OpenStack project datasets allow us to suggest that a patch owner should select more appropriate reviewers who often agree with others' decisions.

INTRODUCTION

Software code review is a process in which reviewers and committers verify patches. In particular, Open Source Software (OSS) projects conduct a review to release higher quality and readability source codes [START_REF] Kochhar | An empirical study of adoption of software testing in open source projects[END_REF]. Endorsed by McIntosh et al. [START_REF] Mcintosh | The impact of code review coverage and code review participation on software quality: A case study of the qt, vtk, and itk projects[END_REF], a code review with a sufficient discussion is one of the most useful practices to contribute to a more effective bug detection process. In addition, a code review has proved to be very helpful in providing important feedbacks to developers for future developments [START_REF] Thongtanunam | Who should review my code? a file location-based codereviewer recommendation approach for modern code review[END_REF].

Nowadays, we have various dedicated tools for managing the code review process. For example, Gerrit 1 and ReviewBoard 2 are commonly used by OSS practitioners to improve the quality of their source codes. Technically, conducting a code review process using these tools is done through patch submissions and reviews, and voting is known to be one of the most commonly used practices to decide whether or not a patch should be integrated into a version control system (i.e., to be accepted or not). In general situation, a patch with higher quality will more likely be accepted than a patch with lower quality; however, not all submitted patches are of high quality. In such cases, voting results can be varied and thus adding more difficulty in deciding whether or not to use those patches. In other words, a final decision cannot be made unless reviewers and committers have reached a consensus.

In our previous study [START_REF] Hirao | Pilot study of collective decisionmaking in the code review process[END_REF], in practice, consensus is not usually reached through the voting system. As a continuation of our previous study, this study further investigates how often a reviewer disagrees with a review conclusion, and what the impact is of a reviewer with a low level of agreement. Hence, we conduct a case study using Qt and OpenStack project datasets to address the following research questions: RQ1: How often does a reviewer disagree with a review conclusion? Results: A more experienced reviewer is likely to have a higher level of agreement than a less experienced reviewer.

RQ2: What is the impact of a reviewer with a low level of agreement in a code review?

Results: A review assigned to a reviewer with a lower level of agreement is more likely to take a longer reviewing time and discussion length. This paper is arranged as follows. Section 2 describes the background to this paper, related work, and a motivating example. Section 3 provides the design of our two research questions and datasets. Section 4 presents the results with respect to our two research questions. Section 5 discusses a qualitative analysis of our research questions, and addresses the threats to validity. Finally, Section 6 concludes this paper and describes our future work.

BACKGROUND

Modern Code Review (MCR)

In recent years, MCR has become popular and widely used in both proprietary software and open source software [START_REF] Bacchelli | Expectations, outcomes, and challenges of modern code review[END_REF]. The use of MCR has made the review process more traceable, which in turn, has created opportunities for empirical software engineering researchers to analyze this process [START_REF] Mcintosh | The impact of code review coverage and code review participation on software quality: A case study of the qt, vtk, and itk projects[END_REF], [START_REF] Thongtanunam | Who should review my code? a file location-based codereviewer recommendation approach for modern code review[END_REF], [START_REF] Bosu | Peer code review in open source communitiesusing reviewboard[END_REF], [START_REF] Rigby | Convergent contemporary software peer review practices[END_REF], [START_REF] Morales | Do code review practices impact design quality? a case study of the qt, vtk, and itk projects[END_REF]. For example, Hamasaki et al. [START_REF] Bosu | Peer code review in open source communitiesusing reviewboard[END_REF], [START_REF] Hamasaki | Who does what during a code review? datasets of oss peer review repositories[END_REF], [START_REF] Mukadam | Gerrit software code review data from android[END_REF] collected rich code review datasets from a collection of open source projects using the Gerrit and ReviewBoard code review tools.

Patch-related activities in MCR

Jiang et al. noted that a submitted patch is not always merged into a version control system [START_REF] Jiang | Will my patch make it? and how fast?: Case study on the linux kernel[END_REF]. Examples of abandoned patches include patches with un- fixed bugs, irrelevant comments, or duplicate patches [START_REF] Tao | Writing acceptable patches: An empirical study of open source project patches[END_REF]. To verify the patches adequately, McIntosh et al. [START_REF] Mcintosh | The impact of code review coverage and code review participation on software quality: A case study of the qt, vtk, and itk projects[END_REF] suggested that reviewers discuss the patches carefully and try to reach a consensus in their discussions. In this way, the final decision of whether or not to use a patch can be made simply based on the reviewers' discussions.

Figure 1 provides an overview of a code review process when using a code review tool. In particular [START_REF] Thongtanunam | Who should review my code? a file location-based codereviewer recommendation approach for modern code review[END_REF], the code review process after patch submission acts as follows:

(1) A patch owner clones a source code repository from a web-based version control system service such as GitHub to their local computers. Next, the patch owner creates patches to fix a bug or to enhance patches. After that, the owner submits the patches to a web-based code review tool.

(2)The patch owner requests that reviewers verify the submitted patches. The reviewers verify the changes. Next, the reviewers post an approving positive vote or a disapproving negative vote, and sometimes also post comments.

(3)If the patch owner needs to revise his or her patches, he or she will update the patches.

(4)If the patches are approved by the reviewers and a committer, the committer will M erge the patches into the main repository. On the other hand, if the patches are not approved by the reviewers and a committer, the committer will Abandon the patches. In this paper, Merge and Abandon are referred to as a final review conclusion.

Weißgerber et al. [START_REF] Weißgerber | Small patches get in![END_REF] found that a smaller patch is more likely to be accepted. In addition, Tao et al. [START_REF] Tao | Writing acceptable patches: An empirical study of open source project patches[END_REF] investigated reasons why a patch was rejected through quantitative and qualitative analysis. Furthermore, to verify a patch adequately, Thongtanunam et al. [START_REF] Thongtanunam | Who should review my code? a file location-based codereviewer recommendation approach for modern code review[END_REF] and Xia [START_REF] Xia | Who should review this change?: Putting text and file location analyses together for more accurate recommendations[END_REF] proposed approaches to recommend a appropriate reviewer for patches submitted by a patch owner based on reviewers' experiences.

Table 1. The voting patterns made in the final review conclusion in Qt.

N = 0 N = 1 N = 2 N = 3 N = 4 N = 5 N = 6 N = 7 P = 0 56, 4268 25, 2090 16, 505 3, 93 2, 23 0, 2 0, 0 0, 0 P = 1 40425, 968 1475, 571 135, 122 19, 28 3, 6 0, 1 0, 0 1, 0 P = 2 11748, 247 920, 135 81, 38 20, 12 5, 0 1, 3 0, 0 0, 0 P = 3 2686, 51 316, 29 54, 6 8, 1 1, 0 0, 0 0, 0 0, 0 P = 4 587, 10 79, 8 17, 1 6, 1 1, 0 1, 0 0, 0 1, 0 P = 5 119, 2 36, 1 7, 1 1, 0 0, 0 1, 0 0, 0 0, 0 P = 6 17, 0 3, 0 4, 0 0, 0 1, 0 0, 0 0, 0 0, 0 P = 7 6, 0 2, 0 0, 0 0, 0 0, 0 0, 0 0, 0 0, 0

A Motivating Example: The Code Review Collaboration Among Reviewers

Based on practical observations explained in the following of this section, we would like to seek a better understanding of disagreements among reviewers and committers. Table 1 and Table 2 show the number of patches in each voting pattern in the two projects when a committer makes the final review conclusion using Qt and OpenStack project data. Each cell has #Merged patches (a value on the left) and #Abandoned patches (a value on the right). These two tables show the patches with less than seven positive votes or negative votes. Then, 91% of patches in Qt and 91% of patches in OpenStack are covered in whole patches.

15% of patches in the Qt and 31% of patches in the OpenStack of all the patches do not follow the majority rule that selects alternatives which have a majority. For example, in Table 2, 467 patches are abandoned, even though #Positive is greater than #Negative in this case. This indicates that even if a patch received higher #Positive, in practice, the patch is not guaranteed to be accepted. Therefore, we believe that not all reviewers who post a vote, which later be in agreement with the final review conclusion in the code review discussion. Thus, in this paper, we study the impact of a reviewer with a low level of agreement in a code review collaboration.

CASE STUDY DESIGN

This study considers two research questions to understand the impact of a reviewer who disagrees with the review conclusion of a patch in a code review collaboration. In the following section, we provide detailed explanations of the case study designed to answer these research questions, and address our experimental Dataset.

Research Questions

RQ1: How often does a reviewer disagree with a review conclusion?

Motivation. Rigby et al. [START_REF] Rigby | Understanding broadcast based peer review on open source software projects[END_REF] pointed out that code reviews are expensive because they require reviewers to read, understand, and assess a code change. Thongtanunam et al. [START_REF] Thongtanunam | Who should review my code? a file location-based codereviewer recommendation approach for modern code review[END_REF], [START_REF] Xia | Who should review this change?: Putting text and file location analyses together for more accurate recommendations[END_REF], [START_REF] Aurum | State-of-the-art: software inspections after 25 years[END_REF] showed that to effectively assess a code change, a patch owner should find an appropriate reviewer who has a deep understanding of the related source codes to closely examine code changes and find defects. In other words, the appropriate reviewer is more likely to assess the patches adequately. However, this reviewer might not always agree with a review conclusion.

Approach. To answer the first research question, we analyze the differences of a reviewer's agreement according to a reviewer's experience (the number of votes in the past). A committer confirms the reviewers' votes to decide a final review conclusion (Merge or Abandon). To analyze the review decision results, we scan comments for the known patterns of automatically generated voting comments in Gerrit as shown in Table 3. Table 3 illustrates five level validating scores ("+2", "+1", "0", "-1", "-2") in Gerrit. A reviewer can vote "+1", "0" and "-1" score. On the other hand, a committer can vote "+2", "+1", "0", "-1" and "-2" for the score [START_REF] Milanesio | Learning Gerrit Code Review[END_REF]. In our study, we use these vote scores "+2", "+1", "0", "-1" and "-2" to analyze assessments by reviewers and committers .

Next, to identify the frequency of votes that disagreed with the review conclusions, we track the voting history of each reviewer. We count the number of times that a reviewer disagreed with the review conclusion of a patch in the past. We need to be careful when calculating the frequency, because some patches have often been updated. In the case where a patch has been updated twice, a reviewer will have two chances to post a vote. In this case, we count the votes twice. Figure 2 shows an example of a voting process. A reviewer X Table 3. The patterns of automatically generated voting comments in the Gerrit.

Role

Score Automatically generated voting comments Committer +2 "Looks good to me, approved" "Looks good to me" Reviewer +1 "Looks good to me, but someone else must approve" "Works for me" "Code-Review +1" "Workflow +1" "Verified" 0 "No score" -1 "I would prefer that you didn't submit this" "I would prefer that you didn't merge this" "Code-Review -1" "Workflow -1" "Doesn't seem to work" Committer -2 "Fails" "Do not merge" "Do not submit" reviews two patches, namely Patch01.java and Patch02.java. As shown in the figure, his or her first positive vote disagrees with the first review conclusion, which was decided upon to be "Updated", because a positive vote implies that this patch does not have any problems. After the patch is updated, his or her second vote is still positive. Finally, the vote of the reviewer X agrees with the final conclusion, which is "Merged". In this case, the rate of agreed upon votes for reviewer X is 1 2 =0.5 (Reviewer X's level of agreement = 50%). Also, a reviewer Y posts a negative vote for only Patch01.java. The rate of agreed votes for the reviewer Y is 1 1 =1.0 (Reviewer Y 's level of agreement = 100%). The level of agreement has a range between 0.0 and 1.0. In summary, Table 4 describes the definition of the agreement and disagreement votes. If a reviewer posts a positive vote (+2 or +1) for a patch and a committer decides to merge this patch, the vote is a vote of agreement. On the other hand, if the committer decides to abandon this patch or a patch owner updates this patch, the vote is a vote of disagreement.

RQ2: What is the impact of a reviewer with a low level of agreement in a code review?

Motivation. In a code review, when a discussion does not always reach a consensus among reviewers and committers, it may take much longer time to completely finish the code review. In addition, it may not be simple to identify which vote a committer should believe. In this second research question, we investigate the impact of code review collaboration with a reviewer who often disagrees with a review conclusion using the reviewer's level of agreement measured in the RQ1. Approach. In this research question, we focus on the minimum level of agreement of reviewers as calculated in each individual review. According to Figure 2, a minimum level of agreement in this example is the level of agreement of reviewer X's, which is 0.5. The minimum level of agreement has a range between 0.0 and 1.0. When the value of a minimum level of agreement is low, it can be implied that the review has a reviewer who often disagrees with a review conclusion. In our opinion, a lower level of minimum agreement has a negative effect on the code review process. To further analyze the negative effect, we define two technical terms as follows:

Reviewing Time: The time in days from the first patch submission to the final review conclusion. We hypothesize that a reviewer with a lower level of agreement may take much longer time to reach a consensus in the discussion. To reduce the chance of a release postponement, the Reviewing Time should be shorter.

Discussion Length: The number of comments which reviewers post into a reviewing board. We hypothesize that a reviewer with a lower level of agreement may disagree more often, so that such a code review needs a much longer discussion period to reach a consensus among reviewers and committers.

Experimental Dataset

We conduct a case study on two large and successful OSS projects, namely, Qt3 and OpenStack4 . These two projects are commonly found in the literature on OSS studies, such as in [START_REF] Mcintosh | The impact of code review coverage and code review participation on software quality: A case study of the qt, vtk, and itk projects[END_REF], [START_REF] Thongtanunam | Who should review my code? a file location-based codereviewer recommendation approach for modern code review[END_REF], [START_REF] Hirao | Pilot study of collective decisionmaking in the code review process[END_REF], [START_REF] Bosu | Peer code review in open source communitiesusing reviewboard[END_REF], [START_REF] Hamasaki | Who does what during a code review? datasets of oss peer review repositories[END_REF], [START_REF] Thongtanunam | Investigating code review practices in defective files: An empirical study of the qt system[END_REF] mainly because these projects contains a large amount of reviewing activity using a code review tool.

Figure 5 shows that originally Qt and OpenStack had 70,705 and 92,984 review reports, respectively. However, we are concerned that reports consisting of no votes or only bot tests' votes are not useful because we focus on a human reviewer. Therefore, we filter out those review reports prior to our case studies. In more detail, we exclude the earliest (oldest) 10% from the two datasets. Based on our observations, most of the data points falling in this range have insufficient beneficial information for the reviewer's agreement's calculation. The review reports used in this study includes 55,523 review reports in the Qt project and 56,038 review reports in the OpenStack.

CASE STUDY RESULTS

RQ1: How often does a reviewer disagree with a review conclusion?

Figure 3 shows the relationship between a reviewer's level of agreement and the number of votes in the past using a Hexagon binning plot. We define a reviewer who has a lower number of votes than the median of the total votes as a less experienced reviewer, and one who has a higher number of votes than this median value as a more experienced reviewer, where the median values of the number of votes in Qt and OpenStack are 8 and 11 votes without an outlier, respectively. From this figure, we found that the more experienced reviewers are more likely to have a higher level of agreement than the less experienced reviewers.

Previous studies [START_REF] Thongtanunam | Who should review my code? a file location-based codereviewer recommendation approach for modern code review[END_REF], [START_REF] Xia | Who should review this change?: Putting text and file location analyses together for more accurate recommendations[END_REF] suggested identifying an appropriate reviewer based on expertise; however, expertise is not necessarily associated with the agreement of a review conclusion. We therefore suggest one more criteria when choosing an appropriate reviewer based on the results of this experiment. That is, if a patch owner needs the reviewers to reach a consensus for their patches as soon as possible, we suggest that the patch owner invites reviewers with a higher level of agreement.

RQ2: What is the impact of a reviewer with a low level of agreement in a code review?

We begin the investigation for our RQ2 with a quantitative analysis of the reviews disagreed upon among reviewers in the two projects. After that, we further analyze the impact of a reviewer with a low level of agreement in terms of Reviewing Time and Discussion Length.

Figure 4 shows the rate of the patches disagreed upon according to a minimum agreement of Qt and OpenStack, respectively. In both projects, we found that 11% of reviews in the Qt and 65% of reviews in the OpenStack did not reach a consensus among reviewers and a committer. In more detail, when a patch owner invites a reviewer who has a minimum level of agreement between The minimum level of agreement The rate of disagreed reviews OpenStack Fig. 4. The rate of review disagreements 10% and 20%, we found that 64% of reviews in the Qt and 70% reviews in the OpenStack did not reach a consensus among reviewers and a committer. Observed by correlation, we found that the rate of disagreed upon reviews and the minimum level of agreement exhibited strongly negative values (r=-0.79 in Qt and r=-0.90 in OpenStack). This means that a decision made by a reviewer with a lower level of agreement is less likely to reach a consensus among those of the other reviewers. Reviewing Time. Figure 5 shows the distribution of reviewing time according to a minimum level of agreement. From this figure, in the Qt project, many patches with a reviewer who has a minimum level of agreement between 70% -90% are more likely to take a much longer reviewing time than others. On the other hand, in OpenStack, the minimum level of agreement appears to be inapplicable to explaining whether or not a reviewer would take a longer reviewing time. Since the features of these distributions seems to be dependent on an individual project, we therefore perform a further analysis of these two projects using a statistical method.

Figure 6 classifies the reviews into two groups, i.e., the Top 50% and the Remaining 50% of the population, where the cut-off is determined using the median values of the minimum level of agreement for each project. In Qt, the values of the minimum, the lower quartile, the median, the upper quartile and the maximum are 0.00, 0.78, 0.83, 0.88 and 1.00, respectively, and that of the OpenStack project are 0.00, 0.40, 0.54, 0.70 and 1.00, respectively. We found that the reviews of the Top 50% are likely to take a shorter reviewing time than those of the Remaining 50%. Confirmed by the Wilcoxon signed-rank test with a p-value less than 0.01, we found that the difference in the distributions between the Top 50% and the Remaining 50% in both Qt and OpenStack are statically significant. Discussion Length. Figure 7 shows that the distribution of discussion length (the number of comments in a review) according to the minimum level of agreement. From this figure, in Qt, many patches with a reviewer who has a minimum level of agreement between 60% -90% more likely to take a much longer time for discussion than others. On the other hand, in OpenStack, many patches with a reviewer who has a minimum level of agreement with less than 20% more likely to take a much longer time for discussion than others. Similar to the Reviewing Time, we found that the features of these distributions also depend on an individual project. Hence, we further perform an analysis based on statistical method as done when we observed the Reviewing Time. Figure 8 shows that the Top 50% are more likely to take a shorter Discussion length. These results were also confirmed by Wilcoxon signed-rank test as being statistically significant with a p-value less than 0.01.

DISCUSSION

In this section, we conduct a qualitative analysis to better understand why the reviewers with a low level of agreement often disagrees with a review conclusion and make the review process longer by reading actual review discussions. In addition, this section also discloses the threats to validity of this study.

A Qualitative Analysis

A reviewer with a lower level of agreement is more likely to overlook problems of source codes. We read an actual review report ID 8141 5 and found that a reviewer named Claudio had overlooked the problem of the test cases, and we found that he disagreed with the final review conclusion (Abandon) of the review. To prove that our approach can explain this case, we calculated the level of agreement of this reviewer and found that the level of agreement of this reviewer was low (classified as Remaining 50%). In addition, we also found similar cases 6 7 as being in agreement with the analysis result of the review ID 8141.

Inviting another reviewer might take a longer discussion period. We read an actual review report ID 28,257 8 and found that a record saying that reviewers assigned to this review could not complete this review and therefore asked another reviewer to verify the patches. For example, in a review ID 28,257, the first reviewer who had a low level of agreement (classified as the Remaining 50%) posted a positive vote and the second reviewer who had a high level of agreement (classified as the Top 50%) posted a negative vote. Therefore, this discussion did not reach a consensus among reviewers. In the detailed record of this review, we found that the second reviewer said that "I recommend asking JsonDb maintainer / developers to review this change before rubber stamping it", indicating that it seemed not to be easy for the second reviewer to make a review conclusion using the first reviewer's review. In addition, we also found cases similar to this review report ID 28,257, which are 9 10 . We therefore suggest that a patch owner invites an appropriate reviewer (i.e., with a higher level of agreement) in order to reach a final review conclusion more easily.

Threats to Validity

External validity. We found that a reviewer with a lower level of agreement takes a longer reviewing time and discussion length. We might obtain new findings if we investigate another project with different characteristics than Qt and OpenStacks. Nonetheless, as our results were subject to a robust statistically significant test, we believe that the results probably would not be much different than what we did found in this study. Internal validity. Not all disagreed upon reviews have problems. It is possible that a reviewer might disagree with a committer's decision, but the both opinions could be correct. This situation might express a valuable discussion. We will investigate deeply the disagreed upon discussion process to determine whether or not a disagreed upon discussion is valuable for a patch integration. In addition, we should also take into consideration the rank of the reviewer's role in the Gerrit system. A maintainer and committer have stronger rights compared to a reviewer, so that a reviewer is likely to agree with their opinions. Therefore, we would like to investigate the level of agreement by considering the rank of the reviewer's role.

CONCLUSION

In this study, we have investigated the impact of reviewers with a low level of agreement using Qt and OpenStack. In our research questions, we found that a more experienced reviewer is more likely to have a higher level of agreement than those who have less experience, and a review assigned to a reviewer with a lower level of agreement is more likely to take a longer reviewing time and discussion length. In our discussion, we found that a reviewer with a lower level of agreement often overlooks problems presented in source codes. From the findings of this study, we suggest that a patch owner should invite an appropriate reviewer to review the patch, where appropriateness is to be determined by using the level of agreement calculated in the entire project. In the future, we will propose a more sophisticated method to invite an appropriate reviewer based on additional important criteria. We believe that this future direction will contribute to a more efficient code review process.

!Fig. 1 .

 1 Fig. 1. An Overview of Modern Code Review Processes

Fig. 2 .

 2 Fig. 2. An example of a voting process.

Fig. 3 .

 3 Fig. 3. The relationship between a reviewer's level of agreement and the number of votes.

Fig. 5 .Fig. 6 .Fig. 7 .

 567 Fig. 5. The distribution of Reviewing Time

Fig. 8 .

 8 Fig. 8. The difference of the Discussion Length between the Top 50% and the Remaining 50%

Table 2 .

 2 The voting patterns made in the final review conclusion in OpenStack.

		N = 0	N = 1	N = 2 N = 3 N = 4 N = 5 N = 6 N = 7
	P = 0	141, 20 4243, 3155 1193, 1214 351, 344 133, 116 47, 47 20, 9 2, 6
	P = 1 11964, 1047 3497, 1444 1185, 752 436, 298 157, 122 67, 37 27, 16 8, 10
	P = 2	6269, 264 2254, 467 865, 302 324, 134 126, 64 54, 25 15, 14 9, 5
	P = 3	3069, 90 1317, 211 514, 124 251, 68 85, 31 33, 25 15, 9 11, 2
	P = 4	1519, 42 690, 106	303, 55 134, 39 51, 25 24, 7 11, 2 1, 3
	P = 5	687, 30	357, 50	146, 37 86, 20 45, 13 13, 5 10, 2 1, 2
	P = 6	370, 13	193, 36	91, 25 52, 13	15, 5 17, 5 8, 2 0, 2
	P = 7	182, 10	116, 29	60, 17	36, 7	23, 5 4, 3 3, 3 3, 2

Table 4 .

 4 The definition of agreement and disagreement patterns

				Review conclusion
				Merge Abandon Update
	Reviewer's Positive (+2 or +1)	agree disagree disagree
	vote	Negative (0, -1 or -2) disagree agree	agree
	#include <stdio.h>			#include <stdio.h>
	void main() { int i, j, a, n, number[30];	X!	Y!	void main() { int i, j, a, n, number[30];
	printf("Enter the value			printf("Enter the value
	of N \n")			of N \n")
	scanf("%d", &n);			scanf("%d", &n);
	printf("Enter the			printf("Enter the
	numbers \n");			numbers \n");
	}			}

Table 5 .

 5 Summary of the studied datasets

	Qt OpenStack

  https://codereview.qt-project.org/#/c/28257 9 https://codereview.qt-project.org/#/c/1048 10 https://codereview.qt-project.org/#/c/7375

	5 https://codereview.qt-project.org/#/c/8141
	6 https://review.openstack.org/#/c/10363
	7 https://review.openstack.org/#/c/10305

8

Gerrit: https://code.google.com/p/gerrit/

ReviewBoard: https://www.reviewboard.org/

http://qt.digia.com/

http://www.openstack.org/

Acknowledgment

This work has been conducted as part of our research under the Program for Advancing Strategic International Networks to Accelerate the Circulation of Talented Researchers.


Stephen Murphy
email: stephen.murphy@bcu.ac.uk

Sharon Cox
email: sharon.cox@bcu.ac.uk

Classifying Organizational Adoption of Open Source Software. A Proposal

Keywords: open source software, technology adoption, staged adoption model, innovation diffusion theory, organizational adoption, factor based model, adoption pathway 1 Awareness

Staged adoption models are a common feature of information systems (IS) adoption literature, yet these are rarely used in open source software (OSS) adoption studies. In this paper, a staged model for classifying the organizational adoption of OSS is proposed, based upon a critical review of existing staged adoption models and factors identified from OSS adoption literature. Innovations in the proposed model include: defined transition pathways between stages, additional stages and a decomposition of cessation of use into four distinct pathways.

Introduction

Software developed by communities using the open source methods espoused by Raymond [START_REF] Raymond | The Cathedral and the Bazaar[END_REF] is increasing in popularity [START_REF] Black | The Ninth Annual Future of Open Source Survey[END_REF]. While exact usage figures are uncertain, some studies have put usage levels as high as 85% [START_REF] Gartner | User Survey Analysis: Open-Source Software[END_REF] and in some specialist fields, close to 100% [4]. Open Source Software (OSS) usage had previously been the preserve of programmers and software experts [START_REF] Levesque | Fundamental Issues with Open Source Software Development[END_REF][START_REF] Raza | Users' Perception of Open Source Usability: An Empirical Study[END_REF], but this 'second wave' [START_REF] Choi | An Exploratory Study on the Two New Trends in Open Source Software: End-Users and Service[END_REF] of OSS adoption by businesses and non-technical users has led to greater press and academic attention [START_REF] Dafermos | Images of Innovation in Discourses of Free and Open Source Software[END_REF]. However, much of this attention has focused on OSS development methods and processes [START_REF] Grand | Resource Allocation Beyond Firm Boundaries[END_REF], with studies of adoption being underrepresented [START_REF] Aksulu | A Comprehensive Review and Synthesis of Open Source Research[END_REF][START_REF] Mount | Adoption of Free and Open Source Software Within High-Velocity Firms[END_REF][START_REF] Ramanathan | An Empirical Investigation into the Adoption of Open Source Software in Information Technology Outsourcing Organizations[END_REF][START_REF] Hauge | Adoption of Open Source Software in Software-Intensive Organizations -a Systematic Literature Review[END_REF]. This paper aims to partly address this gap through the achievement of the following objectives:

1. highlighting that most previous OSS adoption studies are not cognizant of the staged nature of the adoption process;

2. showing that where such studies are stage aware, that the models used to classify adoption are incomplete;

3. showing that existing staged adoption models do not identify all progression pathways;

4. and proposing a model to remedy shortcomings in existing models.

2 Literature Review

Technology Adoption

Adoption is defined as "choosing something for one's use or practice" [14]. The adoption of technology is a long standing area of academic research [START_REF] Venkatesh | Dead or Alive? The Development, Trajectory and Future of Technology Adoption Research[END_REF], but the focus has largely been on the individual as the unit of analysis [START_REF] Rogers | Diffusion of Innovations[END_REF]. While personal technology adoption may be a near-binary state, organizational adoption is a complex process [START_REF] Rogers | Diffusion of Innovations[END_REF] and will naturally follow a more structured and tentative path. This path may be non-linear [START_REF] Howcroft | The Social Shaping of Packaged Software Selection[END_REF], as options are explored, barriers uncovered and priorities identified.

Two broad classifications of adoption theory exist: process based and factor based [START_REF] Newman | A Social Process Model of User-Analyst Relationships[END_REF]. Numerous factor based theories have been proposed to aid in understanding the adoption of technologies, such as Technology Acceptance Model (TAM) [START_REF] Davis | A Technology Acceptance Model for Empirically Testing New End-User Information Systems: Theory and Results[END_REF], TAM2 [START_REF] Venkatesh | A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies[END_REF] Unified Theory of Acceptance and Use of Technology (UTAUT) [START_REF] Venkatesh | User Acceptance of Information Technology: Toward a Unified View[END_REF], Technology Organization and Environment (TOE) model [START_REF] Tornatzky | The Processes of Technological Innovation[END_REF] and Innovation Diffusion Theory (IDT) [START_REF] Rogers | Diffusion of Innovations[END_REF]. While there are numerous examples where these theories have been applied to studies of technology adoption (see [START_REF] Ellis | Open Source Software Adoption by South African MSEs: Barriers and Enablers[END_REF] and [START_REF] Gurusamy | A Case Study of Open Source Software Adoption in Australian Public Sector Organisations[END_REF] for OSS specific examples), there has been some criticism that this application has become formulaic, leading to a degree of stagnation [START_REF] Venkatesh | Dead or Alive? The Development, Trajectory and Future of Technology Adoption Research[END_REF]. Despite this criticism, there are few studies of OSS adoption [START_REF] Aksulu | A Comprehensive Review and Synthesis of Open Source Research[END_REF][START_REF] Mount | Adoption of Free and Open Source Software Within High-Velocity Firms[END_REF][START_REF] Ramanathan | An Empirical Investigation into the Adoption of Open Source Software in Information Technology Outsourcing Organizations[END_REF], and this is an area where understanding remains limited [START_REF] Poba-Nzaou | Risk of Adopting Mission-Critical OSS Applications: An Interpretive Case Study[END_REF].

While factor based theories allow the classification of drivers and barriers, as well as the ability to identify causality, they do not explain how the unit of analysis reached the observed level of adoption [START_REF] Newman | A Social Process Model of User-Analyst Relationships[END_REF]. Such single-epoch methods have been criticized as too simplistic to capture the full complexity of adoption [START_REF] Kurnia | The Need for a Processual View of Inter-Organizational Systems Adoption[END_REF], leading to a loss of processual detail [START_REF] Benbasat | Quo Vadis Tam?[END_REF]. Of the above theories, only IDT, with its commensurate Innovation Decision Process (IDP) [START_REF] Rogers | Diffusion of Innovations[END_REF] fully acknowledges a process view of adoption.

Fig. 1: Levels of OSS adoption (redrawn from Glynn et al., [28])

Staged adoption models are one method by which an adoption process can be more richly documented. Figure 1 shows an existing staged adoption model used by Glynn et al. [START_REF] Glynn | Commercial Adoption of Open Source Software: An Empirical Study[END_REF]. From this, it could be argued that an organization at any stage of this model, other than awareness or interest, could be said to be an adopter. Yet the commitment and maturity of the deployment for those in the evaluation/trial stage will differ greatly from those classified as a general deployment. Only if adoption studies classify their sample by stage can the level and commitment to that adoption be determined and stage specific barriers and drivers identified. A study that does not classify its sample risks missing stage significant factors due to heterogeneous sample composition and potentially limits comparability and generalizability.

Examples of the application staged adoption models to information technologies can be found dating back to the early days of mass-market computing (e.g. [START_REF] Nolan | Managing the Computer Resource: A Stage Hypothesis[END_REF]). However, while existing studies of OSS have sought to examine drivers and barriers to OSS adoption (e.g. [START_REF] Ellis | Open Source Software Adoption by South African MSEs: Barriers and Enablers[END_REF][START_REF] Nagy | Understanding Organizational Adoption Theories Through the Adoption of a Disruptive Innovation: Five Cases of Open Source Software[END_REF][START_REF] Kuechler | Misconceptions and Barriers to Adoption of FOSS in the US Energy Industry[END_REF]), few have identified at what stage adoption is within a unit of analysis [START_REF] Nelson | Understanding Open Source Software: A Research Classification Framework[END_REF]. When staged adoption models are used in OSS studies, critique of the models is limited, even where shortcomings have already been identified in their native field [START_REF] Levy | Exploring SME Internet Adoption: Towards a Contingent Model[END_REF]. This paucity of use and critique contrasts with practice in similar literature, such as the adoption of e-business systems. This field has benefited from numerous staged adoption models [START_REF] Lefebvre | Exploring B-to-B e-Commerce Adoption Trajectories in Manufacturing SMEs[END_REF][START_REF] Subba Rao | Electronic Commerce Development in Small and Medium Sized Enterprises: A Stage Model and its Implications[END_REF], with this pedigree leading to critique and the development of more complex contingent [START_REF] Levy | Exploring SME Internet Adoption: Towards a Contingent Model[END_REF] and latterly hybrid models [START_REF] Elia | Focus of B-to-B e-Commerce Initiatives and Related Benefits in Manufacturing Small and Medium-Sized Enterprises[END_REF].

2.2

A Critique of Existing OSS Classification Models Table 1 illustrates staged models used in previous OSS adoption studies, aligned to the more generic IDP [START_REF] Rogers | Diffusion of Innovations[END_REF] and Fichman and Kemerer models [START_REF] Fichman | The Assimilation of Software Process Innovations: An Organizational Learning Perspective[END_REF]. It can be seen that there is generally good agreement between these generic models, and those of Glynn et al. [START_REF] Glynn | Commercial Adoption of Open Source Software: An Empirical Study[END_REF] and Fitzgerald [START_REF] Fitzgerald | Open Source Software Adoption: Anatomy of Success and Failure[END_REF] used in previous OSS studies. This similarity is perhaps not surprising, as these models have been adapted from the work of Fichman and Kemerer.

Looking at the differences, the models of Fichman and Kemerer, Glynn et al. and Fitzgerald show that the software lifecycle used by Shaikh and Cornford [START_REF] Shaikh | Total Cost of Ownership of Open Source Software: A Report for the UK Cabinet Office Supported by Openforum Europe[END_REF] is incomplete, with early stages relating to awareness and exploration of possible solutions omitted. Likewise, it can also be seen that the IDP lacks resolution at the implementation stage, where software is commonly deployed in several phases of 'roll out' [START_REF] Silic | Technological Risks of Open Source Software Adoption in the Organizational Context -Linux in Munich (LiMux) Case[END_REF].

The models of Kwan and West [START_REF] Kwan | A Conceptual Model for Enterprise Adoption of Open Source Software[END_REF] and Miralles et al. [START_REF] Miralles | CIO Herds and User Gangs in the Adoption of Open Source Software[END_REF] appear classificatory of an organization, rather than explanatory of the adoption process. Many stages appear to be absent compared to alternatives and as such these have been discounted as the basis for the new model and are presented only for completeness.

OSS adoption can fail [START_REF] Goode | Something for Nothing: Management Rejection of Open Source Software in Australia's Top Firms[END_REF][START_REF] Huysmans | Reasons for the Non-Adoption of Openoffice.org in a Data-Intensive Public Administration[END_REF], a fact acknowledged in OSS specific models by Fitzgerald alone. None of the models used in existing OSS studies classify cessation of use, despite this being a feature of both the IDP and Fichman and Kemerer's model. This decomposition of cessation is important. Early rejection will leave little or no legacy data and an essentially unchanged business process. This contrasts with late-stage discontinuance, which will leave behind legacy data and the need to restore old systems, or seek new ones.

Table 1: Existing staged models used in OSS studies [START_REF] Glynn | Commercial Adoption of Open Source Software: An Empirical Study[END_REF][START_REF] Fitzgerald | Open Source Software Adoption: Anatomy of Success and Failure[END_REF][START_REF] Shaikh | Total Cost of Ownership of Open Source Software: A Report for the UK Cabinet Office Supported by Openforum Europe[END_REF][START_REF] Kwan | A Conceptual Model for Enterprise Adoption of Open Source Software[END_REF][START_REF] Miralles | CIO Herds and User Gangs in the Adoption of Open Source Software[END_REF] aligned to the IDP [START_REF] Rogers | Diffusion of Innovations[END_REF] and Fichman and Kemerer's [START_REF] Fichman | The Assimilation of Software Process Innovations: An Organizational Learning Perspective[END_REF]

Comments on model

Despite some categorization, the IDP and Fichman and Kemerer's model is also incomplete with regard to cessation of use. Late stage discontinuance may arise as a natural evolution of the business as the process supported by the software is retired. In other cases, use of the existing software may be discontinued in favor of a new solution or upgraded to a newer version of the existing product. No model fully acknowledges these nuances and as such omits data that may be valuable to the researcher.

Many organizations are unaware of OSS, a factor cited as hindering adoption [START_REF] Morgan | How Perceptions of Open Source Software Influence Adoption: An Exploratory Study[END_REF].

None of the models used in OSS studies have a stage acknowledging this, perhaps explained by their lineage. All have been adapted from generic technology adoption models, perhaps suggesting that such a stage was previously unnecessary due to widespread awareness of the technology in their native field. Another possible explanation is that they have been primarily used to classify existing adoption, and not map the whole adoption process per-se.

Many of the existing models imply a linear progression between stages, with Fichman and Kemerer being explicit about linearity in their work. However, this may not always be the case, something staged models from other disciplines [START_REF] Subba Rao | Electronic Commerce Development in Small and Medium Sized Enterprises: A Stage Model and its Implications[END_REF] and the IDP do note. The path taken through the adoption process, and thus any representative staged model, may have an impact on success or the barriers encountered. For example, a deployment that omits a limited trial may encounter numerous deployment issues which could have been identified and avoided without omission of this stage. However, for certain smaller deployments, it is likely that stages may be safely omitted or combined [START_REF] Lefebvre | Exploring B-to-B e-Commerce Adoption Trajectories in Manufacturing SMEs[END_REF] with a successful outcome maintained, or that experience of similar products allows valid short-cuts to be made [START_REF] Poba-Nzaou | Risk of Adopting Mission-Critical OSS Applications: An Interpretive Case Study[END_REF].

This lack of focus on the adoption process has been previously highlighted as a weakness in OSS adoption studies [START_REF] Aksulu | A Comprehensive Review and Synthesis of Open Source Research[END_REF] and is something that any new model should attempt to address.

Proposed Model

The review of literature has highlighted that existing staged adoption models are incomplete when applied to the field of OSS. The authors therefore propose a hybrid staged model [START_REF] Panayiotou | Micro Firms Internet Adoption Patterns: The Case of the Greek Jewelry Industry[END_REF] for classifying OSS adoption (figure 2) built upon the foundations of the IDP and Fichman and Kemerer's models to address the concerns highlighted in the previous section. While many of the stages of Fichman and Kemerer's model are adopted here, the commitment stage is not as the authors believe this represents a transition pathway, not a stage.

The model has a general element of linearity, proceeding from the top, to the bottom, as indicated by the green arrows. A contingent approach was considered and rejected; the authors being unable to identify evidence to suggest OSS adoption is anything other than progressive. However, it is conceivable that some stages may be skipped [START_REF] Poba-Nzaou | Risk of Adopting Mission-Critical OSS Applications: An Interpretive Case Study[END_REF][START_REF] Lefebvre | Exploring B-to-B e-Commerce Adoption Trajectories in Manufacturing SMEs[END_REF], with orange and red arrows showing pathways that allow stages to be omitted. The color coding indicates the expected increase in risk of failure based upon the deployment following that pathway, with green least risk, orange intermediate and red the highest.

Unlike the OSS specific model of Miralles et al. [START_REF] Miralles | CIO Herds and User Gangs in the Adoption of Open Source Software[END_REF] and how Fichman and Kemerer used their model, the individual OSS package, not the organization, is used as the unit of analysis. This means an organization may occupy many stages at any one time if multiple deployments are proceeding in parallel [START_REF] Saini | Open Source Adoption Index: Quantifying FOSS Adoption by an Organisation[END_REF].

Explanation of Stages

Unaware -this new stage indicates that the potential adopter is unaware that OSS is an option. This allows the model to fulfill the criteria defined by Morgan and Finnegan [START_REF] Morgan | How Perceptions of Open Source Software Influence Adoption: An Exploratory Study[END_REF]. The only progression pathway is into the awareness stage as a potential user must become aware of OSS before they can progress with adoption.

Awareness -an adopter proceeds into this stage when they become aware that OSS as a potential solution to their real or perceived need. This is analogous to the knowledge stage of the IDP and the awareness stage of Fichman and Kemerer.

There is only one progression pathway from this stage, where a potential solution can be explored to progress to the knowledge stage. Knowledge -at this stage a potential adopter has identified one or more potential solutions and have sufficient knowledge to form a favorable or unfavorable opinion. This knowledge is unlikely to be complete and may be based upon their own perceptions (which may be biased) [START_REF] Wilkinson | Vendor Rating and the Evaluation of Risk in New Software: A View from Outside[END_REF], what their competitors may be using [START_REF] Miralles | CIO Herds and User Gangs in the Adoption of Open Source Software[END_REF], opinions of existing users within the organization [START_REF] Ven | The Organizational Adoption of Open Source Server Software: A Quantitative Study[END_REF], or those promoting the solution [START_REF] Howcroft | IT Consultants, Salesmanship and the Challenges of Packaged Software Selection in SMEs[END_REF]. This matches the persuasion stage in the IDP and evaluation/trial stage of Fichman and Kemerer.

There are four progression pathways from this stage. The two paths of least risk are a positive outcome which leads to the evaluation stage and rejection based upon a perceived inappropriateness. The two remaining pathways, in order of increasing risk, are a jump to limited deployment without evaluation and an even riskier jump path to general deployment. Both of these are likely to lead to a greater chance of failure due to the software being a potentially inappropriate fit to the business needs or incompatible with existing technologies.

Evaluation -this stage involves some form of trial and/or evaluation. The ease of trialling an innovation has been found to promote a positive adoption outcome, both generally [START_REF] Rogers | Diffusion of Innovations[END_REF], and in OSS specific studies [START_REF] Morgan | How Perceptions of Open Source Software Influence Adoption: An Exploratory Study[END_REF][START_REF] West | Scope and Timing of Deployment: Moderators of Organizational Adoption of the Linux Server Platform[END_REF]. The trial can range from installation on a single machine for formal or informal testing, to a more structured evaluation process. This is analogous to the IDP's decision stage and the evaluation/trial stage of Fichman and Kemerer. Even though the software may be being used to some extent by a member of the organization, this may not be an objective evaluation [START_REF] Howcroft | The Social Shaping of Packaged Software Selection[END_REF] and this use may only be fleeting.

Due to the limited investment of resource needed to achieve this stage rejection is still a relatively low risk option. Transition pathways from this stage include pathways rejection, or progression to limited deployment if the evaluation proves successful. A jump path to general deployment is available, but this increases the risk of failure as many lessons about avoiding deployment issues can learned via a limited deployment.

Limited Deployment -here a solution has been selected to carry out a business process in a limited way (e.g. the 'germ cell' deployment for the City of Munich [START_REF] Silic | Technological Risks of Open Source Software Adoption in the Organizational Context -Linux in Munich (LiMux) Case[END_REF]) This may be limited to one department, or a limited use throughout the organization. This is an ideal opportunity to identify deployment barriers and usage issues, while the limited scale renders them easier to mitigate or correct. This stage is identical to Fichman and Kemerer's stage of the same name and a sub-division of the IDP's deployment stage.

Transition pathways include a low risk progression path to general deployment if the limited deployment proves successful. At this stage, there is still limited commitment to the software, so exit pathways are still a relatively low risk option. Direct rejection is not a possibility though, as there will be a small amount of legacy data that needs to be dealt with and business processes will require amendment if use is ceased. The exit pathways are therefore retirement, upgrade or migration (which are discussed in detail later).

General Deployment -in this stage the solution is now deployed such that it impacts large numbers of users or widely impacts upon business critical processes. Despite being widespread, the deployment is still immature with problems being encountered and resolved, but these will relate to the larger scale of implementation. This matches Fichman and Kemerer's stage of the same name, and is a subdivision of the IDP's deployment stage.

Successful continued use will allow progression to mature deployment once all deployment issues have ceased. There will be an increasing amount of business data stored in the system, so exiting use is increasingly challenging, but still possible. The exit pathways are therefore retirement, upgrade and migration.

Mature deployment -in this new stage, the solution has been in general deployment for some time and a great deal of resource has been invested over a lengthy period. It is considered 'the' way the business process is carried out by staff. Few issues will be discovered at this stage, as the organization has a legacy of support and deployment for the solution. There is likely to be a large amount of data stored in the system making direct abandonment at this stage too costly to consider. The only pathways from this stage are retirement, upgrade and migration.

Exit Pathways

As already discussed, models previously used in OSS studies do not classify the ceasing of software use [START_REF] Shaikh | Total Cost of Ownership of Open Source Software: A Report for the UK Cabinet Office Supported by Openforum Europe[END_REF]. In their model, Fichman and Kemerer offer some classification, but as categories [START_REF] Fichman | The Assimilation of Software Process Innovations: An Organizational Learning Perspective[END_REF] (it is not fully clear how these differ from a stage in their model). Here it is proposed that cessation is not thought of as stage, but a pathway to another stage. This model proposes the following exit pathways:

Rejection (blue arrow) -where a yet to be implemented OSS solution is no longer deemed suitable during the evaluation or awareness stage it can be rejected. This is analogous to the descriptions used in the IDP and by Fichman and Kemerer. There is little cost to this pathway as it will leave no residual business data and there is no culture of acceptance within the organization to resist removal. The pathway will result in an exit to the awareness stage, as a new solution will need to be sought.

Discontinuing use after deployment is more complex and, as such, there are three pathways discussed in order of increasing risk of failure:

Retirement (black arrow) -software will be retired when the business process it supports is no longer utilized. The data from this software is therefore likewise redundant and may be archived in some form for record keeping purposes. No replacement software is needed, so the process enters the awareness stage as the organization is aware that OSS is a potential solution should a similar need arise in the future.

Upgrade (orange arrow) -if the business process is still relevant, the software may be upgraded to a newer, more functional version. This may be a trivial upgrade or a major new version, but in any event, the new version and the process needed to deploy it will need evaluating prior to widespread use. For this reason, this exit pathway terminates in the evaluation stage.

While there is some risk that the upgrade will fail, this is less likely than if an entirely new solution is implemented. Unless the upgrade involves a significant change in functionality or user interface, it is likely that this will cause minimal disruption to the organization and that there will be little chance of rejection.

Transition (red arrow) -a decision is taken to adopt a completely new solution. This is the highest risk option in terms of potential failure as the new solution may be incompatible with existing systems, practices and data. This pathway terminates at the awareness stage as the organization will be seeking a replacement but must already be aware of OSS to be on this pathway. Careful analysis during the awareness and evaluation phases will be needed to ensure it is well fitted to business processes and any legacy data. Data may need to be migrated to the new system; a time consuming, expensive and sometimes inexact process. Users are likely to require retraining and may oppose the transition, potentially leading to rejection or further transition.

Conclusion and Contribution

Hauge et al. [START_REF] Hauge | Adoption of Open Source Software in Software-Intensive Organizations -a Systematic Literature Review[END_REF] exhorted OSS researchers to focus upon topics relevant to organizations, with Aksulu and Wade [START_REF] Aksulu | A Comprehensive Review and Synthesis of Open Source Research[END_REF] specifically highlighting the lack of understanding of the organizational adoption process. This paper has attempted tackle this gap by discussing the importance of a staged adoption process and has cited evidence to suggest models of such are poorly utilized in existing OSS adoption studies (objective 1). Without classification and awareness of adoption stages, adoption studies risk conflating dissimilar situations leading to inconsistent conclusions and limited generalizability.

While there have been several staged adoption models used is OSS studies, none have fully addressed all needs identified by previous literature and there has been little critique compared to related fields. This appears to be the first paper to evaluate existing staged adoption models used in OSS studies and propose a new OSS specific model to address apparent issues. Issues identified include: implications of linearity, missing stages, insufficient resolution at the implementation stage, a lack of detail regarding ceasing software use and a lack of focus on the adoption process (objective 2). A new model is proposed that builds on the theoretical foundation of Rogers' IDP [START_REF] Rogers | Diffusion of Innovations[END_REF] and the model of Fichman and Kemerer [START_REF] Fichman | The Assimilation of Software Process Innovations: An Organizational Learning Perspective[END_REF] to address these issues.

The proposed model appears to be the first to utilize a stage where the organization is unaware of OSS and to fully decompose discontinued use into four distinct pathways (rejection, upgrade, transition and retirement). In addition, three stages are used to indicate deployment based upon the maturity and spread of the software, with the aim of increasing classificatory resolution of deployment scale and the degree of acceptance. This contrasts with the one stage used by the IDP and two offered by Fichman and Kemerer. The model makes use of defined transition pathways, classified according to risk. These have the additional benefit of allowing the model to be used to track the adoption process as well as classify its current state. Both linear and non-linear paths can be followed through the model and allow adoption to be tracked on a pathway basis to allow successful and failed adoption to be potentially linked to omitted or truncated stages (objectives 3 and 4).

Future Work

The proposed model is a theoretical construct based upon issues identified from the literature. Future work will involve:

• Testing the model to validate stage descriptions and transition pathways. Data will be gathered from field studies to verify the presence of each stage and confirm entry and exit pathways, and criteria.

• Applying the proposed model to classify data from existing studies to potentially resolve inconsistencies related to heterogeneous sample composition. This may allow the resolution of issues where apparently similar studies have led to inconsistent results.

• An analysis of adoption paths through the model for different categories of software (e.g. infrastructure, end user software etc.) in a variety of environments (e.g. differing organizational size, sector etc.) to explore success strategies and pathways that commonly lead to failure.

Fig. 2 :

 2 Fig. 2: proposed OSS staged adoption model

Adoption Stage Glynn et al.'s Adoption stage Miralles et al.'s Adoption Groups Kwan and West's Adoption Stage Fitzgerald's Assimilation Stage Shaikh and Cornford's Lifecycle stage Explanation of stage

  models

	Expanded	Fichman and
	Rogers' IDP	Kemerer's
	Knowledge	Awareness	Awareness	Non-adopters			Organization is aware of
						Awareness/Interest	innovation
	Persuasion	Interest	Interest	Willing			Select	An attitude to innovation is formed by increased knowledge
	Decision	Evaluation/ Trial Evaluation/		Laboratory	Evaluation/Trial	Acquire	Engaging in activities that lead to
			Trial				selection
	Implementation Commitment Limited Deployment	Commitment Limited Deployment	Specialized		Limited Deployment	Implement	Innovation put into use
		General	General	High Users	Strategic	General	Use
		Deployment	Deployment		Mission Critical Support	Deployment:
	Confirmation						Decision regarding innovation
							reviewed
	Discontinuance Discontinuance				Abandonment	Retire	Use of a previously adopted innovation ceases
	Rejection	Rejection				Abandonment	An innovation is discounted as
							potentially adoptable
		Base from which many others are	No exit stages Classifies only usage type, not adoption.	Classifies only usage type, not adoption.	Limited early stage resolution	Limited early stage resolution
		developed.				No exit classification	No exit classification