Maha Shaikh
email: m.i.shaikh@lse.ac.uk

Tony Cornford
email: t.cornford@lse.ac.uk

Framing the Conundrum of Total Cost of Ownership of Open Source Software

Keywords: open source software, total cost of ownership, benefits, exit costs, software adoption

This paper reflects the results of phase I of our study on the total cost of ownership (TCO) of open source software adoption. Not only have we found TCO to be an intriguing issue but it is contentious, baffling and each company approaches it in a distinctive manner (and sometimes not at all). In effect it is a conundrum that needs unpacking before it can be explained and understood. Our paper discusses the components of TCO as total cost of ownership and total cost of acquisition (and besides). Using this broad dichotomy and its various components we then analyze our data to make sense of procurement decisions in relation to open source software in the public sector and private companies.

Introduction

Total cost of ownership (TCO) is considered to be a fundamental issue when making software procurement decisions [START_REF] Ellram | A Taxonomy of Total Cost of Ownership Model[END_REF][START_REF] Ellram | Total cost of ownership an analysis approach for purchasing[END_REF][START_REF] Ellram | Total Cost of Ownership: A Key Concept in Strategic Cost Management Decisions[END_REF][START_REF] Hurkens | Total Cost of Ownership in the Services Sector: A Case Study[END_REF] in organizations yet this is an area that has received limited attention. In this paper we are concerned with TCO but more specifically in relation to open source software (OSS) adoption decisions by organizations1.

This adds yet another layer of complexity because the assessment of open source software procurement is not exactly the same as that for proprietary software [START_REF] Maccormack | Evaluating Total Cost of Ownership for Software Platforms: Comparing Apples, Oranges, and Cucumbers[END_REF]. Indeed, we find that by unpacking the idea of open source TCO we become more aware of the taken for granted in proprietary software procurement decisions. TCO has been defined as an understanding of "the "true cost" of doing business with a particular supplier for a good or service" [START_REF] Ellram | A Framework for Total Cost of Ownership[END_REF]. The idea of a "true cost" and the ability to be able to assess it accurately, however is something most academics and practitioners would agree is not straightforward [START_REF] Wouters | The Adoption of Total Cost of Ownership for Sourcing Decisions -A Structural Equations Analysis[END_REF]. Thus we prefer a definition of TCO offered by Lerner and Schankerman [START_REF] Lerner | The Comingled Code: Open Source and Economic Development[END_REF] which distinguishes between different costs, and TCO is understood as the total cost of providing a functionality using one program. The proper accounting of cost should include total costs of procurement, management and support, associated hardware costs, and when one is thinking of changing software solutions, migration costs" (p107). In this definition we have a range of costs mentioned, all of which need attention before any true grasp of TCO of software can be reached. Some authors claim that there is no such thing as "the cost of software" [START_REF] Lerner | The Comingled Code: Open Source and Economic Development[END_REF] implying that cost is a more multi-faceted issue which needs to be understood better. Thus, like all evaluation decisions TCO has a quantitative and qualitative element. In section 3 below we detail the various elements of each type of cost that we have noted from literature and from our data. We found this to be a recognized issue amongst practitioners in the public sector and the private. Most decisions taken on procurement in either sector are understandably based on cost but this is not the only factor and with open source software, we found that this is not even the most relevant. The larger concern for companies eager to adopt open source software was reduced vendor lockin, and what companies understood as "value for money" (see Fig 1).

Lerner and Schankerman [START_REF] Lerner | The Comingled Code: Open Source and Economic Development[END_REF], with their distinctions in costs, indicate the conundrum of TCO. Companies are becoming more aware of these issues but the smaller ones do not have the resources to actually carry out a full detailed TCO study. How does a company assess the softer costs [START_REF] Carr | Measuring the Cost of Ownership[END_REF] surrounding software procurement especially when the software is open source (a relatively less familiar category for many companies)? It is important to make sense of the categories of cost and exactly what they entail to better manage them and make better informed decisions.

Conundrum of TCO of OSS

Literature in the area of TCO provides some useful models of cost evaluation [START_REF] Russo | A Cost Model of Open Source Software Adoption[END_REF].

What interests us in this study is not only the quantifiable costs but the more amorphous expense that occurs at the start, operation and even migration away from the software adoption. Business analysts [START_REF] Sutor | Managing open source adoption in your IT organization[END_REF][START_REF] Galoppini | Open Source TCO: Total Cost of Ownership and the Fermat"s Theorem[END_REF][13] in this area have made note of the many possibilities that open source offers companies [START_REF] Wheeler | Why Open Source Software / Free Software (OSS/FS, FLOSS, or FOSS)? Look at the Numbers![END_REF] but again these are hard to quantify, and often for smaller companies this is a consideration for the future. Most small companies that adopt open source software do so with the intention of cutting back on license costs (as the license cost of OSS is zero or close to it) [START_REF] Russo | A Cost Model of Open Source Software Adoption[END_REF][START_REF] Phipps | Open Source Procurement: Subscriptions[END_REF]. However, the total cost of ownership is not the same as the total cost of acquisition, or the cost of operation, and this is the space we want to explore with our study. This categorization is necessary as this has temporal consequences for adoption of software. The total cost of acquisition usually refers to the costs that are needed to bring the software to the actual point of use, so theoretically it includes the cost of software (buying it). The total cost of operation is more nuanced a cost but very relevant in the open source domain where companies need to adjust their TCO models to take into account the different areas of expense that OSS involves. Operational costs refer to the policies in place in companies that encourage and aid open source adoption, and very importantly, clear and structured TCO assessments before making procurement decisions.

In this paper we explore the various forms of costs that are necessary for companies and the public sector to explore in detail before making any procurement decisions, but especially those related to open source as this is a relatively new concept for them and different in that the license cost is very low yet other costs seem to mount (but are often ignored and not understood).

Methodology

This study is structured to take place in two phases. This paper reflects some of the results from phase I which includes a data collection pro-forma of twenty-five questions. These questions cover the basics of the company size, name and focus, but then go on to ask some very detailed questions about the various applications that are open source, why they were chosen, if they replaced proprietary software, was any difference in cost experienced, and finally, what prompted this change or need to adopt open source software. The responses to the pro-forma, especially to the last question will be fed into an interview guide. This then takes us to phase II where we aim to conduct between 35-40 in-depth interviews. Access in a number of organizations that responded to the pro-forma has already been negotiated and phase II will take place over the summer of 2011.

Phase I -Pro-forma

Phase 1 involved the creation of the pro-forma which was based on literature and documentation that helped to understand TCO models used in companies. The proforma was used as an early and simple data collection device rather than a fully structured and detailed survey. Our study is more qualitative in approach so the pro-forma was meant simply to gather material to help us set-up interviews, gain access and get some early comments to amend our interview guide. The pro-forma does not ask for facts, and figures on TCO for each organization, instead we ask the respondents to reply in relation to a Likert scale of 1-5.

The pro-forma was set up for access in two ways, document form (available in odt, pdf and doc formats) and an online version set up in SurveyMonkey. The aim of this study is to make sense of adoption of open source software by both private companies and public sector organizations. Though funded by the UK Cabinet Office to assess and evaluate the costs and issues involved in open source adoption by government agencies we decided that a more sound methodology would involve a balanced mix of commercial and public sector organizations. Public sector organizations are not profit orientated yet there is much to learn from private companies and their manner of dealing with open source. The larger idea here is the level of experience and comfort that private companies bring to open source adoption which is sorely lacking in the public sector. There are some exemplary cases of open source adoption by the public sector like the Extremadura case in Spain [START_REF] Zuliani | Migrating public administrations to open source software[END_REF][START_REF] Zuliani | An Experience of Transition to Open Source Software in Local Authorities[END_REF] but there are far more "success" stories of open source adoption by commercial companies [START_REF] Dinkelacker | Progressive Open Source[END_REF][START_REF] Dahlander | Penguin in a newsuit: a tale of how de novo entrants emerged to harness free and open source software communities[END_REF][START_REF] Fitzgerald | The Transformation of Open Source Software[END_REF][START_REF] O'mahony | IBM and Eclipse (A)[END_REF]. The factors that encourage private companies to adopt open source software, especially considering most business models of such adoption indicate that the software itself does not lead to value creation or capture directly [START_REF] West | Challenges of Open Innovation: The Paradox of Firm Investment in Open Source Software[END_REF][START_REF] West | How Open is Open Enough? Melding Proprietary and Open Source Platform Strategies[END_REF][START_REF] Osterwalder | Clarifying Business Models: Origins, Present, and Future of the Concept[END_REF][START_REF] Vargo | Evolving to a New Dominant Logic for Marketing[END_REF], make some of the lessons translatable across both sectors.

The pro-forma was put online for a period of two months (and is still online but for the sake of this paper we only took into account the pro-formas completed in the first two months) and we received twenty-four responses. We also received seven paper based pro-formas sent back to us as scanned documents via email. This made a total of thirty-one pro-formas. We had set the pro-forma to ensure that details of the respondent was a required category. This was done to be able to filter out any responses that were biased, duplicate or simply not completed with any seriousness. Of the 24 online pro-formas two were filled in by people calling themselves "test" and "anonymous". We discounted the results from both these pro-formas. We also had two incomplete pro-formas online. Incomplete pro-formas were those where some questions were skipped. As this exercise was carried out as a precursor and data gathering exercise more for the interest of creating a strong and clear interview guide for phase II we accepted the results of the incomplete pro-formas. Phase II is where the researchers involved in this study hope to gain a more detailed understanding of TCO models and the decision-making process in organizations so it was felt that so long as the pro-formas were recognized to be valid (not anonymous or biased) and useful (filled in 75% of the pro-forma and added some non-mandatory comments that helped us to evaluate the experience of the company with open source) we would include the results to help shape the interview guide for phase II.

The pro-forma has four sections. The aim of section one is to ask for simple information like the name and affiliation of the respondent. This includes the size and name of the company/local authority. Section two prompts for the sort of OSS used by the organization and the time span of use. The aim of section three is to gather details on strategic drivers that lead to OSS adoption, and section 4 is concerned with eliciting the TCO models used.

Phase I Leading to Phase II

Of the total pro-formas we received the majority of them were filled in by small to medium sized private companies (44%). Small to medium sized enterprises included all those with a number of employees ranging between 1-100. We had 24% of the pro-formas completed by employees of large, and in many cases global companies (employees ranging from 101 and above). Public sector replies made 32% of the total.

In phase II we intend to cover a larger portion of the public sector. The pro-forma had a number of questions where respondents were asked to add comments or spell out the category of "other" in more detail. Responses to such questions gave rise to some very interesting issues which will become a part of the interview guide and informed the researchers involved. The respondents for the pro-forma were asked for their contact details and phase II will draw us back to the those that made very intriguing comments. Phase II will involve in-depth interviews focused on 5-7 case studies. Key personnel involved in making procurement decisions and strategy of open source use in the organization will be interviewed.

The cases will be chosen on the basis of whether there has been involvement with open source adoption, use and/or redistribution for at least a period of two years. This is to ensure that there has been time enough for reflection on the process and there are some indicators that show "success" or "failure"more simply, is open source still being used, has the use of it increased over time, and has it spread up the stack. We also want an even mix of public and private organizations to make it possible to reflect across the cases and build on lessons learned for an exchange of ideas. The or-ganizations and their experience with open source needs to have been fruitful at some level but we are equally aware that we need to have examples of less successful implementations as this will enrich our work and understanding of the concerns with open source that can be faced. Indicators for the less successful cases include a return to proprietary software use, move to outsourcing their software development and a shrinkage in their in-house IT department. And lastly, our cases will include local authorities in the UK and other European government as well such as the Municipality of Munich, Municipality of Andalucía.

Analysis and Discussion

The pro-forma results are very interesting and we only have space to share some of the key ideas that emerged. These ideas include the importance of liberty [START_REF] Phipps | Open Source Procurement: Subscriptions[END_REF] and flexibility (reduced vendor lock-in) provided by open source to companies and the public sector, that long term costs vary far more across companies considering their size and experience with open source, short term costs are slightly higher, that most companies choose a combination of open source and proprietary software where their decisions are based on pragmatism and need rather than questions of openness.

In this section we take these broad themes and frame them in relation to another interesting dimension that we noted from our data, that of a more fine-tuned TCO categorization than has been offered so far by other studies. We found that the cost categories were not limited to two broad ones, cost of acquisition and cost of operation, but instead we recognized two other very key cost factors that companies are beginning to take very seriously in relation to open source software procurement decisions cost of adoption and exit costs (see Table 1).

The cost of operation and software are more quantifiable and thus easier to measure and evaluate. Cost of operation includes the expense of conducting a TCO study before making a decision of implementation. This is similar for proprietary software so in some respects this is not specific to open source, but rather to software decisions in general. The cost of software includes the cost of the license, setting up costs, and other costs which are similar to the cost of operation in their lending themselves to be quantified. However, the cost of adoption and exit costs for open source pose some interesting challenges but also opportunities.

Cost of Adoption

The cost of adoption, we found, concerns all the relevant but more qualitative expenses involved with the broad idea of adoption such as the learning necessary when you adopt open source for the first time or for a new part of the stack (see Table 2). Very importantly it also includes interoperability costs which many companies surprisingly ignore even though this is a feature of proprietary software as well.

Exit Costs

Exit costs are yet a more intriguing idea. Respondents agreed that this aspect was the most ignored and yet it formed a very positive aspect of the overall TCO of open source adoption. Table 2 outlines the areas where the costs with open source were considered to be the lowest but take note of the "other" category in Table 2 and Figure 2. Upon reading the comments added by the respondents it was evident that though open source saved the organization money there were costs that had been ignored and ill-understood.

Exit costs include all the expenses of switching from one software to another, various interoperability expenses, costs related to legacy systems, retraining staff and initial teething issue costs. This is an area where at least at present companies feel that open source costs are higher and not clear at all. The low license costs with open source software, according to the respondents though very real can become misleading because companies simply begin to base their decisions on that cost alone and dismiss any other factor.

However, when migrating between one open source product to another the migration costs are then lower because open source is based on open standards and there is greater interoperability. Most companies that had a more long term understanding of their software adoption added comments in the pro-forma to the effect that the migration costs (exit costs) were more favourable for open source and so this became one of the deciding factors in favour of OSS. Simply put, migration costs between one proprietary application to another are always considerable as neither products are open and it is thus difficult to manage the necessary interfacing and interoperability changes across all the software applications that need to "speak to each other".

Vendor Lock-in and Lock-out

Vendor lock-in though a real problem with proprietary software is less so with open source. This may well be a real consideration yet what we note from our study so far is that expertise of the software (be it open source or otherwise) and a lack of good documentation which is a problem with open source often becomes a lock-out. Companies feel discouraged from adopting any software they cannot control [START_REF] Shaikh | Letting Go of Control to Embrace Open Source: Implications for Company and Community[END_REF], but also cannot obtain comprehensive services and troubleshooting. The idea of reaching out to an unknown community [START_REF] Agerfalk | Outsourcing to an Unknown Workforce: Exploring Opensourcing as a Global Sourcing Strategy[END_REF] has a romantic appeal but is not practical. Indeed, such promises spread FUD about open source adoption and lead to lockout because companies avoid anything they are not familiar with. It is easy enough in theory to take code and customize it yet as many respondents noted this is not so in practice. They are forced to hire experts and look for support to communities outside the company. This is a drain on their resources and an expense that was not considered by the decision takers, even if a TCO assessment was carried out before procurement decisions were taken. This would not be such an issue if the software is vendor-supported, however, we have found that many companies and local authorities are drawn to open source because they wish to control the software source and make changes to it both in-house and with the help of a strong external community.

Temporal Element of TCO

Another key theme that arose from our pro-forma data was that of cost temporality.

Of the four costs outlined in Table 1 the cost of adoption and exit costs are relatively quite high for open source software. This is even more marked with the added complexity of the size and experience of the organization. If the company is large and experienced with open source then these two costs are often well-understood and thus less expensive. Large companies can diversify and absorb costs better than smaller companies and this is largely true for the public sector as well. Smaller organizations however usually jump on the open source bandwagon with the naive idea that this will prove cheaper. They have also not undertaken a TCO analysis and if they are not experienced with open source it was then found that the expense of open source surprised many. In some cases, especially in the public sector (coupled with issues of poor interoperability) we have seen a return to proprietary software. Phase II of our study will include this local authority as a full case study to better explore the problems, issues and dilemmas that forced it to return to closed source software adoption.

Pragmatism and the Idea of Value

Most companies, like software developers and hackers work on the basis of pragmatism. If something is good enough and not broken then it will be continued to be used. Open source requires some experience and practice and very key, the licenses involved with open source need good skills and expertise, something most small companies do not have the resources for and the public sector simply does not consider. In effect if something is good enough then change is considered problematic and unnecessary.

Conclusion

This paper maps the initial stages of our study of the total cost of ownership of open source software. The analysis provides some interesting answers to broad questions about cost but more importantly it raises relevant questions about the rather enigmatic quality of assessing the total cost of software, especially open source software. It is a conundrum as most companies lack experience with open source and more to the point, a slightly different understanding of "value for money" is needed. The actual monetary cost of open source software adoption in many cases is not unlike that of proprietary software but it is the liberties, flexibility and control that it provides that draws companies and the public sector towards it. Phase I of our study has given rise to some very interesting findings. The main ideas the respondents focused on included the lack of maturity level of open source software, license confusions and lack of knowledge about the implications of various open source licenses. Other ideas which arose were somewhat more surprising, most organizations do not even attempt a TCO study before making procurement decisions because of the expense such studies involve. The models used to assess TCO are also more suited for proprietary software and companies are not comfortable or skilled to tweak them for open source. And lastly, there is no policy in most companies for open source adoption. These decisions are made more ad-hoc and usually based on pragmatic decisions of use and need rather than cost.

However, there is much detail yet to be teased out in this amorphous area, and during the course of this study we hope to be able to show more clearly where the large part of the costs lie with open source software adoption, exactly where and how they are distinctive in comparison to proprietary software, what strategies and practices the public sector and companies can employ to make more effective use of the unique qualities of open source so that the software can yield a stronger feeling of "value for money". Indeed, can we go so far as to suggest that open source software adoption is an idea that the public sector not only needs to explore more seriously but in fact it will prove more effective, valuable, cheaper and necessary in the future? We aim to be able to provide a more nuanced answer to this and other questions at the end of our study, and encourage other researchers to explore such aspects because we feel that public sector adoption of open source has the potential to have real influence on open government strategies [START_REF] Noveck | Defining Open Government[END_REF], open standards, and open data.

Fig. 1 .

 1 Fig. 1. This figure shows the various factors companies in our pro-forma considered when making a decision to move to open source, and the relative weight of each factor.

Fig. 2 .

 2 Fig. 2. This figure shows the various open source software concentration in implementation in the different organizations.

Table 1 .

 1 The difference with open source is that some respondents stated that they feel they can make the necessary adjustments because the code is open, yet as we are becoming more aware, there is a steep learning curve with all software not created in-house. Upgrades are a growing concern for companies with open source because most open source software tends to adapt and be changed more frequently than proprietary. Of course the choice to upgrade is with the user yet interoperability can also become a problem if one software is upgraded but other applications and infrastructure are not. Different Categories of TCO, and what each involves

		Categories of TCO
	Cost of operation	Cost of software Cost of adoption	Exit costs
	Formal TCO assessment Initial purchase price	Learning	Migration costs
	TCO policy	Monetary costs of set-up	Interoperability	Re-training
	Cost of evaluating software (tinkering)	Customization expense	Support services	Switching costs
		Software scaling cost	Training
			Access to upgrades

Table 2 .

 2 Areas and applications of OSS which Saved Money for Organizations

		Which OSS saved your organization money?
		Agree	Agree some-what	Completely disagree	No change in expense -same
	Applications:	75.0%	12.5%	0.0%	6.3%
	Enterprise systems	71.4%	0.0%	0.0%	0.0%
	Vertical/line of business	58.3%	0.0%	0.0%	0.0%
	Desktop	84.6%	7.7%	0.0%	0.0%
	System's De-velopment	71.4%	7.1%	0.0%	7.1%
	Infrastructure:	84.6%	7.7%	0.0%	0.0%
	OS Platforms	88.2%	11.8%	0.0%	0.0%
	Application Servers	84.6%	0.0%	0.0%	7.7%
	Web services	93.3%	0.0%	0.0%	0.0%
	Networking	86.7%	6.7%	0.0%	0.0%
	Database	86.7%	0.0%	0.0%	13.3%
	Other	72.6%	0.0%	0.0%	0.0%

  In the public sector in the UK we have found that local authorities are beginning to gravitate towards open source simply because of the lower costs promised by open source (due to the recession), however a better understanding of the benefits of open source is needed. The idea of where the value lies in open source software adoption is the needed. And by value we do not mean a monetary value (though it does include this category) we refer to the softer side of benefitsthat of flexibility, openness, freedoms, ability to tweak and customize and along with open standards and open dataa far more open and accessible software environment. This is where the true benefits and cost reductions will come in the future.The real value in open source adoption is clearly the collaborative co-creative ideology and spirit it encourages. This in turn leads to value creation, innovation, and a stronger ecosystem[START_REF] Shaikh | Understanding Commercial Open Source as Product and Service Innovation[END_REF].

Our research is funded by the UK Cabinet Office and the OpenForum Europe. The aim of this research is to assess the various costs and value of open source adoption by the public sector and private organizations.


Landon J Pratt
email: landonjpratt@byu.edu

Alexander C Maclean
email: amaclean@byu.edu

Charles D Knutson
email: knutson@cs.byu.edu

Eric K Ringger
email: ringger@cs.byu.edu

Cliff Walls: An Analysis of Monolithic Commits Using Latent Dirichlet Allocation

Artifact-based research provides a mechanism whereby researchers may study the creation of software yet avoid many of the difficulties of direct observation and experimentation. However, there are still many challenges that can affect the quality of artifact-based studies, especially those studies examining software evolution. Large commits, which we refer to as "Cliff Walls," are one significant threat to studies of software evolution because they do not appear to represent incremental development. We used Latent Dirichlet Allocation to extract topics from over 2 million commit log messages, taken from 10,000 SourceForge projects. The topics generated through this method were then analyzed to determine the causes of over 9,000 of the largest commits. We found that branch merges, code imports, and auto-generated documentation were significant causes of large commits. We also found that corrective maintenance tasks, such as bug fixes, did not play a significant role in the creation of large commits.

Introduction

Artifact-based software engineering research may in some respects be compared to archaeology, a field that has been defined as "the study of the human past, through the material traces of it that have survived" [START_REF] Bahn | Archaeology: a very short introduction[END_REF]. Much like archaeologists, empirical software engineering researchers often seek to understand people. The software engineering researcher, while not isolated from a target population by eons, faces other obstacles that often make direct observation impossible. Many organizations are loath to allow researchers through their gates, in an effort to protect trade secrets or merely to hide shortcomings. Even in cases where researchers are allowed to directly observe engineers building software, the Hawthorne effect threatens the validity of such observations. Time investment and organizational complexity are also issues that pose problems in software engineering research. Direct observation requires significant time investment, making it impossible for a single researcher to observe everything that takes place within a given software organization.

As a result of these barriers, software researchers, like their archaeologist counterparts, take advantage of artifacts-work products left behind in software project burial grounds. Artifacts are collected after the fact, minimizing the confounding influence of the presence of a researcher. Artifacts also help researchers deal with the requirements of studying complex organizations. By leveraging artifacts of the software process, researchers are able to study thousands of pieces of software in a relatively short period of time, an otherwise impossible task.

The open source movement is particularly important to software engineering research, since project artifacts, such as source code, revision control histories, and message boards, are openly available to software archaeologists. This makes open source software an ideal target for researchers with a desire to understand how software is built.

Threats to Artifact-based Research

Unfortunately, the study of artifacts in software engineering is not all sunshine and double rainbows; serious challenges threaten the results of artifact-based research involving open source software projects (such as those hosted on Source-Forge). Since artifact data is examined separate from its original development context, identifying the development artifacts actually recorded in the data can be difficult. It is challenging enough to ensure that measurements taken for a specific purpose actually measure what they claim to measure [START_REF] Briand | Defining and validating measures for object-based high-level design[END_REF]. It is all the more difficult (and necessary), therefore, to validate artifact data, which is generally collected without a targeted purpose.

Many phenomena can easily be overlooked by software archaeologists, such as auto-generated code, the presence of non-source code files in version control, and the side effects of branching and merging. Understanding the limitations of artifact data represents an important step toward validating the results of numerous studies (for example, [START_REF] Bird | Open borders? immigration in open source projects[END_REF][START_REF] Daniel | Do programming languages affect productivity? a case study using data from open source projects[END_REF][START_REF] Hassan | Predicting faults using the complexity of code changes[END_REF][START_REF] Krein | Language entropy: A metric for characterization of author programming language distribution[END_REF][START_REF] Mockus | Two case studies of open source software development: Apache and mozilla[END_REF][START_REF] Tarvo | Mining software history to improve software maintenance quality: A case study[END_REF][START_REF] Xu | A topological analysis of the open souce software development community[END_REF]). Despite on-going efforts to identify and mitigate the limitations of artifact-based research, new threats are constantly emerging.

The focus of this paper is one such threat-the presence of monolithic commits in open source repositories. A close look at the version control history of many projects in SourceForge reveals some worrisome anomalies. Massive commits, which we refer to as "Cliff Walls," appear with alarming frequency. One investigation of Cliff Walls in SourceForge found that out of almost 10,000 projects studied, half contained a single commit that represented 30% or more of the entire project size [START_REF] Maclean | Trends that affect temporal analysis using sourceforge data[END_REF]. These Cliff Walls indicate periods of unexplained acceleration in development, threatening some common assumptions of artifactbased research, especially for studies of project evolution. Cliff Walls thwart attempts to tie authors to contributions, mask true development history, and taint studies of project evolution. We must better understand Cliff Walls if we are to paint an accurate picture of software evolution through the use of artifacts.

Cliff Walls

In [START_REF] Maclean | Trends that affect temporal analysis using sourceforge data[END_REF] we introduced the concept of "Cliff Walls" as unexpected increases in version control activity over short periods of time (see Fig. 1). In the most extreme cases, Cliff Walls represent millions of lines of source code contributed to the repository in less than a day. Such activity is problematic for researchers, especially those investigating the evolution of software, because such sudden surges of source code commits cannot possibly be the result of common incremental development activities. Even a team of "supercoders" would be hard-pressed to produce such massive quantities of code in a short period of time, all the more impossible for the single "author" to which a version control commit is attributed. We are forced to ask what these "Cliff Walls" truly represent.

Definitions

When examining contributions to version control systems, it is logical to discuss them in the context of "commits." However, some researchers have defined "commit size" in terms of the number of files changed in a commit [START_REF] Hindle | What do large commits tell us?: a taxonomical study of large commits[END_REF][START_REF] Hindle | Automatic classication of large changes into maintenance categories[END_REF][START_REF] Hattori | On the nature of commits[END_REF], while others treat the size of a commit as a function of the number of LOC added, removed or modified [START_REF] Arafat | The Commit Size Distribution of Open Source Software[END_REF].

Within the context of this study, a commit refers to the set of files and/or changes to files submitted simultaneously to a version control system by an individual developer. Since we are primarily concerned with the growth of projects over time, Commit size is defined as the total number of lines added or removed from all source code files in a given commit, which allows us to perceive code growth (or shrinkage) within a project. For example, if an existing line in one file is modified, and no other changes are made, the size of the resulting commit would be zero LOC. In this study, we include code comments in the measurement of commit size, allowing us to detect code growth attributable to the insertion of comments within source files.

We use the term Cliff Wall to describe any single commit with size greater than 10,000 lines of code (LOC). We believe this threshold to be sufficiently high that it avoids capturing most incremental development contributions for individual authors. While it is possible that an extremely productive developer could produce 10,000 lines of code in a period of days or weeks, it is unlikely.

Additionally, the methods employed in this study should allow us to identify instances where the threshold fails.

Commit Taxonomies

The ability to distinguish between different types of cliff walls is critical for many artifact-based studies. For example, a researcher attempting to measure developer productivity on a project would likely want to be able to distinguish between large commits caused by auto-generated code and branch merges, as they must be handled differently when calculating code contributions.

In [START_REF] Hindle | What do large commits tell us?: a taxonomical study of large commits[END_REF], the authors present a taxonomy used to manually categorize large commits on a number of open source software projects. Hattori and Lanza also present a method for the automatic classification of commits of all sizes in open source software projects [START_REF] Hattori | On the nature of commits[END_REF]. The taxonomies developed in these studies focus on classifying the type of change made (for example, development vs. maintenance). Both these studies concluded that corrective maintenance activities, such as bug fixes, are rarely the topic of large commits. These studies also found that a significant portion of large commits were dedicated to "implementation activities." The authors consider all source code files added to the repository to be the result of implementation activities.

In [START_REF] Arafat | The Commit Size Distribution of Open Source Software[END_REF], the authors categorize commits from open source projects into three groups, using a rough heuristic based on commit size. Their study makes no attempt to analyze commit log messages. Rather, they divide each commit into one of three groups, based on the commit size: 1) single individual developer contributions, 2) aggregate developer contributions, and 3) component or repository refactoring or consolidations.

In our study, we seek to identify specific behaviors that play a significant role in the creation of monolithic commits. In the next section, we discuss the methods that we will apply in our search for the causes of Cliff Walls.

Latent Dirichlet Allocation

Latent Dirichlet Allocation (LDA) is an unsupervised, hierarchical Bayesian topic model for analyzing natural language document content that clusters the terms in a corpus into topics [START_REF] Blei | Latent dirichlet allocation[END_REF]. In LDA, the model represents the assumption that documents are a mixture of these topics and that each topic is a probability distribution over words. In short, LDA allows us, without any knowledge engineering, to discover the topical content of a large collection of documents in an unsupervised fashion. Given a corpus of documents such as our commit log messages, inference techniques such as Gibbs sampling or Variational Bayes can be used to find a set of topics that occur in that corpus, along with a topic assignment to each word token in the corpus.

A common technique for visualizing the topics found using LDA is to list the top n most probable terms in each topic. For example, one of the topics we found using LDA that concerns bug correction, consisted of the following terms: fix bug fixes crash sf problems small quick closes blah deleting weapon decoder lost hang weapons delphi noted led When brevity is required, it is also common to refer to a topic by just the first two or three most probable terms, sometimes in hyphenated form. Using this notation, the above topic could be referred to as the "fix-bug" topic. Both of these methods for indicating a particular topic (word list and first words) are used throughout this paper.

Because LDA also assigns a topic to each individual token within a commit message, a single commit can, and usually does, contain many topics. This is one benefit of using LDA: each "document" may belong to multiple classes. Such is not the case with many of the supervised classification methods, which typically assign each document to a single class. In the case of commit log messages, allowing for multiple topics in a single message allows us to conduct a more fined-grained analysis.

The approach taken by LDA is an attractive alternative to other classification and clustering methods. Supervised classification methods require a training set of "tagged" data, for which the appropriate class for each instance has previously been specified. In many situations this tagged data does not exist, requiring the researcher to manually assign tags to some portion of the data. This is a tedious, time-consuming, and potentially biased process which can be avoided through the use of unsupervised methods such as LDA. Unsupervised methods may also compensate for a measure of "short-sightedness" by the researcher. In supervised methods, since the classes must be predefined by the researcher, it is possible to degrade the usefulness of the model through the omission of important classes, or the inclusion of irrelevant classes. Unsupervised methods avoid some of these errors since no predefined clusters are specified, allowing the data to better "speak for itself." For our analysis, we used the open source tool "MALLET" which includes an implementation of LDA [START_REF] Kachites | MALLET: A machine learning for language toolkit[END_REF].

Methods

Our data set consists of the version control logs of almost 10,000 projects from SourceForge, acquired in late 2006. This data set has been previously used in a number of studies [START_REF] Daniel | Do programming languages affect productivity? a case study using data from open source projects[END_REF][START_REF] Daniel | Programming language trends in open source development: An evaluation using data from all production phase sourceforge projects[END_REF][START_REF] Daniel | Studying production phase sourceforge projects: A case study using cvs2mysql and sfra+[END_REF][START_REF] Krein | Language entropy: A metric for characterization of author programming language distribution[END_REF][START_REF] Krein | Impact of programming language fragmentation on developer productivity: a sourceforge empirical study[END_REF][START_REF] Maclean | Threats to validity in analysis of language fragmentation on sourceforge data[END_REF][START_REF] Maclean | Trends that affect temporal analysis using sourceforge data[END_REF]. The logs for all projects in our data set were extracted from the CVS version control system. In creating the data set, the original authors filtered projects based on development stage; only projects labeled as Production/Stable or Maintenance were included in the data set. For further description of the data, see [START_REF] Daniel | Do programming languages affect productivity? a case study using data from open source projects[END_REF]. In calculating commit size, we excluded non-source code files, based on file extension. Over 30 "languages" were represented in the final data set, including Java, C, C++, PHP, HTML, SQL, Perl and Python.

Because CVS logs do not maintain any concept of an atomic multi-file "commit" it was necessary to infer individual commits. We utilized the "Sliding Time Window" method introduced by Zimmerman and Weißgerber [START_REF] Zimmermann | Preprocessing CVS data for fine-grained analysis[END_REF]. This resulted in a set of almost 2.5 million individual commits, extracted from over 26 million file revisions. Applying our pre-defined threshold of 10,000 LOC yielded over 10,000 Cliff Walls. We also found that a number of the commits contained log messages that were uninformative. Commits with empty log messages or with "empty log message" were removed from the data to prevent degradation in the quality of topics identified. The resulting set contained 2,333,675 commits, with 9,615 Cliff Walls. We later removed other uninformative commits (see discussion in Sec. 6), ultimately resulting in the exclusion of 6.6% of commits in our data set due to log messages that conveyed no information about the development activities they represent. A disproportionate number of the commits removed were Cliff Walls (an ultimate exclusion of 14.8% of all Cliff Walls). Additionally, very common English adverbs, conjunctions, pronouns and prepositions belonging to our "stop-word" list were removed from the commit messages in order to ensure the identification of meaningful topics.

The LDA algorithm, as implemented in MALLET, requires three input parameters: the number of topics to produce in its analysis, the number of iterations, and the length of the burn-in period. In our study, we elected to identify 150 topics with MALLET. The authors Hindle and German identified 28 "types of change" for the commits classified as a part of their taxonomy [START_REF] Hindle | What do large commits tell us?: a taxonomical study of large commits[END_REF]. Hattori and Lanza, in their study of commit messages, identified 64 "keywords" that were used to classify commits [START_REF] Hattori | On the nature of commits[END_REF]. These prior results gave us reason to believe that 150 topics would be a sufficient number to capture the motivations behind the commits in our data set, with an appropriate level of detail. 1As one of the steps to understanding the Cliff Wall phenomenon, we compare the most prevalent topics found in Cliff Wall commits to those found in the entire body of commits. Instead of running LDA separately on the two subsets of our data, we run LDA once on all of the data and then filter the results to gain a picture of Cliff Walls in contrast to All Commits. This approach ensures that the topics found are consistent across both groups, which helps yield a meaningful comparison.

Analysis & Discussion

Figure 2 provides a first glance at some of the variation exhibited by Cliff Walls. In these graphs, each horizontal bar represents one of the 150 topics generated. The thickness of each bar represents the proportion of tokens in the entire corpus of commits that were assigned to that topic. Commit log messages are fairly evenly distributed over the topics for the general population of commits. However, a small number of topics are considerably more prevalent in the large commits. Tables 1 and2 list the 15 most prevalent topics for all commits and Cliff Wall commits. Similarly, the tag clouds in Figs. 3,4, and 5 begin to give us an idea of the most common topics for our two groups of interest. 2 Each tag in the cloud represents a topic that has been summarized using the "first words" method described in Sec. 3. Like a stratum from Fig. 2, each tag is sized based on the proportion of tokens belonging to that topic. Thus, the largest tag in Fig. 4, "initial-import," is also the largest stratum in the bar chart for Cliff Walls. Tag position and color do not convey any additional information. Figure 5 is an alternate view of Fig. 4, with the dominant "initial-import" topic removed to improve readability. These images provide an overall view of the topics and their proportions for the two groups of interest. We next discuss some of the most prevalent topics and their interpretations.

One of the goals of this paper is to compare the topics produced by LDA with previously hypothesized causes of Cliff Walls. We examine each of these causes to see if LDA is able to identify them as prominent features of large commits. We also examine some of the causes for Cliff Walls that were previously overlooked, but were consequently suggested by LDA. In pursuit of these goals, we have found two views of the data that provide insight into the causes of Cliff Walls: Overall Topic Proportion and Topic Relative Rank. We discuss these in the next two subsections.

Overall Topic Proportion

Tables 1 and 2 display the 15 most prevalent topics in the Cliff Wall and All Commits groups, as determined by proportion of tokens belonging to that topic. For example, the topic "version-release" in Tbl. 1 has a proportion of 1.75%, suggesting that 1.75% of all the words in all of the commit messages in our data set were assigned to this topic by the LDA algorithm. In other words, these tables list the topics most frequently discussed in commit log messages belonging to our two groups of interest. We refer back to these tables frequently throughout this section as we discuss the various causes of Cliff Walls.

Topic Relative Rank

Additional insight into Cliff Walls can be gained through the use of another simple metric. Within each of our two groups, "All Commits" and "Cliff Walls," each topic can be ranked based on its proportion within the group. The difference between a topic's ranking in the two groups is a good indicator of the prevalence of a given topic relative to other topics.

For example, in Tbl. 1 we see that "initial-import" is the 5th ranked topic. In Table 2, the same topic is ranked 1st, for a rank difference of +4. In contrast, as we see from Tbl. 3, the topic "cvs-sync" holds ranks of 101 and 4, resulting in a rank difference of +97. In essence this means that, relative to other topics, "cvs-sync" is discussed more frequently within Cliff Wall commits than it is for the general population of commits.

It is important to note that the difference between topic ranks is not synonymous with a similar difference in proportion. The difference between proportions for the "initial-import" topic is an approximately 25% increase for the Cliff Walls group. This is a very large change in proportion which results in a relatively small difference in rank. It is even possible, given the distinct distributions of our two groups (see Fig. 2) that a negative change in proportion could still result in a positive rank difference.

Code Imports

A "Code Import" occurs when a significant amount of source code is added to the repository. Code Imports differ from "Off-line Development" in that the code added was not developed as part of the project of interest, but instead originated from some other project. The most common example of a code import is probably the addition of the source code for an externally developed library. We found a good deal of evidence that Off-line Development is a significant cause of Cliff Walls. As shown in Tbl. 2, a few prominent topics (particularly 1, of files alone, however, does not indicate a Code Import. Table 1 indicates that topic 1 is quite prominent for all commits, because files are constantly being added to version control systems. In the case of Cliff Walls, however, the size of the commit gives us good reason to believe that the files added contain a great deal of code, and therefore do not represent files added as part of incremental development.

Also, a of the topics do provide additional evidence of Code Imports. Topic 9 contains the term "library" which indicates that this topic relates to the addition of library files to version control. Similarly, the topic "module-python" appears in Tbl. 3 as a topic with a much higher relative rank for Cliff Wall commits. Examination of log messages for which this topic had high proportion yielded messages such as "bundle all of jython 2.1 with marathon so all python standard library is available" and "Add the Sandia RTSCTS module to the code base." These messages are indicative of Code Imports.

Off-line development

In this paper, we use "Off-line Development" to refer to large quantities of code that were developed as part of a project, but for which we have no record. This may be code that was developed without the benefit of version control, or that was developed in a separate repository and then added to the SourceForge CVS repository in a monolithic chunk.

Much of the evidence for Off-line Development is similar to that of Code Imports. Many of the same topics that may refer to code imports ("initial-import," "cvs-sync," and "files-added") could equally be attributed to Off-line Development. Thus it is difficult to distinguish between the true source of many large commits, because it is hard to tell if the files added were developed in conjunction with the current project or separately. Further investigation is required to elucidate the differences between Cliff Walls attributable to Code Import and those due to Off-line Development.

Branching & Merging

Merging is a major factor in the creation of Cliff Walls. The 7th ranked topic for Cliff Walls is a topic dealing with the merging of a branch in the repository. This same topic also appears as one of the largest positive rank differences in Tbl. 3. This indicates that not only are merges a significant factor behind the creation of Cliff Walls but also that the merge topic is significantly more prevalent within Cliff Walls than it is for All Commits. In contrast, the "initial-import" topic is one of the highest ranked topics in both groups.

Auto-Generated Code

Topics pertaining to Auto-generated Code are a bit more difficult to identify. The topic "target-generated" appears to capture auto-generated code quite well: target generated rules rule generate mark reports make generation targets automatically linked policy libraries based generator jam dependencies building Surprisingly, this topic is of relatively little importance, with rank 67 (Cliff Walls) and 113 (All Commits). Such low ranks would seem to indicate that Autogenerated code does not play a significant role in the explanation of Cliff Walls. We did, however, find another example that suggests that Auto-generated Code may be a more significant factor. We were surprised to find that the topic "addedcomments" was the 14th ranked topic for Cliff Walls (see Tbl. 2). Non-source code files had been excluded from our study, and code commenting seemed an unlikely cause for commits on the magnitude of 10,000 lines or greater. Upon appeal to the commit log messages, we found a large number of messages containing text such as "Add generated documentation files," "Documentation generated from javadoc," and "Updated documentation using new doxygen configuration."

Further examination revealed that, at least in the cases mentioned above, these commits consist almost entirely of HTML files. The above comments contain 81, 120, and 448 HTML files, respectively. This suggests that large, comment or documentation related commits may be the result of auto-generated HTML files from documentation systems such as javadoc and doxygen.

It is possible that there may be other significant sources of Auto-generated Code expressed in the topics obtained from LDA. In the above case, the tools that generated the "code" were more effective identifiers of Auto-generated code than were the terms "automatically" and "generated." Further investigation is required to determine whether other such cases exist.

Other Findings

As we hoped, the application of LDA to this problem suggested some potential Cliff Wall causes we had not forseen. Additionally, a few interesting observations served to confirm some of our suspicions about Cliff Walls. In this section we discuss some of these findings.

One discovery of note was the importance of activities related to project releases and new versions. The 2nd most prevalent topic discussed in Cliff Wall log messages is "version-release." Topics 11 and 15 in Tbl. 2 also deal primarily with project releases and versioning, with prevalent terms such as "latest," "version," "beta," "v1" and "v2." It is difficult to tell exactly what is occurring with these commits; most provide little information other than a version number. We suspect that many of these may be the result of merges, and further investigation may determine the true cause.

We were able to gain some understanding of topics which were infrequently discussed in the log messages of Cliff Wall commits. Table 4 shows some of the topics for which the topic rank dropped significantly for Cliff Wall commits. This drop would indicate topics that were discussed much less frequently for Cliff Walls than All Commits, when compared to all other topics. Some of the trends in the table include topics discussing corrective maintenance ("fixed-bug," "fix-typo," 'fixes-minor," "output-debug"), gui tweaks ("button-selection"), and minor implementation details ("string-return," "variable-global," "size-buffer"). It is not surprising that these topics do not significantly occur in the log messages of large commits, but these trends lend credibility to our results.

Table 3 provides another interesting insight. We observe that two topics appear to deal with web technologies: "http-www" and "php-index." In many cases, we found that these topics indicated the presence of a URL in the log message. It is intriguing that this topic surpassed so many others in the Cliff Walls category. We believe that these URLs could convey valuable information about the commit, and may help to identify library code that is being imported, or the location of an external version control repository utilized by the project.

Threats

Some of the most significant threats to our results arise from the data set employed. As previously stated, the data was gathered in late 2006, and is now relatively old. It is possible that the results that we have found do not correspond to the current state of projects in SourceForge. This study is also limited to projects using the CVS version control system. According to our estimates, almost all new projects in SourceForge are now using Subversion instead of CVS. While the two technologies are similar in many ways, it is possible that our analysis would produce different results if conducted using data from Subversion logs.

It should be noted that when the original data was gathered, projects were filtered to include only those projects listed as "Production," "Stable," or "Maintenance," in an effort to limit the data set to include only "successful" projects [START_REF] Daniel | Do programming languages affect productivity? a case study using data from open source projects[END_REF]. As a result, when we talk about topics across "All Commits," we are actually unable to generalize to the entire population of projects in SourceForge. This is significant, because as one estimate found, only about 12% of projects in Source-Forge were being actively developed [START_REF] Rainer | Evaluating the Quality and Quantity of Data on Open Source Software Projects[END_REF]. It is possible, even likely, that a similar analysis, not limited to "Production/Stable/Maintenance" projects would produce different results. However, we do not feel that the depiction of Cliff Walls would change dramatically, as we presume they are rare in defunct projects.

Another significant threat to the validity of our results is the presence of "low quality" topics. We found two types of low quality topics in our results: topics with contradictory terms and topics generated from dirty data. One example of a topic with contradictory terms is the "removed-deprecated" topic ranked 3rd in Tbl. 2. This topic contains the contradictory terms "removed," and "imported" as important terms in the topic. This leads to a topic that is difficult to interpret, as the "meaning" of the topic can vary based on the document in which it is present. To better understand this topic we examined the log messages of 233 Cliff Walls containing the topic. Of those 233, the term "removed" occurred in only 1 message, while "imported" occurred in 154. Obviously, for large commits, "imported" is a much more appropriate description of log messages with this topic.

We found two low quality topics resulting from dirty data in our results. The topics "error-message" and "message-project," the 5th and 6th most prevalent topics for Cliff Walls, are also misleading. We looked at 286 Cliff Wall log messages containing at least one of these two topics, and found that 211 ( 74%) of them contained only the message "no message." These commits should have been removed from the data set prior to the analysis. Exclusion of these commits would result in a data set containing 2,301,620 commits, with 9,199 Cliff Walls, a minor decrease in size. We do not feel that this issue greatly affected the outcomes of this study. However, these topics possibly prevented more appropriate topics from being considered.

In order to improve the results of future studies applying LDA to the Cliff Walls problem, greater effort should be made in LDA model selection. The three input parameters that we were required to specify (number of topics, number of iterations, and burn-in period) could likely be tuned to produce higher quality topics. In particular, this may help us to avoid the issue of topics with contradictory terms.

Conclusions

We are excited by the promise that LDA shows for automated analysis of large commits. Through the use of tag clouds and other views of the data, we have been able to gain an insightful picture into the causes behind Cliff Walls. We found that in most cases, our suspicions of Cliff Walls were confirmed. We found significant evidence that library imports, externally developed code, and merges were the subjects of topics frequently discussed in log messages of large commits. We also found evidence that auto-generated code can, in some cases, result in the creation of cliff walls. LDA also helped us to confirm that maintenance tasks, such as bug fixes, do not occur in large commits with much frequency. These conclusions agree with previous studies on the causes of large commits [START_REF] Hindle | What do large commits tell us?: a taxonomical study of large commits[END_REF][START_REF] Hattori | On the nature of commits[END_REF].

We found that it was difficult to use commit log messages to distinguish library code imports from imports of large amounts of project code. However, in some cases we are able to identify library imports. We also hope that, in the future, the URLs included in some Cliff Wall commit messages may be used to identify other instances of library code imports.

We believe that LDA is a welcome alternative to many of the methods that have previously been used for classification of commit log messages. While we invested a great deal of time manually interpreting the results produced by LDA, we were able to avoid the tedium of data tagging required by most supervised classification tasks.

Future Work

The role of large commits in software evolution is still largely unclear. In this study, we have examined the causes of Cliff Walls for a particular subset of all software projects-those that are relatively successful, are hosted on Source-Forge, and that use CVS for version control. In order to better understand Cliff Walls, we need to build upon this subset. First, research should consider investigating Cliff Walls as they occur in other version control systems. CVS is no longer the most significant version control system, since others, such as Subversion and GIT have risen to take its place.

SourceForge is only one of many environments in which open source software is developed. There are various open source forges and foundations, each with its own tools, communities, policies, and practices that influence the software development that occurs therein. It is possible that some of these other environments may prove more welcoming to those interested in studying the evolution of software. An effort should be made to characterize and compare the Cliff Walls that exist in other open source development communities, such as the Apache and Eclipse Foundations, RubyForge, and GITHub, to name a few. Of course the study of large commits should not be limited to only open source organizations, but should be investigated wherever possible.

More information may also be gained through a more in-depth analysis of the Cliff Walls themselves. The largest commit in our data set was over 13 million lines of code. In contrast, the smallest Cliff Wall contained 10,001 LOC. In this study, both of these commits, as well as everything in between, were lumped into the same class: Cliff Walls. It is likely that such a large level of granularity hides much that can be learned about the causes of Cliff Walls. We believe that there are opportunities to better understand this phenomenon by examining more closely the causes behind Cliff Walls of differing magnitudes.

Fig. 1 .

 1 Fig. 1. Cliff Walls in the Firebird Project

Fig. 2 .

 2 Fig. 2. Topic distribution for All Commits (left) and Cliff Wall Commits (right)

Fig. 3 .Fig. 4 .Fig. 5 .

 345 Fig. 3. Topic Tag Cloud: All Commits

Table 1 .

 1 Top 15 Topics for All Commits

	# Proportion			Key Terms
	1	1.75% version	release updated	update	final
	2	1.43%	file	branch initially	added	java
	3	1.42%	fixes	minor	small	cleanups	updates
	4	1.38%	data	minor updates	fixes	bugfixes
	5	1.38%	initial	import commit	checkin	revision
	6	1.36%	added comments comment documentation	javadoc
	7	1.29%	fixed	bug	bugs	incorrect	couple
	8	1.28%	added support	info	extended	basic
	9	1.28%	error message	errors	handling	checking
	10	1.26% removed	code template	commented unnecessary
	11	1.25%	fix	bug	fixing	small	bugs
	12	1.18%	fix	typo corrected	correct	errors
	13	1.17%	fixed	bug	wrong	crash introduced
	14	1.17%	page	link updated	links	url
	15	1.14%	code cleanup	source	clean	cleaned

Table 2 .

 2 Top 15 Topics for Cliff Walls

	# Proportion			Key Terms
	1	25.74%	initial	import commit	checkin revision
	2	5.11% version	release updated	update	final
	3	2.30% removed deprecated sources	constant imported
	4	1.63%	cvs	sync	tabs	real converted
	5	1.60%	error	message	errors	handling checking
	6	1.58% message	project messages	error	testing
	7	1.51% merged	merge	head	merging	main
	8	1.47%	code	cleanup	source	clean	cleaned
	9	1.29%	files	added dialogs	library directories
	10	1.21% directory	moved common	dir structure
	11	1.17%	xml updated	api	latest	version
	12	0.98% update	added format	updating creation
	13	0.93%	script	makefile	install	configure	sh
	14	0.89%	added comments comment documentation	javadoc
	15	0.89%	double	beta	v1	v2	values

Table 3 .

 3 Largest "Positive" Rank Differences

	All Rank Cliff Wall Rank		Key Terms
	+97 101	4	cvs	sync	update repository
	+97 112	15	beta	v1	v2	v3
	+93 131	38	org	cvs synchronized packages
	+91 98	7	merged merged	trunk	stable
	+91 100	9	files added directories	library
	+89 99	10	directory moved	structure	location
	+88 142	54	cc	net		sourceforge
	+80 128	48	module python	py	libsrc
	+73 119	46	system specific	platform	devel
	+71 133	62	http www	urls	net
	+71 105	34	web	site	resource component
	+67 109	42	php index	http	forum

Table 4 .

 4 Largest "Negative" Rank Differences

	All Rank Cliff Wall Rank		Key Terms
	-120 13	133	fixed	bug	wrong	crash
	-118 12	130	fix	typo corrected correct
	-116 7	123	fixed	bug	bugs incorrect
	-105 44	149	button selection	tab	dialog
	-90 23	113	fixed problems	issue	bad
	-89 30	119	string	return	null	true
	-80 51	131	variable	global unused	define
	-77 26	103	output	debug	print messages
	-76 49	125	problem	fixed	patch solution
	-75 36	111	size	buffer	limit	bytes
	-75 54	129	function	calls	static	inline
	-74 3	77	fixes	minor cleanups cosmetic

The other two parameters, number of iterations and length of burn-in period, are required by the Gibbs Sampling inference method employed by MALLET. We refer the reader to[START_REF] Wallach | Rethinking LDA: Why priors matter[END_REF] for a description of LDA as it is implemented within MALLET, including a description of Gibbs Sampling. For these two parameters we used the default values provided by MALLET; 1,000 iterations with

dedicated to burn-in. Further work should investigate the possibility of more appropriate values for all three parameters, as discussed in Sec. 6.

All tag cloud images were generated using Wordle (www.wordle.net).

Acknowledgements

The authors would like to thank Dan Walker of the BYU Natural Language Processing Lab for his willingness to provide insight and guidance on the methods used in this paper.


Tetsuo Noda
email: nodat@soc.shimane-u.ac.jp

Terutaka Tansho
email: tansho@riko.shimane-u.ac.jp

Shane Coughlan

Standing Situations and Issues of Open Source Policy in East Asian Nations: Outcomes of Open Source Research Workshop of East Asia

East Asia nations have made some progress with this technology, and started to introduce OSS for e-government systems during the early part of this century. Many countries granted it a central role in their policies. The reasons for this include adoption of software based on standard specification, liberation from vender lock-in, or opposition to the market control of proprietary software. However, the primary reason is to reduce adoption costs for e-government systems. While this policy work is useful, there is a great deal more that needs to be done. The OSS adoption policy in each nation of East Asia must be accompanied by technological progress in domestic IT service industries or US multinationals will expand at the cost of local businesses. If this continues unchecked it will create a new form of lock-in for East Asian nations. Some Asian nations are trying to promote their domestic IT service industries, putting their OSS adoption policy to practical use, and this workshop will provide case studies of that work. It will also provide a forum for discussing current challenges and opportunities around both policy and practical implementation issues across Asia.

Introduction

As part of Shimane University's research into Open Source matters, the Research Project Promotion Institute 1 has run a project called 'Stabilization and Business Models for Open Source Software' since 2008. One deliverable has been the hosting of a seminar entitled 'Open Source Research Workshop in East Asia' on November 26th and 27th 2010. This seminar was attended by Japanese and academic thoughtleaders as well as a diverse range of international participants.

East Asian nations have made some progress with Open Source technology in the last decade, building on the early introduction of OSS for e-Government systems at the turn of the last century. Many countries granted OSS a central role in their policies for reasons like adherence to standard specification, freedom from vender lock-in, or opposition to the market control of proprietary software. However, the primary reason is to reduce adoption costs for e-Government systems.

While the initial governmental policy work has been useful, a great deal more needs to be done before the value proposition offered by OSS is realized. One key example is that the OSS adoption policy in each nation of East Asia must be accompanied by progress in domestic IT service industries to prevent multinationals expanding at the cost of local businesses. This is a consideration given that Open Source originates from the West coast of the USA and is still primarily developed and enhanced by US corporations. It could even be said that the current technical evolution of OSS is driven mainly by companies originating from the United States. While the inherent benefits of OSS extend beyond the boundaries of enterprises, organizations and nations, and OSS has the potential to foster new business markets in regions other than North America, the current status quo has the potential for a new form of lock-in for East Asian nations.

Some East Asian nations are trying to promote domestic IT service industries by putting their OSS adoption policy to practical use, and this workshop provided case studies of that work. It also provided a forum for discussing current challenges and opportunities around both policy and practical implementation issues across East Asian nations.

The intention was to extract the aspects of Open Source adoption policy that are not accompanied by the technological progress in domestic IT service industries, and from this derive an indication of the role government should play in East Asia and in other developing countries generally.

At the workshop Open Source thought-leaders from Japan, China, South Korea, Vietnam, The Gambia and Ireland discussed the nuances and known outcomes of Open Source adoption policies, and contributed to the publication of a proceeding as a special issue of our bulletin2 in conjunction with the event. As the contentions of the researchers are contained in the proceeding, the focus of this paper is to explain the presentations of the researchers and the main points of discussion at the workshop.

Open Source Policy in East Asian Nations

Open Source Policy in Japan

Mr. Shunichi Tashiro, the chief officer of Open Software Center at the Japanese Industrial-Technology Promotion Agency3 , introduced Open Source Policy in Japan and the activity of his department.

He explained that software is built on platforms, and that it ultimately cannot work without interoperability with other software. Openness of the platforms and interfaces are crucial to provide freedom to software developers and users, and therefore to foster a healthy, competitive environment in the national and international software industry. The Open Software Center was founded in 2006 to promote Open Source software and open standards as an important aspect of information services, development methods, assessment criteria, standardization systems, and research studies. The core proposition is that OSS enhances knowledge sharing and sustains a collaborative development environment. The key issue is probably 'Open Standards' rather than Open Source. Open Standards are technical standards for which specifications are publicly available, that any part can use and that any party can participate in regarding further development. These are critical to ensure the interoperability of software and freedom of action for software developers and users.

One example of practical engagement with this issue in Japan is the standardization of the programming language Ruby, which is now in the screening process of JIS (Japanese Industrial Standards) and will soon obtain JIS certification before submission to ISO (International Organization for Standardization) in 2011. This will encourage the Japanese-created Ruby language to be adopted in e-Government systems and enterprise environments across the world. In this way a Japanese government agency is supporting the promotion of the domestic information service industry while also contributing positively to the global IT market.

Open Source and the Software Industry in China

Dr. Dongbin Wang, the researcher of Center of China Study, Tsinghua University [START_REF] Yi | Comparative study on the implementation and performance of open source activating policy between Korea and Japan[END_REF] , introduced the history of the Internet and Open Source in China.

Before China joined in WTO in 2001, software piracy was very popular and lead to a large loss of potential revenue every year for the industry. One high profile example is that Microsoft's software products were frequently copied and illegally installed on computers across the private and public sectors, a situation that somewhat ironically also helped Microsoft become the de-facto norm in computing for the local market. However, after 2001 the Chinese government began to increase the legislation and enforcement around software piracy, and companies like Microsoft accompanied this shift in the local market norms by taking a much stricter line regarding unauthorized coping of their products. One side effect of this was for many cyber cafes to react by switching to Linux and other Open Source software to reduce licensing costs.

It is possible to conclude that intellectual property rights protection is a doubleedged sword for Open Source development and adoption. Recently many large companies in China have formally preferred Linux as a pre-installation operation system, but due to the rampant piracy of Windows in the past there is an on-going familiarity and desire for Microsoft and other proprietary products. In effect, the over commercialization and effective commoditization of proprietary software presents a key obstacle to Open Source expansion in China.

Open Source Software -Education, Practice and Applications at the

University Of Engineering and Technology

Dr. Nam Hai Nguyen & Dr. Quang Hieu Le from the University of Engineering and Technology school of the Vietnam National University, Hanoi(UET) [START_REF] Yi | Enhancing the Circulation: Some Implications on the OSS Policy and IT Industry Promotion In[END_REF] provided information on policies and challenges experienced by the Vietnamese government regarding OSS development in their country.

The national government has long emphasized a focus on and priority for OSS development and deployment. For example, in 2006 the government mandated that procurement of IT products and services should give preference to investment, adoption and application of OSS and OSS-based software products, especially those with quality and functionality equal to proprietary software supplied by domestic enterprises UET has engaged with the trend towards OSS by exploring best practices in the field and building educational knowledge to share with students and with society-atlarge. Through these activities the faculty members of UET play an important role in advancing the widespread use of OSS in teaching and research activities, and in practical deployment outside of the confines of academia. To further both the activities of UET, and the Vietnamese engagement with OSS generally, efficient international cooperation is of crucial importance. UET

Free and Open Source Software Governance: Turning Potential into Deliverables

The presentation examined some of the key governance approaches and resources that help turn the potential of FOSS into a deliverable, whether that deliverable is a product, revenue stream or altruistic solution to a shared problem. The concept of governance has become increasingly important because -while Free and Open Source Software (FOSS) offers tremendous potential to create technology platforms and develop business opportunities -the best methods to obtain results from FOSS adoption have not always been clear. Simply being 'open' or using third party code appears to have limited value on its own, and does not address the management requirements or legal imperatives that stakeholders face. Deriving ongoing value from FOSS requires understanding the ideas and norms that underpin the field. FOSS has a premise that third-party sharing and contribution delivers enhanced value over proprietary approaches to managing software, and that FOSS can deliver this value through copyright licenses that allow everyone to use, study, share and improve software code. As this idea has reached the mainstream, the licenses are increasingly the subject of legal and management scrutiny, and best practices have inevitably emerged for adopting, developing and deploying code subject to their terms.

It was explained how organizations ranging from the Linux Foundation to Free Software Foundation Europe have built networks, encouraged publications and developed tools to assist their own audiences and the broader community of all potential FOSS stakeholders, and some of these governance projects or organizations were explored in brief case studies.

Discussion and Conclusion: The Potential Crowding-Out Effect of Government Policy

The majority of the presentations therefore engaged with the role of each Asian government or educational institutional in furthering Open Source policy and its practical application. One immediately observable point from their exploration is that each nation discussed places importance on Open Source adoption and supports it politically. However, the point was raised that excessive policy engagement by central government might spoil the development of Open Source communities by creating a crowding-out effect. There are many stakeholders involved and the interests of these stakeholders are not necessarily entirely in sync. The development methods of Open Source software often include collaboration on the Internet through the participation of a number of developers, corporations and other stakeholders. This environment is competitive not only towards proprietary software but also within itself, and as such requires the flexibility that markets and their associated competitiveness provide.

As the motivation factors of Open Source software developers and investors vary, they tend to participate in creating code and therefore platforms with broadly applicable value. Government engagement needs to take this into account, and avoid causing the risk of constrained, prescriptive environments for Open Source development. This applies in the educational field too. While it is clearly important to explore and integrate OSS in educational environments, especially in the context of computer engineering, such coverage needs to be broad. The field needs to be presented in a context that does not ignore the way that OSS often appears, grows and delivers value outside of the traditional constraints and assumptions of formal engineering education. The discussion suggested that governmental policy around OSS is not clear cut. While positive policy is to be encouraged, the value offered by OSS needs to be understood clearly to ensure it is not inadvertently smothered by well-meaning but ineffective mandates. This indicates that perhaps there are 'new' issues of Open Source adoption policy in East Asian nations to consider, and all participants at the workshop agreed to proceed with examination of how these issues can be resolved in a future workshop.

  is a Development and China Study, Chinese Economy and Development Strategy. http://www.facebook.com/group.php?gid=48859734768 5 http://www.vnu.edu.vn/en/contents/index.php?ID=932 leading proponent of such cooperation, is seeking to build knowledge-sharing bridges with the broader global community.

Shimane University is executing forward empirical and theoretical research on the productivity of the business model's construction is to be done by the cooperation of the industrial-government-academic-community complex. http://albatross.soc.shimaneu.ac.jp/oss/index.html

Journal of Economics Memoirs Of The Faculty Of Law And Literature, Shimane University No.37 Special Issue "Open Source Policy and Promotion of IT Industries in East Asia" Nov.2010

The Open Software Center is an organization within the Information-Technology Promotion Agency (IPA), Japan, one of Independent Administrative Agencies 1 in Japan, and is operated under the budget of the Japanese government. http://www.ipa.go.jp/index-e.html

Center for China Study (CCS), Tsinghua University is a leading academic think-tank for policy making in China. Its researches cover most fields of China Studies, which include