Kristiina Nurmela
email: kristiina.k.nurmela@jyu.fi

Antti Pirhonen
email: antti.pirhonen@jyu.fi

Airi Salminen
email: airi.salminen@jyu.fi

Kristiina Nurmela¹

Antti Pirhonen²

Airi Salminen²

Accessibility of Public Web Services: A Distant Dream?

Keywords: Accessibility, public administration, WCAG

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Building the so-called information society is usually conceptualised primarily as a technological task [START_REF] Halttunen | Slower, Better. Do Information Society Visions Have Healthy Alternatives?[END_REF]. This can be seen, for instance, in the national information society programmes. Finland was among the first ones to prepare one, in the mid-1990s [START_REF]Finland's Way to the Information Society[END_REF]. That report contained a technically oriented futuristic vision. More recently, the official objectives have been more human-centric. The [Finnish] National Knowledge Society Strategy for 2007-2015 is entitled "A renewing, human-centric and competitive Finland" [START_REF] Renewing | Human-Centric and Competitive Finland[END_REF]. The objective for developing the society is expressed as "A good life in the information society" [START_REF] Renewing | Human-Centric and Competitive Finland[END_REF]. However, the approach of this document is still very techno-optimistic. Like in many other countries, high expectations have been set concerning the opportunities of networked society for democracy [see e.g. 4]. This kind of enthusiastic visions should be taken dubiously, if they don't include critical analysis.

Public Web services are meant, at least in principle, for all citizens. They should be in a form which is accessible for everyone. For instance, people with disabilities should be taken into account in the design of the services.

We wanted to find out to what extent the accessibility has been taken into account in the design of public Web sites. Well-known criteria for Web content accessibility have been defined in the guidelines of the World Wide Web Consortium (W3C), called Web Content Accessibility Guidelines (WCAG). Our study was carried out in two steps. First we accomplished a meta-analysis of 17 studies concerning the accessibility of the Web sites of public administration. Then we assessed the accessibility of Web pages of 12 ministries of the Finnish government. The assessments were based on the WCAG guidelines. Before describing the two parts of the study and their results, we first discuss the core concepts of our study: disability, accessibility, and evaluation of accessibility.

Perspectives on Disability

The concept of disability (along with related concepts like impairment and handicap) has been a topic of lively debate for decades, at least from the early 1960s [START_REF] Oliver | Defining Impairment and Disability: Issues at Stake[END_REF][START_REF] Oliver | Understanding Disability: from Theory to Practice[END_REF]. Analysis of all the nuances of the debate is out of focus of the current study. We contend ourselves with the introduction of two, clearly distinctive main approaches: the medical and the social approach to disability. The medical model of disability is probably the most mundane approach in conceptualising what disability is all about. In it, people are classified on the basis of medical criteria to those who are disabled and those who are not. In the approach, disability is seen primarily as individual's problem which should be overcome. Due to the complex relationship between an individual and the society, individual's problem obviously concerns society as well.

In the traditional medical definition of disability, the stumbling block, causing strong arguments is obvious: "In the context of health experience, a disability is any restriction or lack (resulting from an impairment) of ability to perform an activity in the manner or within the range considered normal for a human being" [START_REF]World Health Organisation) International Classification of Impairments, Disabilities, and Handicaps: A Manual of Classification Relating to the Consequences of Disease[END_REF] As can be seen, this definition goes "out of the frying pan into the fire": while defining disability, it introduces another, similarly disputable and even ethically questionable concept of normality of a human being. Even though the expression has been smoothened in the updated version of the same document [START_REF]World Health Organisation) International Classification of Impairments, Disabilities, and Handicaps: A Manual of Classification Relating to the Consequences of Disease[END_REF], the individual's perspective is still present.

The contradictory approach is commonly called as social model of disability. In this perspective, the relationship between an individual and the society is approached from the opposite direction: disability is seen to result from the structures which prevent some individuals from participating in the function of the society. In other words, no one is disabled as such; the disability appears in the discordance between the individual and the environment.

The contradiction between the social and medical views of disability is nicely compromised in the Convention on the Rights of Persons with Disabilities by United Nations [START_REF]Convention on the Rights of Persons with Disabilities[END_REF]. In its Article 2, the concept of universal design is defined to mean:

"… the design of products, environments, programmes and services to be usable by all people, to the greatest extent possible, without the need for adaptation or specialized design. "Universal design" shall not exclude assistive devices for particular groups of persons with disabilities where this is needed."

In other words, the priority is to design for all, yet admitting that in some cases, special arrangements are necessary.

Even though practical implications for the design of public services may be in many cases the same regardless of the perspective, the distinction between the medical and social approaches is important in the long run.

Perspectives on Accessibility

Regardless of the perspective on disability, accessibility is a fundamental issue and criterion in the development of public services. This is based on wide agreement about the rights of the disabled people, formalised in the Convention on the Rights of Persons with Disabilities [START_REF]Convention on the Rights of Persons with Disabilities[END_REF], in Article 9: "States Parties shall also take appropriate measures:

… (f) To promote other appropriate forms of assistance and support to persons with disabilities to ensure their access to information; (g) To promote access for persons with disabilities to new information and communications technologies and systems, including the Internet; (h) To promote the design, development, production and distribution of accessible

information and communications technologies and systems at an early stage, so that these technologies and systems become accessible at minimum cost." In other words, importance of information access has been seen such a central right, that it is stressed by expressing the issue from the points-of-view of information, related technology and even the development of technology.

Sir Timothy Barnes-Lee himself, the director of the World Wide Web Consortium (W3C), has been one of the key persons behind recent accessibility efforts e.g. in the UK [10]. Indeed, the Internet and the World Wide Web are the technologies which have become essentials for citizens. It is therefore quite natural, that much of the accessibility efforts concern the World Wide Web.

Evaluation of Accessibility

Web accessibility is usually bind to the accessibility guidelines of W3C, called Web Content Accessibility Guidelines (WCAG), version 1.0 from 1999 [START_REF]Web Content Accessibility Guidelines (WCAG) 1.0[END_REF] and 2.0 from 2008 [START_REF]Web Content Accessibility Guidelines (WCAG) 2.0[END_REF]. The two versions have a lot in common; the later one, however, is applicable in a wider range of devices. For instance, version 2.0 contains four general accessibility principles hierarchically above the detailed guidelines; the principles are thus applicable even when the technology specific, detailed guidelines appear irrelevant.

The WCAG guidelines provide a number of checkpoints that can be used to check the accessibility of Web content. Each checkpoint is associated with one of three priority levels. Level 1 criteria or checkpoints define the basic and most critical features that accessible content must satisfy, level 2 checkpoints refer to the criteria that the content should satisfy, and level 3 to the criteria that may be satisfied. The checkpoints in guidelines are constructed in a hierarchical manner so that the requirements concerning the same feature get higher when proceeding from level 1 to level 2, and further to level 3. For the content to be tested, for example, a Web page or a Web site, the guidelines define three conformity levels: A, AA, and AAA. In order to be on level A, all level 1 criteria have to be satisfied. On level AA, all the criteria of levels 1 and 2 have to be satisfied. Finally, on level AAA, all criteria of all levels have to be satisfied.

There are several other guidelines and criteria for accessibility, but in this study, we focus on WCAG.

WCAG can be applied both in the design of accessible Web content as well as in the evaluation of accessibility of existing content. The evaluation methods can be divided into automated, semi-automated, and manual. Since in the definition of WCAG it is stated that the criteria has to be testable [START_REF]Web Content Accessibility Guidelines (WCAG) 2.0[END_REF], the success criteria are concrete and many of the criteria can be tested automatically.

Meta-Analysis of Previous Public Administration Accessibility Studies

In order to get an overall view of the current state of accessibility in public administration Web sites, we carried out a meta-analysis of existing relevant accessibility studies which have been reported in scientific forums.

The Data

The criteria and method for the article sample was as follows: They had to be published in the proceedings of international conferences, journals, or in the final reports of large research projects. The reports were searched from the databases of ACM, SpringerLink, and Elsevier. Google Scholar was also utilised, as well as the reference lists of relevant articles. The keywords used in the searches were "WCAG 1.0", "WCAG 2.0", "accessibility", "Web sites", "public government" and "public administration". Since WCAG 1.0 was released in 1999, only articles which have been published after that were included. We found 23 studies in total. In a closer inspection, it was found out that four of them did not report the application of WCAG detailed enough for the needs of this analysis. In addition, one report was excluded because of quality problems (did not actually use WCAG 2.0 criteria even if it was argued in the report that it did). The final list of 17 included studies is provided in Table 1. The names in the third column refer to the automated tool used. The figures in the 4 th and 5 th column indicate the percentages. "*" stands for N/A.

2.2

The level of accessibility

As can be seen in Table 1, in four of the included studies none of the analysed pages met even the WCAG level A. In three studies, less than ten per cent of the pages reached level A. In six studies, the percentage of level A pages was between 10 and 25. In four studies only the percentage of pages that satisfied level A criteria, was above 25.

In 10 of the included studies the conformance to level AA was assessed. In them, the percentage of the level AA pages was 2 at its best. In six studies the conformance to level AAA criteria was assessed as well, but not a single page met that level.

It has to be kept in mind that for reaching the compliance level A the page cannot fail in a single level 1 criterion. For reaching the compliance level AA, the page has to satisfy all level 1 and 2 criteria. And only pages satisfying all of the criteria, on all three levels, reach level AAA compliance. This means that one failure on level 1 is enough to prevent calling a page WCAG compliant, on any level, even though all other criteria were satisfied. Therefore it is important to take a look at the results concerning individual success criteria and the most typical failures concerning them.

2.3

Typical accessibility failures WCAG 1.0 Priority 1 Checkpoints  Checkpoint 1.1: "Provide a text equivalent for every non-text element." This is essential e.g. for blind users applying screen readers. All of the included 17 studies mentioned this as one of the main accessibility problems.  Checkpoint 12.1: "Title each frame to facilitate frame identification and navigation." Even though the use of frames is not as usual as previously, they remain one of the most common accessibility challenges. Screen readers, for example, do not necessarily read the contents of frames. The titling of frames was found as a problem in 7 studies.  Checkpoint 6.3: "Ensure that pages are usable when scripts, applets, or other programmatic objects are turned off or not supported. If this is not possible, provide equivalent information on an alternative accessible page." Often, scripts are disabled, so if the use of a Web site requires the functioning of scripts, they are an accessibility problem for anyone. Six of the studies found problems in the use of scripts.

WCAG 1.0 Priority 2 Checkpoints

Ten of the included studies assessed level AA conformity. The most common failures concerned:

 Checkpoint 3.4: "Use relative rather than absolute units in mark-up language attribute values and style sheet property values." From the point of view of accessibility, it is important that the user-interface elements are scalable to different needs and terminals. Eight studies mentioned this as an issue.  Checkpoint 9.3: "For scripts, specify logical event handlers rather than devicedependent event handlers." Device dependent technology is problematic since especially disabled people may be unable to use standard devices and have to use customised technology instead. The violation of this guideline was detected in five studies.  Checkpoint 13.1: "Clearly identify the target of each link." Important for users with cognitive disabilities and users with visual impairments in particular. Classified as a problem in four studies.  Checkpoint 3.2: "Create documents that validate to published formal grammars."

Valid code ensures that it works in different platforms, including assistive technologies. Either HTML or CSS code was found invalid in four studies.  Checkpoint 12.4: "Associate labels explicitly with their controls." This is important especially for the visually impaired users, who are unable to utilise visual cues in associating e.g. a form field and its help text.

WCAG 1.0 Priority 3 Checkpoints

In the six studies assessing level AAA conformance the priority 3 failures mostly concerned  Checkpoint 4.3: "Identify the primary natural language of a document."  Checkpoint 5.5: "Provide summaries for tables."  Checkpoint 10.5: "Until user agents (including assistive technologies) render adjacent links distinctly, include non-link, printable characters (surrounded by spaces) between adjacent links." WCAG 2.0 Surprisingly, only one of the studies contained analysis of the accessibility problems on the basis of version 2.0 of WCAG. Therefore, it was not possible to make any comparisons or generalisations about them.

Summary of the meta-analysis

Table 1 shows that in average the level of accessibility in terms of WCAG is low. It also shows major differences in the results, especially concerning the level A compliance of public sector Web pages of the case countries or areas. Based on this metaanalysis we cannot, however, make any kind of ranking of the countries or areas in respect to their WCAG level A compliance. Likewise, any other kind of direct comparisons among the included studies is problematic due to their evident differences; without going into details in the differences in the data and analysis methods of the included studies, it is easy to see that already in the number of pages assessed in the studies there is a huge range: from 5 to 272778. The earliest of the studies has been published in 2002, the latest in 2011. Our expectation was that some improvement might be visible during the time frame from 2002 to 2011. To our surprise no such improvement could be seen. In other words, there is no sign in this sample that accessibility had improved over recent years. Another surprise was related to the individual failures and their types. Most of the violations of WCAG checkpoints were plain and apparent. Since the form of WCAG is extremely pragmatic and concrete, most of the identified accessibility violations would have been easy to avoid. This may indicate an attitude problem rather than lack of resources. The issue is further discussed in the last section of this paper.

Accessibility of the Web Sites of Finnish Ministries

After the meta-analysis indicating major accessibility problems of the Web sites of the public administration in the USA and many other countries we wanted to assess the accessibility level of the Finnish public administration.

The Data and the Analysis

We chose the Web sites of the Finnish ministries for the assessment because the Finnish Government has emphasized for several years the importance of building the Finnish Information Society. In the Finnish Government there are 12 ministries. From the ministry Web sites we analysed 108 pages altogether. The chosen pages included the front page of each of the ministries, and the second-level pages which were linked from the main navigation bar of the front page.

In order to gain commensurability with analysis of previous studies, we used a similar kind of approach: We analysed the WCAG compliance and searched for typical accessibility failures. Most of the analysis was carried out with the help of automated accessibility tool, called Worldspace [START_REF]Worldspace[END_REF]. We ended up to an automated tool since this strategy is usually high in reliability. Automatic analysis is prone to validity problems [START_REF] Vigo | Automatic web accessibility metrics: Where we are and where we can go[END_REF], though, which has to be taken into account in the interpretation of results. In addition to the application of Worldspace, the validity of the HTML code was checked with the markup validator of W3C. From the available automated tools Worldspace was chosen since it is free, covers both versions of WCAG, is included in the listing of W3C and is easy-to-use. It covers all conformity levels of WCAG 1.0 and levels A and AA of version 2.0.

Since automated analysis is purely based on the HTML code, ideally all the results would be manually checked. In the current study, in terms of the resources available, full scale manual inspection was not an option.

3.2

Detected accessibility failures

Criteria: WCAG 1.0 Our analysis shows that out of the 108 Web pages, only 9 reached the level A conformance of WCAG 1.0. Of these 9, 8 were pages of the Ministry of Transport and Communications, which is a positive indicationat least the Ministry which is responsible for the domestic Internet policy has applied the accessibility guidelines. Unfortunately, even this ministry's expertise has not yielded above level A: no single page in the whole sample reached the AA conformance, not to speak about AAA.

The common violations of WCAG 1.0 are quite in accordance with the previous studies. We now list the ones which came up in more than half of the pages, or at least three most common problems in each level.

The foremost violation of the first level checkpoints concerned the checkpoint 1.1: "Provide a text equivalent for every non-text element." The failure to meet this criterion was detected in 80 per cent of the pages. The second most common issue on the first level was about the checkpoint 12.1: "Title each frame to facilitate frame identification and navigation." The difference between the frequency of 1.1 problems and 12.1 problems was huge: only 10% of the pages violated guideline 12.1. The third most common violation was about 6.1: "Organize documents so they may be read without style sheets", which was violated in 8% of the pages. In other words, just by providing text equivalents for all non-text elements, the accessibility rating of these pages would have been hugely higher.

On the second priority level the problems were more evenly distributed among checkpoints:  13.1: 91% of the pages violating ("Clearly identify the target of each link")  3.2: 91% ("Create documents that validate to published formal grammars")  12.4: 85% ("Associate labels explicitly with their controls")  3.5: 68% ("Use header elements to convey document structure and use them according to specification")  3.4: 63% ("Use relative rather than absolute units in markup language attribute values and style sheet property values")  10.1: 62% ("Until user agents allow users to turn off spawned windows, do not cause pop-ups or other windows to appear and do not change the current window without informing the user")

Of these, only 3.5 and 10.1 did not come up in the list of most common accessibility issues in the review of previous studies. This shows that the types of accessibility problems are quite similar on the Web sites of public administration in different countries.

On the third priority level, the number of problems was huge, the most common ones distributed as follows:

 10.5: 95% ("Until user agents (including assistive technologies) render adjacent links distinctly, include non-link, printable characters (surrounded by spaces) between adjacent links")  9.5: 88% ("Provide keyboard shortcuts to important links (including those in client-side image maps), form controls, and groups of form controls")  13.6: 88% ("Group related links, identify the group (for user agents), and, until user agents do so, provide a way to bypass the group")  2.2: 81% ("Ensure that foreground and background color combinations provide sufficient contrast when viewed by someone having color deficits or when viewed on a black and white screen")  4.3: 71% ("Identify the primary natural language of a document")

A good example is the most common of the level 3 failures. Guideline 10.5 had been very simple to follow. Whether it is a question of negligence, ignorance or something else, obviously cannot be answered on the basis of this study.

Criteria: WCAG 2.0

In terms of the WCAG version 2.0, none of the pages of our sample reached even the lowest level of conformance (A). In the review of previous studies, the conclusions were similar; only very few pages of public administration reached the A level.

According to the results of the automated analysis, the most common accessibility problems concerned criteria on the first priority level, in particular, the following criteria:  1.3.1: 93% ("Information, structure, and relationships conveyed through presentation can be programmatically determined or are available in text")  3.3.1, 89% ("If an input error is automatically detected, the item that is in error is identified and the error is described to the user in text")  1.1.1, 77% ("All non-text content that is presented to the user has a text alternative that serves the equivalent purpose, except for the situations listed below")  3.1.1, 71% ("The default human language of each Web page can be programmatically determined")  4.1.2, 67% ("For all user interface components…, the name and role can be programmatically determined; states, properties, and values that can be set by the user can be programmatically set; and notification of changes to these items is available to user agents, including assistive technologies")  2.4.4, 61% ("The purpose of each link can be determined from the link text alone or from the link text together with its programmatically determined link context, except where the purpose of the link would be ambiguous to users in general") Surprisingly, only one problem was detected by Worldspace concerning the second level success criteria. The failure concerned the criterion 1.4.3: "The visual presentation of text and images of text has a contrast ratio of at least 4.5:1"(…plus some UIelement specific refinements). From the tested pages 49% failed to meet the criterion.

Discussion

There are today a great number of different public services available through Web sites. The limitations in the accessibility of the content of a site may fully prevent using the main functionalities of the site. Our study reveals that the level of Web content accessibility of public administration is in low level. Since accessibility of public Web services is essential in terms of the principles of democracy, improving the accessibility is a necessity.

We used WCAG as a criterion for accessibility in this study. Those guidelines have their strength in being concrete and measurable. They do not, however, take a stand on more ambiguous qualities like attitudes or ethical values. Therefore, there may be a risk that the usage of guidelines is mechanical fixing of errors. In other words, application is first designed and only after the initial design accessibility guidelines are utilised. An analogous problem has already been handled in human-computer studies for quite a while ago: When personal computers became common, user-interface was seen as a means to make the complicated computer technology usable for everyone. It was found soon, however, that a nice user-interface cannot compensate fundamental usability problems. The solution was to change the process: rather than creating unusable technology and only after that try to make it usable with a fancy user-interface, we should include the user's point-of-view in all stages in design.

We argue that the approach described above should be applied to the design of Web content. Rather than creating inaccessible content and later try to fix it with the help of guidelines, the content should be designed accessible in the first place. The form of technology should not be the primary criterion for the development of public servicesaccessible service may not always be the fanciest looking and not even the cheapest option.

It can be concluded that the construction of accessible services is not a huge technological challenge. The biggest challenge is in the attitudes. Designers of public services should see their role as constructors of democratic society rather than only as application designers [START_REF] Pirhonen | The Design of Information Technology as Design of the Information Society[END_REF].

Table 1 . The included studies

 1

	1 http://www.erigami.com/truwex/
	2 http://oa.mo.gov/itsd/cio/architecture/domains/interface/PC-Accessibility-
	MOITAccessiblityStds-WatchfireWebxact060606.pdf
	3 http://www.tawdis.net/
	4 http://www.aprompt.co.uk/45/Website_Accessibility_Testing/


Hironobu Takagi
email: takagih@jp.ibm.com

Susumu Harada
email: haradas@jp.ibm.com

Daisuke Sato
email: dsato@jp.ibm.com

Chieko Asakawa

Lessons Learned from Crowd Accessibility Services

Keywords: Crowd-sourcing, accessibility, digital book, captioning, Web accessibility

Crowd accessibility services for people with disabilities, driven by crowd-sourcing methods, are gaining traction as a viable means of realizing innovative services by leveraging both human and machine intelligence. As the approach matures, researchers and practitioners are seeking to build various types of services. However, many of them encounter similar challenges, such as variations in quality and sustaining contributor participation for durable services. There are growing needs to share tangible knowledge about the best practices to help build and maintain successful services. Towards this end, we are sharing our experiences with crowd accessibility services that we have deployed and studied. Initially, we developed a method to analyze the dynamics of contributor participation. We then analyzed the actual data from three service deployments spanning several years. The service types included Web accessibility improvement, text digitization, and video captioning. We then summarize the lessons learned and future research directions for sustainable services.

Introduction

Accessibility technologies have advanced significantly over the years, but there still remain many issues that technology alone has not been able to solve. Screen readers and refreshable Braille display technologies opened up a wide array of information to people with visual impairments, but areas such as the automatic conversion of visual information into textual descriptions remain major challenges. Automatic speech recognition and remote captioning technologies have expanded the communication possibilities for people who are deaf or hard of hearing, but for informal speech or conversations in noisy environments, automatic recognition has yet to attain practical levels of reliability.

In recent years, there have been rapid developments of crowd-sourcing technologies to bridge the gaps in mechanical computing technologies, based on harnessing and combining the perceptual, cognitive, and intellectual abilities of human beings. Pioneering services such as the ESP Game [START_REF] Von Ahn | Labeling images with a computer game[END_REF] have demonstrated the power of this approach. Crowd accessibility services have emerged as a category of crowd-sourcing service focused on supporting people with disabilities. General crowd-sourcing ser-vices sometimes are designed to train machine intelligence systems instead of supporting humans. For example, building a dictionary for machine translation can be crowd-sourced to people without directly specifying the human beneficiaries or their specific objectives.

In contrast, crowd accessibility services focus on supporting specific people with disabilities. A captioning service seeks to support people who are deaf or hard of hearing, or a book digitalization service can focus on supporting readers and learners with visual disabilities [START_REF] Benetech | Accessible books and periodicals for readers with print disabilities -Bookshareaccessible books for individuals with print disabilities[END_REF][START_REF] Ishihara | Transforming Japanese archives into accessible digital books[END_REF]. One category of new services is called "vision as a service", and it helps blind people to recognize all of the things around them [START_REF] Bigham | VizWiz: nearly real-time answers to visual questions[END_REF][START_REF]IQ Engines: image recognition and visual search[END_REF]. The existence of human beings who need support strongly distinguish accessibility services from other services.

In various ways, crowd accessibility services can enable dream-like services for people with disabilities. However, it is sometimes difficult to sustain these services because of the small sizes of the target populations. For sustainable services, the supply of workers and the work requests should be balanced, but this is often difficult for crowd accessibility services. This characteristic generally obliges service providers to depend upon unpaid crowd volunteers.

At the same time, crowd accessibility services also have characteristics that present special opportunities. Since crowd accessibility services tend to be focused more on social services and not commercial aspects, we have learned that these contributors tend to be more motivated by altruistic factors rather than by monetary gains or by the diversion and novelty offered by quick and simple tasks, which are among the motivators for traditional crowd-sourcing services.

How can these services succeed and endure while recruiting new and active contributors? How can we motivate them? How can we insure the quality of outputs? Little work has been done to analyze the issues that are specific to crowd accessibility services, since the analysis methods themselves are challenging given the diverse characteristics of the services and the limited availability of data for such analyses.

In this paper, we offer some answers to these questions by analyzing data from three services of our own. After the review of related work, an analysis method is proposed by taking into account the practical availability of data. After examining our data in the observations section, our lessons learned focus on four topics. Before offering our conclusions, a wide range of future research directions are discussed for social infrastructures based on crowd accessibility approaches.

2

Related Work

Crowd Accessibility Services

Crowd accessibility services are within the broad category of crowd-sourcing services, but are characterized by their focus on people with disabilities as the beneficiaries of the services. Image labeling for accessibility is one of the popular applications for crowd accessibility. Dardailler [START_REF] Dardailler | The ALT-server[END_REF] pioneered the approach of labeling images using the power of the crowd in the ALT-server proposal that stores alternative text on a remote server. The ESP game [START_REF] Von Ahn | Labeling images with a computer game[END_REF] used the gamification approach in labeling images to support both image searches and accessibility. This meant the service was not purely a crowd accessibility service, but suitable for general and accessibility applications. Optical character recognition (OCR) technology was combined with crowd-sourcing in the WebInsight system [START_REF] Bigham | WebInSight: making Web images accessible[END_REF]. This was the first approach of combining human intelligence with computer intelligence to improve accessibility. Social Accessibility [START_REF] Takagi | Social Accessibility: achieving accessibility through collaborative metadata authoring[END_REF][START_REF] Takagi | Collaborative Web accessibility improvement: challenges and possibilities[END_REF] focused on comprehensive and high quality Web accessibility improvements including image labels. VizWiz [START_REF] Bigham | VizWiz: nearly real-time answers to visual questions[END_REF] explored real-time labeling by recruiting Web workers from the Amazon Mechanical Turk service to answer vision-related questions submitted as photos by blind users using smartphones. [START_REF]IQ Engines: image recognition and visual search[END_REF] is a commercial service to label photos in real-time by combining an image recognition engine with crowds of workers. This service tried to expand the scope from a pure accessibility application to a more general augmented reality application by using general purpose image recognition technologies. In spite of such technical advances such as gamification, OCR integration, authoring tools, and image recognition integration, sustainability still remains an important issue to be addressed by crowd accessibility services. Bigham et al. [START_REF] Bigham | The design of human-powered access technology[END_REF] reviewed the history of crowd accessibility services, summarizing the features and characteristics of 15 services and proposing 13 design dimensions with which to compare the services, such as motivation, accuracy, and reliability. [START_REF] Benetech | Accessible books and periodicals for readers with print disabilities -Bookshareaccessible books for individuals with print disabilities[END_REF][START_REF] Valerie | Bookshare crosses 100,000-member milestone -Bookshare -accessible books for individuals with print disabilities[END_REF].

Methods to Analyze Participation

This section describes some methods to analyze participation in the crowd services from two perspectives, objective (such as statistics and log analysis) and subjective (such as interviews or surveys). Wikipedia is the largest service that is powered by a crowd of volunteers without monetary incentives, which has led to many analyses of its participants. According to an internal article titled "Wikipedians", there are about 16 million registered accounts and about 0.3 million of them edit Wikipedia articles on a monthly basis [START_REF]Wikipedia:Wikipedians -Wikipedia, the free encyclopedia[END_REF]. The number of edits per account ordered by magnitude follows a Zipf distribution, which forms a straight line on a double-logarithmic graph [START_REF] Nielsen | Zipf distribution (power law) of Website popularity[END_REF]. Swartz [START_REF] Swartz | Who writes Wikipedia?[END_REF] reported that about 73.4% of Wikipedia editing is done by the top 1,400 users, and most of the remaining editing is minor, such as fixing incorrect spellings. This phenomenon is also described as a 90-9-1 rule [START_REF] Nielsen | rule for participation inequality: lurkers vs. contributors in internet communities (Jakob Nielsen's Alertbox)[END_REF], since 1% of participants make almost all of the contributions, 9% make minor contributions, and the remaining 90%, the "lurkers" [START_REF] Nonnecke | Lurker demographics: counting the silent[END_REF], only read the results. Stewart et al. [START_REF] Stewart | Crowdsourcing participation inequality: a SCOUT model for the enterprise domain[END_REF] showed that the distribution is different in a closed service limited to company employees, reporting values of 33-66-1.

Objective study is key to gaining an overall picture of the participation in these services, though subjective studies reveal many of the psychological factors driving participation. Nonnecke et al. [START_REF] Nonnecke | Why lurkers lurk[END_REF] studied why lurkers lurk in crowd-sourcing services. Bryant et al. [START_REF] Bryant | Becoming Wikipedian: transformation of participation in a collaborative online encyclopedia[END_REF] interviewed active Wikipedians to analyze how they had evolved from lurkers to leading contributors of Wikipedia. A typical user initially visits Wikipedia to obtain information, then begins making minor contributions and learning about the rules and conventions of the community, and finally becomes reg-istered as a contributor. Their aim changes from just reading or polishing the articles to improving the quality of Wikipedia itself.

Evaluated Services

Over the past several years we deployed three crowd accessibility services, as summarized in Table 1. For each service, the following sections provide a summary of the service, characteristics of the supporters, task descriptions and definitions of the task units in the logged data, and methods for quality assurance.

EBIS

EBIS [START_REF] Ishihara | Transforming Japanese archives into accessible digital books[END_REF] was originally created to rapidly digitize physical books for the blind and other people with print disabilities, focusing on Japanese books that have up to 10,000 characters in various styles, which makes the process quite difficult compared to languages that use phonetic alphabets. The books are unbound and scanned to create images, and then processed through an OCR engine to generate the initial digital output. The raw output from the OCR engine is not accessible, since it tends to contain various character recognition errors and structural problems. There are several steps, but we focused on a crowd-sourcing process.

EBIS is using a check-by-expert approach as the quality assurance method. A skilled contributor was assigned as a proofreader to each book to identify any errors that might have escaped the eyes of the contributors in the earlier phases. This phase was managed manually and with mail-based information exchange. EBIS used gamification based on points and grades. According to contribution, each user earned some number of points and climbed the grade ladder based on the accumulated points.

The system was announced internally within IBM as part of the company's centennial 'celebration of service' event as one of the encouraged volunteer opportunities. This meant the contributors were unpaid volunteers recruited from a large pool of potential participants. CCES is a crowd-sourcing system that adds captions to digital videos [START_REF] Sobhi | Collaborative Caption Editing System -Enhancing the Quality of a Captioning and Editing System[END_REF]. Adding text captions to digital video tends to be a time consuming task, especially when the task work is being performed ondone for content for which no text transcript is available. The CCES service aims to speed up this process by splitting up the video into short segments, automatically segmenting it at detected breaks between phrases, and offering a user interface through which a captioner can type in the text as they listen to the clip. Each CCES task consists of a 30-second clip of video content that is to be transcribed and submitted by the captioner. The 30-second clip is actually split into roughly 10 or so sub-clips that approximate phrase utterances, and the supporter can review and transcribe them one at a time.

The expert check method was used as the quality assurance method in CCES. The system has a user interface to allow administrators to check and fine-tune captions. For example, administrators can adjust the timing, position on screen, length of one caption, and other attributes that are used for caption color coding, such as the gender of a speaker. The system was also announced internally within IBM as part of the company's centennial 'celebration of service' event. As of the writing of this paper, the CCES service has been running for about one month in its newest version.

Social Accessibility (SA)

The Social Accessibility [START_REF] Takagi | Social Accessibility: achieving accessibility through collaborative metadata authoring[END_REF] service was one of the first crowd accessibility projects we developed at our group. It provides a mechanism through which Web consumers such as blind computer users can identify accessibility problems with certain webpages, and then submit requests for improvements to a central server. The request is then made available to contributors who use a Web interface to create a fix for the accessibility problem by creating metadata that augments the original webpage, perhaps by adding alternative text to images. Since the metadata is stored on our server without altering the original website and the Web consumer can retrieve the metadata using our browser plugin, the service enables quick turnaround and wide reach.

The SA service was available to the public, with anyone able to sign up either as a requestor or a contributor. A requestor could submit a request by simply using a browser plugin and pressing a special hotkey when an inaccessible page was encountered. A contributor could view the pending requests with our online interface, and also view the webpage with the problem together with a panel for entering the various kinds of metadata information. A task in SA was a collection of metadata that a contributor submitted for one particular page in one session.

SA employed the user report as its main quality assurance mechanism. Each end user could report incorrect or suspicious errors whenever they used metadata within a seamless user interface with the same tool that applied the metadata to the webpages. Volunteers could also check and improve the quality of the metadata by using the tool that created the metadata, but this was also uncoordinated volunteer work.

Analysis Method

Given the diversity of crowd accessibility services, a standard analysis method itself is hard to develop. Therefore, we have developed an analysis method by focusing on the factors affecting the sustainability of crowd accessibility services. The method consists of two types of analysis to cover various aspects, such as transitions and gaps involving participation statuses (Transition Analysis).

The number of new contributors and the ratio who become committed contributors are the critical factors for assessing sustainable contributor participation. One major problem for each analysis method is the limited availability of data that applies across services. Each output of a comparative analysis method should be based on data that is available for all of the crowd accessibility services.

To determine specific challenges for the services, the gaps (or barriers) for participants should be assessed. To increase the number of new contributors, we should determine the gaps facing each new contributor. To increase the number of committed contributors, the gaps in returning should be studied. The transition analysis was designed as a tool to identify these gaps within services. The dynamics of participation can be considered as a series of transitions among participation statuses. Fig. 1 shows our definition of the transitions and Fig. 2 is a bar chart of a model of the statistical transitions among the statuses. The total number of people in all of the statuses refers to the total number of participants at the end of that day.

Visitors are people who visited the service, but did nothing. This is the initial state for a participant. If a person finds and visits the service, but decided not to contribute, then the person's status is "visitor". A registered visitor is a person who has registered with (or subscribed to) the service but has not yet contributed. This level of involvement is close to the definition of "lurkers" in social media (Section 2.2). In the case of a crowd accessibility service, people may not be simply lurking, but actively trying to figure out ways to contribute. Ideally, this status should be minimized. The number can be easily calculated from the contributor table in each service. Usually, the contributor table records the registration date as a property. This same data can be used to count the number of changes in the status of a registered user.

A new contributor is a person who is contributing to the service for the first time and an experienced contributor is someone who has contributed to the service at least twice. In Fig. 2, all of the contributors are new contributors on the first day, and after the second day there will be a mix of returnees (experienced contributors) and new contributors. These two types of contributors indicate different steps in the service. The gap between registered visitors and new contributors reflects the difficulties in starting to contribute after subscribing to the service. An iceberg chart visually shows the time changes of the ratio between registered visitors and newcomers (bottom-side dark gray and upper-side black areas). The gap can be also measured as a ratio:

Registration-contribution ratio = (total number of contributors) / (total number of registered visitors)

In contrast, the number of returnees (experienced contributors) is a metric to assess the degree of activity of committed contributors. By checking the changes over time in the numbers of new contributors and experienced contributors, we can visually assess the ratio of returnees. Thus, the returnee ratio from new contributors to experienced contributors can be seen as a metric to assess the difficulty of the transition to become a committed contributor.

Returnee ratio = ((total number of contributors) -(number of one-day contributors))/ (total number of contributors)

The definitions of new contributors and experienced contributors take into account the ease-of-calculation based on the properties of output data in a usual crowd accessibility service. For example, metadata is the output in SA, and each piece of metadata has properties including a link to its creator and its creation date. Any service with functions to track "who" worked on "which" task units will have such a data table.

An inactive contributor is a person who has not done any work on a given day. This definition means that the active contributor statuses (new and experienced) are temporary and that most of the contributors are in this inactive contributor status.

Task Analysis

The number of completed tasks in a unit of time is a clear quantitative metric for the activity level of a service. This task analysis focuses on measuring and visualizing the dynamics of completed tasks by new contributors and experienced contributors separately.

The definition of a task unit varies according to the target of a service. One transcribed line of text can be one task unit for captioning services, and one confirmed character can be one task unit for an OCR correction service. The definition can also be affected by the characteristics of the task management for the service. For example, when an OCR correction service manages correction work in units of "one page" then each page can be a task unit. A page can be an acceptable unit as long as the granularity of the task unit is sufficiently fine for the analysis.

Observations

We examined three of our crowd accessibility services using the analysis methods described in Section 3. In this section, we will present the analysis of the data for each service. The results will be summarized in the following section as a set of lessons learned.

EBIS

Fig. 4 shows the analysis results for EBIS. Fig. 4(b) shows several multi-day peaks in the completed tasks. The Within each of these periods, contributors successfully processed 10 to 25 books. The task shortage was strongly linked to the very success of the service. Contributors actively worked and completed all of the available tasks in a short period of time. As seen in the aggregate task count chart in Fig. 4(b), the EBIS service had other periods when there were no tasks being performed by the contributors. The charts show that work typically occurred in spurts spanning two to four days. The contributor data shows that majority of the contributors worked for no more than three days, roughly corresponding to the average duration of a spurt. The contributor status charts in Fig. 4(a) also show that when the second spurt occurred around November 8, 2011, almost all the contributors were new first-time contributors, meaning that almost none of the previous contributors returned.

The first spike on November 8 was created by an announcement to all employees on the front page of a portal. The small peak on November 17 was created without any announcement, but just by people who discovered the availability of the service. That is why the majority of those contributors were returnees, as shown in Fig. 4(b). The last peak was created by an email announcement to experienced contributors about the availability of new books. In this way, the service successfully engaged with the experienced contributors and they completed the tasks much more quickly than the administrators could prepare additional books. The underlying cause of the task shortage was the bottleneck due to expert phases in the process. The availability of books was limited by the performance of the experts, and this meant the contributors had to wait until each batch of processed books was proofread before being able to continue with the next batch. This can also be seen in Fig. 4(b), showing that when a new batch of books became available on November 8, 2011 (after a 2-month hiatus), almost all of the tasks were completed by new contributors and few of the previous contributors returned.

We received various informative comments as feedback from the contributors. One category of comments was about the effectiveness of the gamification feature that allowed the contributors to climb up a grade ladder by completing more tasks (Section 3.1). Some of contributors mentioned that the indication of the remaining task units to go up a level was highly motivating. Not only because of the explicit gamification feature, but many contributors mentioned the task itself gave them an impression of a "game" and that helped them to concentrate on the tasks. Many contributors mentioned that the user interface was similar to the "brain training games" on mobile game consoles (such as the Nintendo DS).

CCES

Fig. 5(a) shows the contributor status chart for CCES. It can be seen that throughout the duration of the service, there was a relatively steady flow of contributors performing tasks each day, both new and experienced contributors. A survey was conducted to get feedback from contributors. Among all of the contributors, 92.2% of the contributors answered good (51.0%) or very good (41.2%). The other options are 5.9% neutral, 0% bad and 0% very bad with 2.0% no answer. This high ratio of acceptance backs up the result of the steady flow of contributors.

The dips around November 26, 2011 and December 4, 2011 coincided with weekends, reflecting the fact that most contributors performed the tasks on weekdays. The graph also shows that each day there were at least some new contributors joining to perform tasks (indicated by the black bars). Such a participation pattern indicates a successfully operating service, with contributors' task output remaining relatively steady and new contributors steadily joining the service. Fig. 5(b) shows the same tendency as Fig. 5(a). New and experienced contributors share the tasks in a balanced way. The direct interpretation of the observed data is that the service was not sticky enough to encourage contributors to complete many tasks, even for experienced contributors. Instead, many new people joined the service and each quickly completed a small number of tasks. The phenomenon was not statistically significant and we clearly need more data to study such differences.

Social Accessibility (SA)

Two key characteristics of the SA service that distinguish it from the other services we have deployed were the length of time the service was available and the contributor pool. As seen in Fig. 6(b), the SA service continued to observe contributor activity for well over six months. When the SA service was launched, it was announced not only within Japan but also globally, resulting in nearly 700 registered users during the deployment period (Table 2). Among the three crowd accessibility services we deployed, SA had the largest number of contributors (98), with the largest number of contributors who contributed for more than two days (52).

The SA service was also able to stimulate a surge in contributor activity when an improved user interface was announced on November 27, 2008 (Fig. 6(a)). The surge was mainly due to experienced contributors who returned to the service as shown in While the SA service continued to yield contributions for over six months, the conversion from registered users to contributing users was relatively low compared with the other services, 14% (Table 2). From the contributor data, a majority of contributors contributed less than 10 days total. This suggests that the prolonged continuation of contributions may in large part have been due to a very small number of high- ly dedicated and productive contributors. This view seems to be supported by the contributor status chart (Fig. 6(a)) which shows that for the last two-thirds of the service period, the average number of contributors per day was between one and three. The contributor chart also shows that during this period while the contributor count remained low, the number of registered visitors was continuing to increase. What this suggests is that new people were registering for the service, but were not converting into contributors. This represents missed opportunities, and may point to some limitations of the service that were turning away new contributors or inhibiting them from taking the first step.

6

Lessons learned

Importance of Task and User Interface Design

One of success factors we found was the ease of completing tasks. One example is in CCES. One of the reasons for the high registration-contribution ratio (56%, Table 2) compared with other services may be due to the easy to understand task and productive user interface. The task goal of "captioning" may be familiar for people from seeing captioned TV programs. In addition, the interface makes it easy for a contributor to repeatedly listen to a short clip of the video and to type in the text, and provides convenient support for pausing and briefly rewinding the playback as the user types without requiring more explicit control by the user.

EBIS seems to have a similar situation. The registration-contribution ratio is lower than CCES but some of that is probably due to the periodic task shortages resulting from the task analysis (Fig. 4(b)). The success of the design can be seen from the rapid and large quantity of task completions, as mentioned in Section 5.1. Among the concerns expressed by the contributors, one of their highest priorities was "a rapid response from the server". This highlights the importance of having a streamlined user interface and a responsive system to fully benefit from the potential of large numbers of human participants. In SA, several factors might cause the low registration-contribution ratio (14%, Table 2), such as shortages of user requests, a side effect of its being a public service (while the other services are all private), and the long term of its deployment. Another factor may be the steeper learning curve compared with CCES and EBIS, since the metadata required by the SA service was quite specific to the Web accessibility domain, and good results depended upon some knowledge of concepts such as heading navigation, alternative text, etc. Among the tasks in the services, character correction (EBIS) and captioning (CCES) are relatively simple tasks and do not require unusual expertise, which made the design of their tasks and their user interfaces easier than SA.

Necessity of Task Shortage Management

The importance of the management of tasks, especially for handling task shortages, was one of most important lessons we learned. SA used a request-answer concept as the basic crowd-sourcing model, but that means the model depends on the number of requests arriving from people with disabilities. That made the system especially vulnerable to task shortages.

EBIS experienced a task shortage (Section 5.1), but the cause was different from the SA situation. The process for quality assessment (Section 3.1) became the bottleneck of this service, and resulted in task shortages. The long gaps between short spurts of contributions were due to the fact that the contributors would quickly finish processing a batch of books in a period of a few days, but the next batch could not be made available to the contributors until the post-processing had been completed for those books. This post-processing step involved a single proofreader who carefully read through the entire book. Such proofreading can only be performed by skilled contributors when high quality is a concern, and this step could not be eliminated because the partner organizations wanted high-quality results. The gap between the crowd-handled work (with EBIS) and the proofreading was large. There are other alternative methods for such quality assurance phases, but no perfect alternative exists. Majority voting is a popular method but it sometimes requires large number of contributors if the quality criteria are high.

In contrast, CCES was managed in a better way. Right from the launch of the service, the service maintainers carefully monitored the consumption rate of the content by the contributors. For instance, during the first few days after deployment, the maintainers noticed that English content was receiving relatively little attention, and decided to augment the input data feed with more Japanese content during the first content update on the first weekend (November 18). The task count on the following day (November 19) more than doubled, apparently due to the increased availability of new content.

In summary, the lessons we learned fit into four areas: (1) tasks should be designed not to cause task shortages, (2) bottleneck phases should be minimized, (3) task shortages should be carefully monitored, and (4) methods for responding to task shortages should be predefined.

Engagement with Continuous Contributors

We learned that the top contributors played an important role across the services. In CCES and SA, they contributed to the services over many days, and achieved some of the highest average task completions per day. While it may not be prudent to depend on the emergence of such top contributors in every service, it is better for service owners to prepare for and take advantage of such contributors when they do appear. Identification is the first step to working with top contributors. The next step is to reach out to these top contributors. The service owners can derive various insights from the top contributors for improving their service. It may be possible to receive quite specific feedback regarding the service from the perspective of an experienced contributor that could be used to help improve the service for the rest of the contributors. They may also have developed various methods or strategies for approaching the tasks that may benefit other contributors as well.

For example, during the deployment of the SA service, we were able to meet with one of the top contributors by chance. We were able to extract various insights from him. He was much older than we had expected, actually one of the early baby boomers. He was a retired office worker, had lots of spare time and was trying to find something meaningful for his life. SA fit well with his current objectives. His motivations for participating in the service were very personal. The service owner team could clearly understand a role model contributor.

7

Discussion of Future Research Directions

Mechanisms to Propagate the Sense of Contributions

One of the most committed contributors to SA was neither an engineer nor a Web professional, but he had passion and time for volunteer work (Section 6.3). Since SA was a request-based service, he commented that he could directly understand the users' problems and the value of his work. His comments suggest the importance of the special sense of altruistic contribution as an incentive in accessibility services. Crowd accessibility services are characterized by the existence of human beneficiaries, the people with disabilities who are always behind the system. Because of this characteristic, we learned that a personal sense of contribution to those who need help can be an important incentive factor. In the services we studied, CCES notified the contributors when a video was posted with their captions, EBIS notified them when a book they had worked on was uploaded to the digital library, and SA had a function to notify a contributor "when someone used the metadata you created" by using a blinking icon [START_REF] Takagi | Collaborative Web accessibility improvement: challenges and possibilities[END_REF]. One of the topics we have not explored was the cultural effects of the propagation of the sense of contributions. Most of the contributors for these three services were Japanese, and therefore cultural difference among countries could not be examined in this study. We suspect that the sense of altruism may differ among cultures, so further research activities in this area would be important.

Mechanisms for Accessibility Skill Development

Accessibility improvements often require specific skills for contribution. SA required basic knowledge of Web accessibility, such as appropriate alternative texts according to contexts and appropriate heading levels, and so the task familiarity for contributors was low (Section 6.1). The tasks for CCES and EBIS were more familiar than the SA work but still required specific knowledge that affected the output quality. For example, the notation of sound effects in captions, and techniques to find appropriate kanji characters from character tables. These tasks require knowledge and skill to extend them to environments for people with disabilities. Section 6.1 discussed the importance of easy-to-use interfaces and easy-to-understand tasks, but systematic mechanisms to develop "accessibility skills" in contributors offer another big area to explore.

7.3

Challenges of User-side Service-quality Assurance

One of the specific challenges for crowd accessibility services is the difficulty of quality assessment on the user side. SA relied on user reporting as its most important quality assurance method (Section 3.3). User reporting is a popular method used by many social services in such forms as the "illegal content report" of most video sharing services. However, for crowd accessibility services this can be difficult to implement. For SA the quality of an alternative text can be assessed by some straightforward criteria (such as spelling mistakes, lack of clarity, etc.), but this assessment is not possible for many kinds of problems. For example, if a malicious volunteer added an alternative text such as "Cancel" to an OK button, it would be extremely difficult to detect what was wrong by pushing the button. The end users need support because of their limited cognitive abilities, and that also limits their ability to assess the quality. This is one of the characteristic challenges in some types of crowd accessibility services.

Influence of Unintended Fun Factors for Increasing Engagement

We saw cases in which the unintended fun factor contributed to motivate contributors in serious accessibility services. In the CCES, the video clips were extracted from longer and sometimes enjoyable videos about substantive topics. From the comments in the survey, contributors found themselves becoming interested in the content itself as they performed the transcription tasks. A similar unintended fun factor also worked in EBIS, since in the final check the contributors read through an entire book and they could enjoy it. We found that the sense of contribution was the primary motivation for the contributors to accessibility services, but the unintended fun factor strengthened engagement with each service, thus increasing the participation. Compared to the intentional fun factors like gamification, unintended fun can easily be overlooked. The influence of unintended fun for serious accessibility services is worth investigating in future research.

Possibilities for Senior Citizens as Contributors

Another implication is the importance of encouraging potential contributors. For the leading contributor mentioned earlier, we had not expected such prominent involvement by a senior citizen. This anecdotal evidence suggests that senior citizens may be especially good candidates as volunteers for crowd accessibility services. Based on this experience, we studied several crowd-sourcing tasks including accessibility work for seniors ranging in age from 60 to 80. Half of them answered that they would be willing to do such tasks "without any fee", in contrast to a smaller percentage of younger participants who agreed to the no-fee model.

7.6

Possibilities for Internal Organizational Crowd-sourcing Two out of the three services were deployed within enterprise environments. (See Section 3 and Table 1.) These environments were not open to the public, so they can be regarded as constrained environments. However, our experiments showed some advantages in deploying crowd applications within an enterprise. The first advantage is the potentially large number of contributors in a large corporation, making the deployment similar to a general public environment, but with better control. We can announce, maintain, and analyze the results by using same framework as used for public services. Second, contributors within a corporation have a strong rationale not to act maliciously or to submit low-quality work in their enterprise environment partly because of their lack of anonymity. Such activities can be tracked, which is a strong incentive for the contributors to strive for high quality work. The third point is that services will be more sustainable due to the official support from the corporation as part of its CSR (Corporate Social Responsibility) efforts.

Conclusion

This paper discussed some lessons learned through an analysis of three crowdaccessibility services. After reviewing the related work, we developed visualization techniques that reflect the status changes of the contributors and of the completed tasks. The lessons learned were considered in the observation section by highlighting four topics, (1) design of the tasks and the user interface, (2) task shortage management, (3) contributor engagements, and (4) data collection and monitoring methods. Beyond these lessons learned, challenges and future research directions were discussed. The sense of contribution can be an important incentive especially for accessibility services. Better mechanisms for developing accessibility skills are imperative to improve the quality of the services. User side quality assurance poses unique challenges for accessibility services. The influence of unintended fun factor is worth investigation to increase engagement with the services. Also, senior citizens and corporate workers can be untapped new contributing resources.

Crowd accessibility has a huge potential to improve many situations for people with disabilities by combining human intelligence and computer intelligence. We are studying various ways to make the approach more sustainable in our society. A standard analysis method is itself challenging and methods should be evolved to unveil more specific problems within services. We hope that this paper will contribute to improving crowd accessibility strategies toward the level of more reliable and sustainable services.

Fig. 1 .Fig. 2 .

 12 Fig. 1. Transition Diagram among Participation Status

Fig. 4 (Fig. 4 (

 44 Fig. 4(a). Contributor Status Chart for EBIS (2011/7/24 ~ 2011/12/14)

Fig. 5 (

 5 Fig. 5(a). Contributor Status Chart for CCES (2011/11/11 ~ 2011/12/28)

Fig. 6 (Fig. 6 (

 66 Fig. 6(a). Contributor Status Chart for SA (2008/7/8 ~ 2010/3/29)

Table 1 .

 1 Evaluated Services

		Service type	Primary target	Start date	Service	Scope	Quality
				users		duration	assur-
							ance
	EBIS	OCR correction Blind and other	2011/8/23	2011/12/15	Internal Expert
				print disabilities			check
	CCES	Captioning	Deaf and hard of	2011/11/11	2011/12/28	Internal Expert
				hearing			check
	Social Ac-	Web accessibil-	Blind and people	2008/7/8	2010/3/31	External User
	cessibility	ity	metadata	with visual im-			report
		authoring	pairments

CCES (Collaborative Captioning Editing System)

Table 2 .

 2 Contributor Status Transition Table

	Registered	Registration-	Contribu-	Returnee	Experienced
	visitors		contribution	tors (at least	ratio	contributors
			ratio	once)
	EBIS	340	21%	71	41%	29
	CCES	209	56%	116	52%	60
	Social Accessibility	690	14%	98	53%	52