Arto Puikkonen

Anu Lehtiö

Antti Virolainen
email: antti.virolainen@nokia.comanu.lehtio@helsinki.fi

You Can Wear It, But Do They Want to Share It or Stare at It?

Keywords: Performative Wearable Devices, Social Interaction, Game Spectatorship

de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Rapid and continuous miniaturization of electronics and the fast evolution of new materials (e.g. nano materials) bring interesting possibilities for the use of wearable technologies. But despite that fact that wearable solutions prior have mainly dwelled in designs that aid us in our lives, the purchase of clothing is most often based on other factors than the level of aid received. Human actions are performative by nature and we are performing to the people surrounding us, to the spectators of everyday life [START_REF] Rico | Designing for Performative Interactions in Public Spaces[END_REF]. Similarly, we choose the clothes we wear based on our own preferences, but also to either avoid or to catch the eye of our spectators. Wearable technologies are an interesting means to provide more elaborate and lively ways to support these needs.

We wanted to test a wearable concept that would support the wearers' desires of expressing self, but would also be entertaining for the spectators and maybe even entice social interaction. For the purpose, we designed a piece of clothing that was visually appealing, appeared to be interactive to enhance the spectator interest towards it and had a social component in the form a game. Our design was aimed to be contrary to many other wearable technologies. We aimed heavily on the matter of visual appeal, instead of technological advances.

We chose to design a t-shirt that portrays an on-going game of Tic-Tac-Toe. The shirt was equipped with nine LED displays including a controller unit to control the displays. The purpose was to build the prototype in a way that the person wearing the shirt would not have to worry about the technology but wear it as any normal shirt. For the spectators the form of the T-shirt was supposed to look like a normal T-shirt.

To prevent the shirt becoming rigid and uncomfortable to wear, custom led displays were made. Discrete LEDs were assembled on a flexible substrate instead of using traditional rigid printed wiring boards. Wiring between the displays and a microcontroller was done with conductive strings. A normal T-shirt was used to hide the technology and give a nice finished look and feel while keeping the displays and wirings safe and sound between the two layers of textile (Fig. 1). Pre-programmed tictac-toe game software was stored on the memory of the micro controller and games were run continuously during the test period.

We formed our main research question based on the notion of spectator behavior: Does bringing this type of wearable technology into the context of peoples' everyday environment somehow overstep the social boundaries established for inappropriate attention? To this end we dug deeper into the subjective cognizance of the participants: How did they feel about wearing this type of wearable technology in their everyday environment and their descriptions and interpretations of the reactions and responses of people they came across while wearing the shirt.

Related Work

The key concept contributing to our study from social sciences was 'civil inattention' introduced by Erving Goffman [START_REF] Goffman | The Presentation of Self in Everyday Life Garden City[END_REF]. According to Goffman, social interaction is based on performative actions that take on dramaturgical forms. Bearing that in mind, civil inattention refers to a certain type of behavior that aims at maintaining some privacy for the performer in surroundings that are otherwise public. This behavior consists of e.g. showing disinterest, appearing to be absent-minded and/ or acting oblivious as to what's happening. This type of behavior can typically be seen e.g. in urban setting where a lot of people inhabit a limited space. You politely "pretend" not to or make sure that you don't pay too much attention to what others are doing, saying etc.

In the context of games civil inattention is often not beneficial. Quite the opposite, immediate interaction around gaming forms a substantial part of the total experience. When features for social interaction around games are enabled, playing games is more fun and engaging [START_REF] Gajadhar | Shared funn is doubled fun: Player enjoyment as a function of social setting[END_REF][START_REF] Gajadhar | Rules of Engagement: Influence of Co-Player Presence on Player Involvement in Digital Games[END_REF]. Additionally, Game cafes have been reported to garner their success from supporting conversation around games [START_REF] Jonsson | A Public Space of their own. A Fieldstudy of a Game Café as a Third Place[END_REF]. Friendships are built and maintained around gaming.

But interaction with the ones around you is not just about socializing. Reeves et al [START_REF] Reeves | Designing the spectator experience[END_REF] have presented a taxonomy and resulting design principles for designing for the spectator. According to Reeves, interfaces should be approached from thinking them as secretive, expressive, magical or suspenseful. Each approach comes with its own benefits and concerns, but aims to create the best spectator experience. Around gaming, it has been reported that spectators most appreciate games designed to support visible actions and tactics, units in competitive play and emotions evoked during competition [START_REF] Cheung | Starcraft from the Stands[END_REF].

In addition to the multitude of technological problems related to wearable computing, social acceptability forms a noteworthy barrier. The bottom line being, as described by Rantanen et al [START_REF] Rantanen | Smart Clothing for the Arctic Environment[END_REF], in addition to the technological features provided by a piece of clothing, it also has to provide the esthetical and functional properties one expects from clothing in general. Several issues related to wearable computing can be in opposition to these required properties, such as bulk [START_REF] Dunne | Smart Systems': Wearable Integration of Intelligent Technology[END_REF] and dangling objects [START_REF] Holleis | Evaluating capacitive touch input on clothes[END_REF]. With wearable controllers even the placing of the controls has been reported to have an impact on the social acceptability. Holleis et al [START_REF] Holleis | Evaluating capacitive touch input on clothes[END_REF] found in their study that controllers on trousers, wristbands or separate bags received acceptance. On the contrary, the upper body, like a shirt or a scarf, was generally seen as not acceptable. Interestingly, Gabaret et al [START_REF] Garabet | Exploring Design through Wearable Computing Art(ifacts)[END_REF] found out that control has an effect on the social acceptability. The less control the wearer appeared to have over a wearable device, the more acceptable the device seemed to be to the ones around you. This notion was based on designing and comparing three different performative and artistic wearable computing devices.

The most interesting performative wearable devices in regards to our study are T-Qualizer [START_REF][END_REF] and BubbleBadge [START_REF] Falk | The BubbleBadge: A Wearable Public Display[END_REF]. The T-Qualizer displays a music-related working equalizer on a t-shirt. The BubbleBadge is a wearable display designed to look like a brooch. As a brooch, it is meant to be placed on your chest, thus supporting face-to-face interaction.

User Study

To answer our research question, we planned a field trial with 10 participants. The age of the participants varied between 19-33, with 6 male and 4 female participants. Since we were interested in social acceptability issues that might arise if entertaining, wearable technology was introduced to workaday environment, we thought it best to actually bring the technology to users. Therefore the study was conducted at a university campus with both students (6) and staff [START_REF] Gajadhar | Rules of Engagement: Influence of Co-Player Presence on Player Involvement in Digital Games[END_REF]. All participants were recruited on the spot. Each of the participants was asked to wear our Tic-Tac-Toe T-shirt for a period of approximately one hour. Participants were encouraged to go about their daily duties as usual, not to let this experiment be in the way of their normal routines and not to avoid any of the customary situations that might occur. The length of one hour was chosen based on the rhythm of life at the campus. Participants were recruited during their breaks and the test period ended during the expected next break. Thus, the hour long cycle consisted of going somewhere, staying somewhere and coming back from somewhere. These places, called somewhere in this case, were classrooms, auditoriums, office spaces, cafes and lunch restaurants. A normal day at the campus constitutes of many of these cycles, and extending the experiment would have only resulted in multiple similar cycles. Also, due to the design of our t-shirt, more prolonged experiments were expected to make the t-shirt mundane in a relatively short period.

To get an idea on what was going on while the participants wore the t-shirt, we applied a light version of ethnographic methodology based on a form of participant observation [14]. To his aim, 5/10 participants were shadowed. With shadowing the focus was on getting insight on the reactions of both the "performer" and the "spectator". In this case, the performer is the person wearing the t-shirt and the spectator anyone in the vicinity of the t-shirt. To get this insight, a moderator followed the user for the whole time the t-shirt was worn and took notes. Both the performer's impressions and comments and the observed reactions of spectators were documented. 5/10 users were shadowed. The rest were expected to spend most of their time in areas that we couldn't access due to drawing too much attention to ourselves. Dividing the users into two groups enabled us to evaluate, at least on a superficial level, if the shadowing had had any impact on the performers' answers to the questionnaire. Naturally, during the shadowing the moderator tried to be as inconspicuous as possible.

During shadowing, the comments of the users were written down as well as the reactions and the comments from spectators. Since the users went about their business as they usually would and the moderator aimed not to be intrusive, we weren't able to get all the comments made.

Participants weren't explicitly encouraged or told to draw attention to the shirt. In the short brief we gave about the study, we informed the participants that the t-shirt was "a display" for an actual ongoing tic-tac-toe game played on the Internet. The participant would hence basically act as a display for it. The participants were given a mobile phone to give a more authentic feel of an actual online game. We told the participants that the mobile phone was downloading the game from the Internet. We chose to fabricate the story of an actual online game to support the feeling of actually being interactive and as the T-shirt's game data in reality could benefit of being drawn from the Internet.

After returning the t-shirt to us, or when the shadowing moderator ended the session, the participants were asked to fill out a questionnaire. In a situation, where someone acted as a spectator for a longer period of time, the spectator was also asked to fill out a form that was especially drawn up for this type of a situation. The questionnaires consisted of both rating scales and open-ended questions about performer experience, spectator awareness, acceptability, interest towards the technology and areas of improvement. In addition, the participants gave rich verbal commentary after the test session. These comments gave us knowledge that weren't present in the filled out questionnaire. In total, the combination of shadowing, questionnaires and verbal commenting gave us a thorough understanding of the results.

Results

The participants wore the t-shirt in varied locations, thus meeting a varied amount of possible spectators and varied level of direct interaction with others. Based on our observations the t-shirt was worn in the corridors, small classrooms, a large auditorium, small office rooms, open office spaces with cubicles and in cafes. The most common reactions the users met with while wearing the shirt were looks, (hidden) stares, smiles, laughter, and "raised eyebrows". The users didn't find this attention in the least disturbing. These overall reactions evoked by the shirt were quite subtle in nature and according to the users even surprisingly so. A comment made by one of the participants serves as an example on this: "I was surprised, because it felt like people were paying less attention to me than usual." (User 7, Female) These notions by participants seem to suggest the presence of Goffman's [START_REF] Goffman | The Presentation of Self in Everyday Life Garden City[END_REF] 'civil inattention', possibly even in an amplified form. It's socially acceptable to e.g. glance at a person you're passing by in a corridor. However, based on the observations, the glances by passers-by were very discrete and sometimes even missing altogether. Consequently, the observations made during shadowing support these partcipants' notions of somehow "being overlook".

The avoiding of eye contact, looking through someone etc. are typical forms of civil inattention. And this was the case in many situations. Our observations noted glances from a distance, but a lack of glances from shorter distances. It seemed that the spectators noticed the performer and that "something was up" from afar. So when they got closer, they were careful not to stare etc. In the context of civil inattention this could be interpreted as a sign of trying to assure that no embarrassment to one self or the performer would follow.

All the participants (6/10) that showed the shirt to someone did so only to people they already knew. The most common reasons for showing the shirt to someone were: the users thought the shirt was fun and/or wanted to see the reactions of their friends/co-workers/fellow students. Also, by drawing attention to the shirt themselves, the users reported feeling more in control of the situation. The users, who didn't show the shirt to anyone explicitly, explained this being due to not seeing anyone they knew. Since the users were instructed to go about their business as usual, a few of them reported it being useless or meaningless e.g. "to sit in one's cubicle where there are no people around." (User 1, Female). This creates one limitation to the people's willingness to wear the shirt to places, where interaction around the shirt might not naturally occur. If there's no one to show the shirt to or no one to take a look at it, there seems to be little point in wearing it at all. This issue didn't come up with participants, who wore the shirt for their lunch or coffee breaks.

When compared to the findings with BubbleBadge [START_REF] Falk | The BubbleBadge: A Wearable Public Display[END_REF], where the most fun was had by the spectators, it is interesting to realize the impact of civil inattention. It was clear from the users' responses that when attention was not received, the biggest factor for fun came from sharing. This sharing most arose in the form of showing the shirt to friends, who are natural recipients for sharing. Additionally, as no active or forthright attention or actions exhibited by strangers were reported, it would seem that the social interaction between the spectator and performer was greatly influenced by social closeness instead of just the actual setting. During shadowing it was also observed that social aspects of hierarchy might play a part in the matter. "Why did I turn on my heels? That's a good question, I didn't even think about it. Those are my bosses and if they saw with this shirt on I'm sure they'd start making jokes or something." (User 7, Female)

When the shirt was shown to somebody, the spectators were mainly interested in finding out how the shirt works. As the shirt was designed to be somewhat magical, based on the taxonomy of Reeves et al. [START_REF] Reeves | Designing the spectator experience[END_REF], this question was to be expected. When the social situation allowed the question to be asked, the spectators naturally wanted to understand "the magic". The spectators also asked what does it do and why the participant was wearing it. Two out of ten participants reported that someone had expressed interest in doing something with the shirt themselves -they either wanted to be able to play the game or wear the shirt. 8/10 participants wanted to be able to have some control over what's happening on and with the shirt. Most common concern was that the possible content of the shirt might be something the participant would find meaningless or being opposed to personal taste or values. "I'd like to decide what I'm displaying, yes. I wouldn't want to e.g. advertise McDonalds or a band I really don't like." (User 9, Male) Also the ability to turn the shirt on and off was mentioned. The tic-tac-toe game currently displayed on the shirt was seen as ok, but almost all of the users suspected that it wouldn't engage spectators for long. However, this wasn't seen as a big problem by the participants since the users didn't expect anyone to look at the shirt "for hours" anyway (User 10, female).

When choosing to utilize Tic-Tac-Toe in the design, we acknowledged that one design would not perfectly suit all participants. Naturally, when choosing a piece of clothing to wear, the clothing is often chosen based on the most appealing design. Our decision to use Tic-Tac-Toe was influenced by wanting an interactive design that still could be about self-expression and be appealing. Based on the feedback from the participants, the idea behind the idea was appealing, but not the current design of it. The participants stated that in an everyday situation the design should be more refined. When evaluating the current appeal, the shirt received an average of 2,4/5. On the other hand, the participants reported rather a high willingness (avg. 4,1/5) to wear the shirt again, based on how it suited their social surroundings. Some users even expressed enjoying the possibility "to be a little on view" (User 3, female) as a motivation for wearing the shirt. These results combined confirm similar findings made by Rantanen [START_REF] Rantanen | Smart Clothing for the Arctic Environment[END_REF]. That is, in addition to the appealing technological features, wearable technology has to provide the esthetical and functional properties one expects of clothing in general.

Even though the average for the willingness to wear the shirt suggests that the everyday surroundings didn't present a problem for wearing the shirt, the answers to open-ended questions offer a corrective to this. The users listed suitable situations and places for using the shirt as follows: bar, party, for opening up a conversation, a night out, concert and nightclub. Work, business meeting and formal occasions were on the top of the list of the most inappropriate places to wear the shirt. On the other hand, for some participants the shirt "gave on excuse" to be on view. A few spectators also brought up this matter, combining it with what they termed to be the personality of the user. "It's ok I guess, but for X (User 6) to be wearing that… it's not really his style, his thing, if you know what I mean? It's not his personality."

When talking about the idea of the t-shirt, the participants also stated understanding that the idea of being a display on the move requires that the information presented has to be short. The users suggested that in addition to games (Worms, Tetris, Super Mario, Chess) the shirt could display e.g. weather forecasts, advertisements, news and videos. Using the internet for content retrieving was seen as a good idea and facilitating many varied options for the content displayed.

Conclusions

In this paper we have presented a T-shirt that portrays an on-going game of Tic-Tac-Toe. With the help of the T-shirt, our goal was to understand the impact of civil inattention with wearable performative clothing. For this goal, we conducted a user study of 10 participants, who each wore the T-shirt for an hour in their everyday surroundings. To fully understand the different aspects of the matter, we dug deeper into the subjective cognizance of the participants with shadowing, questionnaires and discussions.

Our results indicate that civil inattention exists when using performative clothing. Our participants reported noticing some glances, but at the same time, noticing also an interesting lack of glances. Similarly our observations based on shadowing, support this notion. To compensate the lack of attention to something that the participants felt excited about, the participants chose to share this experience with their friends. The participants also suggested acceptability towards utilizing wearable technology as a performative and eye-catching element. Our results were also in alignment with previous studies showing that wearable technologies need to provide aesthetic and functional properties, just like clothing in general. Based on our relatively small sample data, we cannot make deeper claims of the reception of such technologies, but we feel that this is a good starting point for further investigation related to social acceptability. And even the fact of agreeing to wear the shirt in the populated surroundings, where the users inhabit and work, can be considered to be at least a weak sign of acceptance for introducing wearable, personal technology for such surroundings. Since the study was the first done with our T-shirt, it acts primarily as an explorative study for testing and mapping out the social acceptability issues. The results highlight the acceptability issues that arise when introducing a new technology into the mix. To this end, the study setup provided excellent results. On the other hand, the results are mainly relevant to the introduction, not long-term use. To understand the long-term issues, more research needs to be done.

For future work, we aim to support and study features for richer interaction and enhanced spectator experience.

Fig. 1 .

 1 Fig. 1. Tic-Tac-Toe T-shirt in use displaying an on-going game.


Susumu Harada

Jacob O Wobbrock
email: wobbrock@u.washington.edu

James A Landay
email: landay@cs.washington.edu

Voice Games: Investigation Into the Use of Non-speech Voice Input for Making Computer Games More Accessible

Keywords: Computer games, accessible games, speech recognition, nonspeech vocalization

We conducted a quantitative experiment to determine the performance characteristics of non-speech vocalization for discrete input generation in comparison to existing speech and keyboard input methods. The results from the study validated our hypothesis that non-speech voice input can offer significantly faster discrete input compared to a speech-based input method by as much as 50%. Based on this and other promising results from the study, we built a prototype system called the Voice Game Controller that augments traditional speech-based input methods with non-speech voice input methods to make computer games originally designed for the keyboard and mouse playable using voice only. Our preliminary evaluation of the prototype indicates that the Voice Game Controller greatly expands the scope of computer games that can be played hands-free using just voice, to include games that were difficult or impractical to play using previous speech-based methods.

Introduction

Computer games today offer much more than just entertainment. They can provide social as well as educational value, and new genres of games such as "games with a purpose" (GWAP) [START_REF] Ahn | Designing games with a purpose[END_REF] are emerging that combine gaming with productive output. However, almost all of these games are designed to be played using some form of manual input device, typically a keyboard for generating discrete input and a pointing device such as a mouse for generating continuous input. Consequently, people with various forms of motor impairments who find it difficult or impossible to use manual input devices are excluded from reaping the benefits of such games. Is there a way to make these games accessible to people with motor impairments?

There are a number of compelling reasons to explore how to make computer games more accessible. First, hands-free access to computer games can benefit a wide range of people, including nearly 300,000 children under the age of 18 in the U.S. alone (or 1 in 250 children) who have been diagnosed with arthritis or other rheumatologic conditions [START_REF] Sacks | Prevalence of and annual ambulatory health care visits for pediatric arthritis and other rheumatologic conditions in the United States in 2001-2004[END_REF]. Also, for people with more severe forms of motor impairment, computer games and game-like social applications (such as SecondLife [START_REF]Second Life Official Site[END_REF]) may be one of the few viable avenues for entertainment and social interaction, both of which play an important role in improving quality of life [START_REF] Riemer-Reiss | Factors associated with assistive technology discontinuance among individuals with disabilities[END_REF]. Second, research into effective hands-free game input can also spur future designs of novel multimodal games. Third, an increased understanding of how to effectively control computer games hands-free can also inform how to better design general user interfaces for interaction using nonmanual input modalities.

Strengths and Limitations of Speech Input for Game Control

Speech input is one of the hands-free input modalities that holds potential for addressing the above challenges. Unlike eye trackers or head trackers, a speech recognition system does not require elaborate or expensive hardware, and recent operating systems-such as versions of Microsoft Windows since Vista-include speech recognition software for free. Furthermore, the human language enables virtually a limitless number of words and phrases to be uttered, each of which can be mapped to a discrete command to be executed on a computer. This can be beneficial for controlling a number of computer games that require numerous keystroke combinations to be executed. Also, the ability to utter natural language commands compared to the need to remember and recall arbitrary keystroke combinations can be especially beneficial for novice users and may lead to a boost in performance [START_REF] Tse | Multimodal multiplayer tabletop gaming[END_REF].

However, speech input also has a number of inherent limitations that are preventing it from being an effective hands-free game control modality. First, the time it takes a person to complete uttering a word or a phrase can be a significant factor in games that require sub-second timing. The processing time required to recognize the spoken utterance also adds to this delay. Related to this is the limitation on the maximum number of utterances that can be recognized within a short period of time. When the need arises for dynamically issuing multiple input events at a rapid and varying rate, the per-utterance delay imposes a limit on how many such utterances can be recognized within the short time span.

Another major limitation of speech-based input is its inability to specify smoothly varying input. Output from a speech recognizer is discrete in nature, as the output is not generated until a word or a phrase has been recognized. The result of a single recognition is a single event, with the recognized utterance as the parameter. This discrete nature makes it significantly challenging to specify continuous input such as smooth motion of a pointer through two-dimensional space.

The current limitations of speech-based game control methods can be summarized in the context of the space of input signals expected by computer games. As mentioned earlier, most existing computer games are designed to be played using devices that can provide some combination of discrete input signals-such as a keyboard-and continuous input signals-such as a mouse or a joystick. However, they vary widely in the degree of temporal and spatial fidelity demanded from each type of input signals. For discrete input signals, they can range from simply requiring execution of a few non-time-critical commands, to demanding precise and rapid execution of multiple and possibly simultaneous commands. For continuous input signals, they can range from simply requiring non-time-critical selection of a point, to demanding fast and fluid steering. "Strategy games" such as card games and puzzle games typically do not require fast or precisely-timed inputs in either of these two dimensions. On the other hand, "skill-and-action" games such as most arcade-based games often require both discrete and continuous input signals to be executed rapidly and fluidly. Existing speech-based game control methods are currently limited to offering hands-free input signals within the subset of this input signal space defined by the lower ends of the discrete and continuous input signal dimensions.

Potential of Non-speech Voice Input for Game Control

Fortunately, our previous work and work by others have shown that such limitations of traditional speech-based input methods, especially their inability to provide fluid continuous input, can be addressed effectively by augmenting them with the use of non-speech voice input [START_REF] Harada | The Vocal Joystick: evaluation of voice-based cursor control techniques for assistive technology[END_REF][START_REF] Harada | VoiceDraw: a hands-free voice-driven drawing application for people with motor impairments[END_REF][START_REF] Igarashi | Voice as sound: using non-verbal voice input for interactive control[END_REF][START_REF] De Mauro | Easy access to graphical interfaces by Voice Mouse[END_REF][START_REF] Sporka | Whistling User Interface (U3I). User-Centered Interaction Paradigms for Universal Access in the Information Society[END_REF]. In particular, the Vocal Joystick engine [START_REF] Malkin | The Vocal Joystick Engine v1.0. Computer Speech & Language[END_REF] can continuously track non-speech vocal properties including volume, pitch, and vowel quality to enable fluid interface manipulations such as smooth mouse pointer control. A longitudinal study involving participants with and without motor impairments found that people can learn the directional vowel mapping after a few hours of training, and even approach the performance of physical joysticks for pointing tasks [START_REF] Harada | Longitudinal study of people learning to use continuous voice-based cursor control[END_REF]. The Vocal Joystick engine has also been successfully integrated into applications such as a hands-free drawing program called VoiceDraw [START_REF] Harada | VoiceDraw: a hands-free voice-driven drawing application for people with motor impairments[END_REF], demonstrating its expressivity and controllability in the context of continuous steering tasks. In a sense, these systems fill a part of the void left by current speech-based methods in the computer game input signal space mentioned above, namely the far end along the continuous input dimension

We felt that the expressivity of the Vocal Joystick engine could also be harnessed to expand voice modality's coverage of the game input signal space along the discrete input signal dimension as well, by enabling faster, more responsive discrete input than speech-based methods. We hypothesized that by using the Vocal Joystick engine to generate discrete instead of continuous input events in response to directional vowel vocalizations, we could achieve significantly faster input speeds as compared to using spoken commands. Such a method would also nicely complement the continuous input mode already offered by the Vocal Joystick engine, as the same directional vowel sound mapping can be used, maintaining consistency and likely enhancing learnability. The combination of these two methods, along with traditional speechbased input methods, could greatly expand the coverage of hands-free voice-based computer game input methods.

Towards this end, we conducted a quantitative experiment to determine the performance characteristics of non-speech vocalization for discrete input generation in contrast to existing speech and keyboard input methods using an application we built called the Vocal D-Pad. While it was expected that non-speech voice input will be faster than a speech-based input method, the results from the study showed that non-speech voice input is significantly faster, with indications that it may further approach the performance of keyboard input with additional training. Based on these promising results from the study, we built a prototype system called the Voice Game Controller, which combines the rapid discrete input capability of the Vocal D-Pad, the fluid continuous input capability of the original Vocal Joystick application, and the natural language input capability of traditional speech-based input. Our preliminary evaluation of the prototype indicates that the Voice Game Controller greatly expands the scope of computer games that can be played hands-free using just voice, to include games that were difficult or impractical to play using previous speech-based methods.

Related Work

There is little prior work on general voice-driven input techniques for computer games. Most of what exists can be classified as either tools that map speech commands to keystroke combinations, or individual games specifically designed to use voice input in their own way within the game.

Current Speech-Based Game Controls

Speech input in games has primarily been used for communication among multiple players [START_REF] Gibbs | Proximity-based chat in a first person shooter: using a novel voice communication system for online play[END_REF][START_REF] Wadley | Towards a framework for designing speech-based player interaction in multiplayer online games[END_REF] or for command-and-control-style applications [START_REF] Tse | Multimodal multiplayer tabletop gaming[END_REF]. There have also been some tools developed to enable a player to use their voice to control various aspects of computer games. Most common are tools that translate spoken commands into specific actions, usually in the form of keystroke combinations. Tools such as Shoot [15], Voice Buddy [START_REF]Voice Buddy Interactive Voice Control Version 3.0[END_REF], and VR Commander [START_REF]VR Commander -Voice Command and Control for you Computer[END_REF] have been specifically marketed for voice control of games, and map spoken commands to a sequence of keystrokes. Such functionality can be quite useful in games such as flight simulators in which there are a large number of possible commands that can be executed at any time, often mapped to obscure key combinations. Other general "speech macro" tools such as Vocola [START_REF]Vocola -A Voice Command Language[END_REF] and WSR Macros [START_REF]Windows Speech Recognition Macros[END_REF] offer similar functionality, but do not include any game-specific functionality. Most of these tools act as keystroke generators and are often used in conjunction with a physical keyboard or a mouse as an augmentative input modality. One limitation of these tools is that they cannot be used on their own to play games that require continuous input from pointing devices such as the mouse or joystick.

Aside from using speech for dictation or command-and-control, today's commercial speech recognizers also enable the execution of pointing tasks using speech. Features such as the MouseGrid and SpeechCursor [START_REF] Harada | The Vocal Joystick: evaluation of voice-based cursor control techniques[END_REF] offer the ability to control the mouse pointer to a limited extent. Although these features do enable basic pointing and clicking tasks to be performed using speech, they take significantly longer than the mouse [START_REF] Harada | The Vocal Joystick: evaluation of voice-based cursor control techniques[END_REF], and their discrete nature makes them unsuitable for any task that requires rapid pointing or smooth steering such as games.

Games Designed to be Controlled by Voice

There have been a number of games designed specifically to be controlled using the player's voice. Most of them use relatively simple vocal features such as pitch and volume. Commercial games such as Karaoke Revolution [21] and Rock Band [START_REF]Rock Band®[END_REF] track the pitch of the players' singing and generate a score based on how close they match the target pitch track. PAH! [START_REF]PAH! -The first voice-controlled and activated game on the web[END_REF] and shout n dodge [24] are Flash-based sidescrolling shooter games that continuously map the volume of the microphone input to the vertical position of a spaceship. PAH! also enables the user to shoot by making the plosive sound "pah." Racing Pitch [START_REF]Racing Pitch[END_REF] uses the pitch of the audio input to control the speed of a slot car, with higher pitch vocalization causing the car to travel faster. Igarashi [START_REF] Igarashi | Voice as sound: using non-verbal voice input for interactive control[END_REF] presents a set of simple games that demonstrate the use of pitch changes and sound detection to control simple movements of the game character. His games involve making rising or falling pitch sounds to move a character up or down, varying the duration of the vocalization to control how far a character moves, and making plosive sounds to shoot. In Humming Tetris, Sporka et al. [START_REF] Sporka | Non-speech input and speech recognition for real-time control of computer games[END_REF] map various pitch inflection "gestures" to various controls such as move left, move right, and rotate.

While these games were designed from the onset specifically to be controlled by voice, and while we cannot expect many of the existing or future computer games to be redesigned in such a way, they provide insight into the types of non-speech vocal properties that can be used to map to existing game inputs.

The Vocal Joystick Engine

As mentioned in the introduction, one of the promising developments in the area of voice-based input has been the Vocal Joystick engine [START_REF] Malkin | The Vocal Joystick Engine v1.0. Computer Speech & Language[END_REF], which not only tracks the continuous vocal parameters such as volume and pitch, but also vowel quality. The Vocal Joystick engine samples audio input from the microphone every ten milliseconds, and for every frame, if the incoming sound is a vowel sound, reports the recognition probability for each of the nine vowel classes shown in Figure 1. These probabilities are then aggregated based on the radial arrangement of the vowel classes to yield an overall continuous directional vector, which can be used to steer a mouse pointer. The Vocal Joystick engine library [START_REF] Malkin | The Vocal Joystick Engine v1.0. Computer Speech & Language[END_REF] provides an API through which third- party application developers can gain access to such information in real-time. The engine can also recognize non-vowel sounds, or discrete sounds, such as "ck" and "ch." The original Vocal Joystick application enabled voice-driven control of the mouse pointer using the radial mapping of vowel sounds to directions as shown in Figure 1, as well as click and toggle of the left mouse button using the discrete sounds "ck" and "ch," respectively. The volume was also mapped to the pointer speed, enabling the user to control both the pointer movement direction and speed fluidly and interactively. Studies have shown the Vocal Joystick to be effective for such basic mouse pointing and steering tasks [START_REF] Harada | The Vocal Joystick: evaluation of voice-based cursor control techniques for assistive technology[END_REF][START_REF] Harada | Longitudinal study of people learning to use continuous voice-based cursor control[END_REF].

Due to the extremely short latency for vowel sound recognition compared to spoken word recognition, and the directional nature of the vowel sound mappings in the Vocal Joystick, we felt that non-speech voice-based discrete input can also offer significant advantages over speech-based methods. The following section describes the experiment we conducted to test this hypothesis.

Experimental Analysis of Non-speech Voice-based Discrete Input

To better understand how non-speech voice input compares to keyboard and speech input as a method for generating discrete input signals, particularly under contexts requiring rapid and precisely-timed input, we conducted a reaction time (RT) experiment using an application we created called the Vocal D-Pad 1The experiment involved the presentation of directional visual stimuli and user responses generated via one of three input modalities: key (four arrow keys on a keyboard), speech (words "up," "down," "left," and "right"), and voice (directional vowel sounds based on the Vocal Joystick). The specific objectives and the setup of the experiment can be better understood in the context of the Model Human Processor [START_REF] Card | The Psychology of Human-Computer Interaction[END_REF], summarized next.

. The Vocal D-Pad is built using the Vocal Joystick engine library and generates emulated directional arrow-key events in response to recognized directional vowel sound utterances. The application can also emulate the corresponding directional keys being held down for the duration of the vocalization, as well as adjacent keys being held down simultaneously (e.g., the left and up keys being held down when the vowel sound for the upper-left direction is vocalized). The system also interfaces with the Microsoft Windows Speech Recognizer to enable mapping of spoken command input to emulated keyboard events, akin to the speech macro tools mentioned in the Related Work section.

Experiment Background

Figure 2 shows a typical processing flow during a reaction time task in terms of the Model Human Processor. The Model Human Processor is an abstraction of the human cognitive-perceptual-motor system that can be useful in analyzing basic computer task performance. Upon the onset of a stimulus in a reaction time task, the user's perceptual processor activates to process the relevant sensory input. The cognitive processor then deciphers the stimulus and determines the appropriate response. Finally, the motor processor converts the response action into motor signals and transmits them to the relevant motor systems such as the hand or the mouth. We also introduce a new module called the system processor, which is not in the original model since it is not part of the user. This represents any processing that the computer may need to perform on the input signal received from the user. The system processor does not add much time in cases such as button presses, but if the input is a spoken command, the system processor needs significantly more time to recognize what was uttered before it can provide its result as input to a client application. While the system processor time will be affected by the speed of the computer being used, the relative ranking of its value across different modalities should be unaffected.

Figure 2 also illustrates various measures that are of interest in a reaction time task. Reaction time is defined as the time between stimulus onset and response onset. Within the context of our experiment, response onset is defined as the earliest point when the system detects that some user response has been initiated (e.g., for a buttonpress response, when the button switch is closed, and for a voiced response, when the microphone first detects voicing activity). System time is the time between response onset and input onset-when the system finishes processing the response signal from an input device and generates a corresponding input signal for client applications. For keyboard input, the system time is a negligible value, but for speech input using a general recognizer, the value can be significant. We refer to the total elapsed time between stimulus onset and input onset as the input time.

There are two types of reaction time tasks that are commonly administered: simple reaction tasks and choice reaction tasks. In a simple reaction task, there is only one type of stimulus, and the user is instructed to issue a fixed response as soon as they perceive the stimulus. In a choice reaction task, the user is presented one of a number of possible stimuli, each with a corresponding response that must be correctly executed. The difference in the response times between these two tasks manifests itself in the cognitive processor time.

Hypotheses

Based on the model and representation presented above, we formulated the following hypotheses about the differences between keyboard, speech, and non-speech voice modalities with regards to their performance in reaction time tasks (summarized visually in Figure 3).

• Hypothesis #1: For trained users, the difference in reaction times across modalities is expected to be relatively small since the perceptual processor time and the cognitive processor time should be constant across modalities.

In particular, reaction time for voice and speech modalities should be nearly identical given that the same motor system is involved and thus the same amount of time should be spent in the motor processor. The reaction time for the key modality may be slightly faster than that for voice or speech given that the vocal cord activation may take longer to generate an audible sound compared to the movement of a finger. • Hypothesis #2: For a novice user under the choice reaction task using the voice modality, the unfamiliarity with the sound-to-direction mapping would most likely lead to a longer cognitive processor time and thus longer reaction time compared to the key or speech modality. • Hypothesis #3: For both the simple and choice reaction tasks, the system processor time for the key modality is expected to be significantly faster than that for the speech modality, and the system processor time for the voice modality to lie somewhere in between, since non-speech vocalization should take less time to utter and be recognized compared to a speech command given its relative simplicity. While we established the above three hypotheses in order to gain better insight into the nature of the differences among the three modalities' input times, we were also interested in the overall magnitude of those differences. We expected the key modality to be faster than the voice modality, and the voice modality to be faster than the speech modality, but the pragmatic question was, by how much, particularly in the context of computer games?

Participants

Eight participants ranging in age from 21 to 34 were recruited to take part in the experiment. Two of the participants had motor impairments that affected the use of their hands for manipulating a keyboard and mouse (MI group), and the rest did not (NMI group). All eight participants had participated in prior Vocal Joystick user studies and had used the Vocal Joystick for an average of four hours each and had thus been familiarized with the directional vowel mappings. One of the MI participants (MI1) who had arthrogryposis multiplex congenita was unable to use the hands to manipulate a mouse or keyboard, but was able to use a mouth stick to press the keys on a keyboard. The other MI participant (MI2) had muscular dystrophy and was able to use the back of her fingers to press the keyboard keys.

Apparatus and Procedure

Each participant took part in a series of reaction time trials in which a test application presented them with a visual stimulus in the form of a blue arrow (100×100 pixels) on a computer screen (15" LCD monitor with resolution of 1400×900 pixels) pointing in one of the four cardinal directions. The participants were asked to respond to the stimulus in a manner dependent on a particular condition. Under all conditions, the test application waited for the correct key-down event and measured the elapsed time. The two independent factors and their levels were:

• Input modality: {key, voice, speech} • Reaction task type : {simple, choice} The key modality involved pressing one of the four arrow keys on a full-sized USB desktop keyboard. The voice modality involved uttering one of the four cardinal vowel sounds in the Vocal Joystick vowel compass (Figure 1), which the Vocal D-Pad processed to generate an emulated key-down event. Finally, the speech modality involved the utterance of a direction word ("up," "down," "left," or "right"), also processed by the Vocal D-Pad in a similar fashion. The Windows Speech Recognizer's general dictation grammar was disabled and only the grammar consisting of the four directional words was activated to minimize the error rate and maximize the processing speed. For both the voice and speech modalities, each participant underwent a basic adaptation process to tune the system to their voice. Under the voice modality, the user vocalized each of the four vowel sounds for two seconds while the Vocal Joystick engine recorded them and updated its acoustic model. Under the speech modality, the user read several paragraphs of passage as part of the Windows Speech Recognizer voice training wizard process.

Under the simple reaction task, the stimulus was always an up arrow, and the participant was instructed to respond as quickly as possible using a fixed response (up key for the key modality, the upward vowel sound for the voice modality, and the word "up" for the speech modality). The time between the end of one trial and the appearance of the next stimulus was randomized between one and two seconds (this range of pre-stimulus interval was found in past studies to yield the fastest reaction times while reducing predictability [START_REF] Izdebski | Effects of prestimulus interval on phonation initiation reaction times[END_REF][START_REF] Shipp | Physiologic stages of vocal reaction time[END_REF]). Under the choice reaction task, the participant was asked to generate a response matching the arrow stimulus as quickly and as accurately as possible. In this mode, the delay between trials was one second. For all trials, both the reaction time and system time as defined in Figure 2 were measured. In the case of the key modality, the response onset and input onset were both considered to occur simultaneously when the key-down event was generated; therefore, its system time was always zero. In the case of both the voice and speech modality, the response onset was registered when the first voiced audio frame was detected by the Vocal Joystick engine, and input onset occurred when the corresponding emulated key-down event was registered.

Results

Aggregate results from the NMI group are presented first, followed by results from the MI participants. Figure 4 summarizes the results from the NMI group. The results are grouped first by the task type, with each bar corresponding to an input modality showing the reaction time and system time portions that made up the total input time. Henceforth, results reported as significant are at p < .05 level.

Differences in Reaction Times

For the simple reaction task, reaction times for voice and speech modalities were comparable as expected, since they both involve the same muscle groups. Reaction time for the key modality was significantly shorter as expected [START_REF] Nebes | Vocal versus manual response as a determinant of age difference in simple reaction time[END_REF][START_REF] Venables | Reaction times to auditory and visual stimulation in schizophrenic and normal subjects[END_REF]. This is most likely due to the vocal cord activation taking longer to generate an audible sound compared to the time it takes for the finger muscle activation to result in a key being pressed. This confirms our hypothesis #1. The average difference in these reaction times (176 milliseconds) can be interpreted as the approximate difference in the motor processor times between the manual and vocal modalities Under the choice reaction task, voice modality reaction time was slower than that for speech (782 milliseconds versus 649 milliseconds). This could be seen as supporting hypothesis #2, since even though the participants have had four hours of exposure to the Vocal Joystick, their association of the vowel sounds with their corresponding directions may not yet be automatic. The fact that this difference did not manifest itself in the simple reaction task indicates that the slowdown is most likely due to the extra cognitive processing required under the choice reaction task. Our prior work [START_REF] Harada | Longitudinal study of people learning to use continuous voice-based cursor control[END_REF] has shown that users can fully memorize the directional vowel mappings in under 5 hours, but it may require more practice for recall of directional vowels to become as quick and as automatic as directional words. This suggests that with practice, voice modality reaction time under the choice reaction task could approach that of the speech modality. Figure 4 depicts this potential improvement on the choice reaction task input time bar.

Differences in System Times

The comparison of system times across task types within each modality verifies that the system processor time did not depend on the task type, as expected. The significantly shorter system time for the voice modality compared to the speech modality supports our hypothesis #3, and reflects the major advantage of the use of non-speech vocal input. While it has not been precisely determined how much of this difference is due to response time or processing time, it is most likely the case that the majority of it is accounted for by the difference in processing time.

Differences in Input Times

The comparison of the total input times reveals that overall, the key modality was the fastest (as expected), at 301 milliseconds for the simple reaction task and 433 milliseconds for the choice reaction task. However, the input time for the voice modality was also significantly faster than speech, by almost 50% in the simple reaction task and by 35% in the choice reaction task. The latter difference can be improved to 45% if the difference in cognitive processor time mentioned above is subtracted. While the voice modality input time still lags behind the key modality, that difference is significantly less than the improvement over the speech modality.

Results from MI Participants

The results from the two participants with motor impairments are summarized in Figure 5. For MI1, under the choice reaction task, reaction times across all three modalities were comparable (average of 1,050 ms after the learning effect adjustment described above). The fact that the key modality reaction time was comparable to voice and speech modalities is likely due to the difficulty of moving the mouth stick quickly in response to the stimulus. Also, under the choice reaction task, the voice modality input time was comparable to the key modality and almost 40% faster than the speech modality. Results for MI2 were similar as MI1, with voice modality input time under the choice reaction task (average of 780 ms after the learning effect adjustment) being only 25% slower than the key modality and almost 45% faster than the speech modality. These results offer a promising sign for the comparative advantage of the voice modality over the key modality, especially for these participants with motor impairments.

Another interesting observation is the fact that the voice and speech input times for both MI1 and MI2 were longer than the average time for the NMI group. Our observations seem to suggest that this may be due to the effect of the MI participants' motor impairments on their lung capacity and their ability to rapidly vary vocal parameters. Further study is needed to determine the variability of such effect among individuals and how much of it can be accounted for via training.

Summary of Results

Figure 6 illustrates the results from the above study in a concrete example context of the game of Pac-Man. In the figure, the player's Pac-Man character (in yellow) is assumed to be at a standstill at a juncture, with the choice of moving up or down to move away from the chasing ghost character (in pink). The ghost character is approaching Pac-Man at a fixed speed, as indicated by the distance labeled "1 second." The multiple positions of the ghost character with corresponding input modality labels represent the closest point from Pac-Man that the ghost can be allowed to get before it is too late for the player to decide on the direction to move, execute the corresponding input, and have it be registered by the game in time to avoid contact with the ghost. In other words, these distances are the visual representations of the total input times under the choice reaction task shown in Figure 4. The position labeled "voice expert" shows the shortened distance that could be achieved if the reaction time for the voice modality is assumed to have become equal to that for the speech modality as described above. The figure highlights how significant an improvement the non-speech voice-based discrete input can offer over speech-based input, especially in the context of fast-paced time-critical computer games such as Pac-Man. The experimental analysis of non-speech voice-based discrete input yielded promising results over the speech-based input method. To leverage this potential and to situate it within the context of actual computer games, we built a system called the Voice Game Controller that extends the capability of traditional speech-based input by incorporating the rapid discrete input capability of the Vocal D-Pad and the fluid continuous input of the Vocal Joystick. The Voice Game Controller augments, rather than replaces, current speech-based input methods by retaining the option to use the dictation and command-and-control modes when non-time-critical speech input is appropriate.

The Voice Game Controller runs as a background process on the same computer as the one on which the game program is running. It translates the user's vocalizations into keyboard, mouse and joystick (KMJ) signals that game programs expect as input. Figure 7 shows a high level representation of the Voice Game Controller architecture. The Voice Game Controller utilizes the Windows Speech Recognizer to recognize spoken command input, and the Vocal Joystick engine [START_REF] Malkin | The Vocal Joystick Engine v1.0. Computer Speech & Language[END_REF] to process non-speech vocalizations uttered by the user. Since each game requires different sets of input controls, the mapping between vocal input and corresponding KMJ signals is specified separately for each game in Voice Game mapping profiles. The input mapping engine passes the recognized voice input through a Voice Game mapping profile corresponding to the currently active game program, and generates the corresponding KMJ signal via the keyboard, mouse, and joystick emulator modules. Because the emulated KMJ signals are generated at the system level, they are application-independent and thus can be used to supply voice-driven KMJ input to non-gaming applications as well. From the game program's perspective, it is as if the user is using a standard keyboard, mouse, or joystick. The keyboard emulator module can generate key-down and key-up events for any key or set of keys, as well as repeated key-down events to emulate the user holding down a key. The mouse and joystick emulators can generate two-dimensional directional vectors (when in relative mode) or positional vectors (when in absolute mode), as well as button down and button up events corresponding to physical buttons on these devices. All these types of signals can be specified as the output of a mapping in a Voice Game mapping profile.

There are several ways in which spoken command input and non-speech vocal input can be mapped to the KMJ signals just described. First, spoken commands and non-speech discrete sound vocalization can be mapped to any of the key events for the keyboard emulator or to the button events for the mouse and joystick emulators. Second, non-speech vowel vocalization can be mapped to two-dimensional vectors based on the Vocal Joystick vowel compass mapping shown in Figure 1, with its magnitude possibly determined by the volume or pitch. This capability alone expands the realm of games playable by voice to include the numerous mouse-driven games. Third, non-speech vowel vocalization can also be mapped to key and button events similar to spoken commands and discrete sounds. This is done by determining the vowel sound with the maximum probability every frame, instead of aggregating the probabilities across all vowels, and generating a binary output of whether or not that probability exceeds some threshold. If it does, it is treated as the vowel onset and a corresponding key or button event can be generated. The vowel offset event occurs for the current vowel when the vowel probability drops below the threshold, a different vowel's probability becomes higher, or the user stops vocalizing. The directional mapping of the Vocal Joystick vowel sounds make them ideal for mapping to the arrow keys in the context of games.

In the following section, we present our initial findings from a preliminary evaluation of the Voice Game Controller we conducted in order to assess its effectiveness in the context of four different games, comparing its performance to speech-driven control as well as to the participants' preferred input devices.

Preliminary Evaluation of the Voice Game Controller

Results from the comparative analysis of categorical input above showed that the voice input modality has the potential to offer significant performance gains over general speech input, approaching the performance of key input. To determine how this advantage translates to actual game play, we conducted a preliminary evaluation of the Voice Game Controller with the following four games to assess the viability of using the system to play real computer games.

Selected Games

We chose four games that lie at various points in the input signal space. Two Rooms [START_REF]Armor Games: Two Rooms[END_REF] is a puzzle-based maze game in which the player controls square pieces using directional arrow keys. Although the time to complete each puzzle is considered, once the sequence is known, the emphasis is on how consistently and reliably the desired directional input can be generated. Pac-Man is a classic arcade game in which the player controls the Pac-Man character through a maze using directional arrow keys, attempting to eat all the white dots while avoiding four ghost characters. The game requires precise timing to avoid contact with a ghost character or to make a turn. The player's score corresponds to the number of white dots and frightened ghosts Pac-Man consumes. Tetris is a puzzle game in which the player rotates and positions falling two-dimensional tiles using arrow keys such that the tiles form solid horizontal layers, which then disappear from the playing field, yielding points for the player. It is necessary to be able to execute a rapid sequence of left or right movements to quickly move a piece into the desired position. Finally, Fish Tales [START_REF] Tales | [END_REF] is a game in which the player controls a red fish in a two dimensional scene by using the mouse. The objective of the game is to eat fish that are smaller than the player's fish, while avoiding contact with larger fish. The score is based on the number of fish eaten. All of the fish move at varying speeds and directions, requiring the player to smoothly and rapidly steer the mouse pointer.

Evaluation Setup

Five of the eight participants who took part in the reaction time study, including the two participants with motor impairments (MI1 and MI2), participated in the evaluation of the Voice Game Controller. Each participant came in for four 90-minute sessions separated by 48 hours or less. Throughout the sessions, we observed the users play each of the four games using three input modalities: non-speech voice input mode, speech-only input mode, and the user's preferred input device. The speechonly input mode represents existing speech-based control methods, namely uttering directional words to emulate corresponding arrow key events and using features such as Speech Cursor and Mouse Grid for pointer control. For the preferred input device, the three participants in the NMI group chose the mouse and keyboard, while MI1 with arthrogryposis chose a mouth stick with the MouseKeys feature for controlling the pointer, and MI2 with muscular dystrophy chose the touchpad and keyboard.

During the first session, participants were shown how to use the Voice Game Controller and the speech-only input mode, and then introduced to the four games. For the remainder of the sessions, they played each of the games for 15 minutes at a time, spending 5 minutes on each of the three input modalities per game. At the end of the final session, a timed evaluation was conducted in which the participants played all games for five minutes each using each of the three input modalities. The speech input modality was omitted for the mouse-based Fish Tales game because the existing speech-based cursor control methods of Speech Cursor and Mouse Grid were too slow to be able to play the game at all. For the Two Rooms game, the measure of performance was based on the average completion time across all completed stages. The other three games were measured based on the average score achieved.

Results and Observations

Figure 8 summarizes the results from the evaluation described above. The data is presented in two graphs, with the graph on the right showing how the Voice Game Controller did compared to the speech modality, and the graph on the left showing how it did compared to the user's preferred device. The horizontal axes represent how much better or worse the Voice Game Controller did with respect to the corresponding modality, with ±0% representing no difference in performance, and positive values indicating that it did better by that much percentage points.

Compared to speech, the Voice Game Controller performed significantly better across all games for all participants (MI1's speech data was omitted due to incomplete tasks). For Fish Tales, the fact that the participants were able to play it at all using voice was a clear win over speech, with which the game could not be played. Within the NMI group, Voice Game Controller was 40% faster than speech in Two Rooms, and the average score was 50% higher in Pac-Man and 500% higher in Tetris. Within the MI group, Voice Game Controller was 32% faster than speech in Two Rooms, and the average score was 150% higher in Pac-Man and 160% higher in Tetris.

The better performance by the preferred input devices was expected, as was seen in previous comparative studies [START_REF] Harada | The Vocal Joystick: evaluation of voice-based cursor control techniques[END_REF]. Within the NMI group, the Voice Game Controller performance was 40% worse than the preferred device for Two Rooms, and between 60% and 65% worse for the other three games. For the MI group, the Voice Game Controller performance was comparable to their preferred devices for Two Rooms, and between 60% and 70% for the other three games. While the preferred device performed better than the Voice Game Controller even for the MI group, both of the MI participants expressed that the fatigue induced by attempting to manipulate their physical devices was less desirable than the hands-free nature of the Voice Game Controller. None of the participants raised vocal fatigue as an issue during the study.

Conclusion

This paper presented results from our investigation into the viability of using nonspeech voice-driven input for expanding the scope of computer games that can be controlled hands-free using voice only. Particularly, our experimental analysis into the use of non-speech vocalization for discrete input revealed a significant performance improvement over existing speech-based input method. Based on these findings, we built a prototype system called the Voice Game Controller that integrates the expressivity of traditional speech-based input with the fluid continuous input offered by the Vocal Joystick engine, as well as the non-speech voice-driven discrete input functionality of the Vocal D-Pad. The better performance obtained by the Voice Game Controller compared to standard speech-driven input in actual games suggests that voice input can become a viable modality for people with motor disabilities to play many of the mouse and keyboard-centric games that have previously been beyond reach for them. The expressiveness and hands-free nature of voice-driven input may also serve as a new game input modality for the general population, expanding the realm of interactive entertainment.

Figure 1 :

 1 Figure 1: The Vocal Joystick sound map showing the vocal features tracked by the Vocal Joystick engine. The vowel compass (left) shows the vowel sounds mapped to each radial direction. Red letters in each word approximate the corresponding sound. Discrete sounds such as "ck" and "ch" are also tracked, as well as changes in volume and pitch. The mapping to mouse pointer movements and events as used in the original Vocal Joystick application are shown

Figure 2 :

 2 Figure 2: Processing flow during a reaction time task based on the Model Human Processor [27] and the relevant measures. The system processor has been introduced to account for the time spent during response execution (response time) and recognition (processing time). At input onset, the recognition result is sent to the client application.

Figure 3 :

 3 Figure 3: Summary of the hypotheses of the experiment. Hypothesis #1 states that reaction times across modalities should be relatively similar, especially between voice and speech for trained users. Hypothesis #2 states that for a novice user, the voice modality may incur additional time to recall the unfamiliar sound-to-direction mapping. Hypothesis #3 states that the system time should be almost zero for the key modality, and that it should be significantly less for the voice modality compared to the speech modality.

Figure 4 :

 4 Figure 4: Summary of the aggregate results from the categorical input experiment for the NMI group. The results are grouped by task type. Each bar shows the contribution from reaction time and system time towards the total input time, as defined in Figure 2. Error bars show 95% confidence intervals for total input times.

Figure 5 :

 5 Figure 5: Summary of the results from the categorical input experiment for the two participants in the MI group, presented in the same format as Figure 4.

Figure 6 :

 6 Figure 6: An illustration of the implication of the result from the categorical input study in the context of a fast reaction time game Pac-Man. For each of the three input modalities the figure shows the closest point that the "ghost" can be allowed to get to Pac-Man where there is just enough time for the player to decide on the direction of escape (in this case up or down), execute the corresponding response using the modality, and have the response be registered by the system.

Figure 7 :

 7 Figure 7: Voice Game Controller architecture. User's utterances are first processed by the Windows Speech Recognizer and the Vocal Joystick engine, and the corresponding keyboard/mouse/joystick signals are emulated based on the currently active Voice Game mapping profile.

Figure 8 :

 8 Figure 8: Results from the Voice Game Controller user evaluation. The graph on the left shows how each participant scored in each game using the Voice Game Controller relative to their preferred device. The graph on the right shows the relative performance of the Voice Game Controller against speech-based input. ±0% indicates that the Voice Game Controller score was on par with the corresponding modality, and positive percentage indicates that the Voice Game Controller score was that much higher.

The name "D-Pad" comes from "directional pad," a game input device consisting of four directional buttons.

Acknowledgments. We would like to thank all the study participants who volunteered their time to try out our system. This work was supported in part by the National Science Foundation under grant IIS-0326382. Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. This work was also supported by the University of Washington Royalty Research Fund.