Susanne Bødker
email: bodker@cs.au.dk

Clemens Nylandsted Klokmose
email: clemens@cs.au.dk

From Persona to Techsona

Keywords: Persona, Techsona, Activity Theory

In this paper we introduce the notion of techsona, as a possible counterpart design instrument to personas. We use a case study to illustrate a design process where techsonas help pinpoint technological ideas and alternatives and analyze these systematically. While a persona captures "imagine a user…", a scenario "imagine a situation…" the techsona gives the missing piece of "imagine a technology…" We show how a recently developed activity theoretical model can serve both as a framing for techsonas and as an analytical interface between personas and techsonas. The paper discusses the potentials and problems of the techsona and concludes that the techsona truly helps with a missing piece in persona and scenario-based design.

Introduction

In our work with interaction design, we have used scenarios and personas in various forms over the years. The capacity, and possible and plausible actions of humans are captured and communicated through these. Hence, the use of personas and scenarios help us answer the question of how potential users reach out towards current or hypothetical future technologies. The reverse side, however, needs to be addressed similarly: How do the action possibilities of an artifact, whether a current or hypothetical artifact, reach out towards the potential user? We miss a set of design instruments, matching personas, in order to capture and communicate open issues regarding the artifacts designed, design decisions and even technological alternatives. In this paper we present techsonas to provide this match.

Personas are connected to scenarios in that they often work together, and share the same qualities of capturing both the current situation and human action possibilities on the one hand, and the future on the other [START_REF] Bødker | Scenarios -setting the stage for reflection and action in user-centered design[END_REF][START_REF] Bødker | Scenarios as springboards in design[END_REF]. Scenarios focus on particular situations, whether typical or critical, and they can generally be used throughout the design process to hold on to design decisions while being open for interpretation. It is our claim in this paper that we need the techsona to counterpart personas and scenarios in order to help users and designers reason about technology on a more abstract and hypothetical level than prototypes. We need means to capture and concretize technological alternatives and thought experiments without necessarily manifesting them as prototypes. Hence, we want to provide a light-weight yet systematic means for ad-dressing questions like: What if we used a wall-mounted multi-touch screen for our ticket vending machine, would that benefit or impede our users?

Accordingly, we need to populate the gap that we are addressing here with a tool that helps focus on the critical detail in terms of use of future artifacts, and acts as counterparts to personas at the levels from human fear and values to specific action possibilities and constraints. In addition this tool needs to capture elements of existing technologies that are useful when designing the future.

From personas and scenarios to techsonas and back again

Personas [START_REF] Chang | Personas: from theory to practices[END_REF][START_REF] Cooper | The Inmates are Running the Asylum[END_REF][START_REF] Friess | Personas and decision making in the design process: An ethnographic study[END_REF] are defined as ways of representing real people throughout the design process. According to Cooper [START_REF] Friess | Personas and decision making in the design process: An ethnographic study[END_REF], personas are hypothetical archetypes, defined with rigor and precision, discovered in the investigation process. Personas allow for an understanding of the values, fears, etc., of the users [START_REF] Friess | Personas and decision making in the design process: An ethnographic study[END_REF]. Personas are used in two ways: by their designers in their own design process and for communication with others [17]. In interaction design personas are often used in conjunction with scenarios. Scenarios are most well known from Carroll [START_REF] Carroll | Making Use: Scenario-Based Design of Human-Computer Interactions[END_REF]. He points out that scenarios are stories about people and their activities; that they are specific and fluid at the same time and they point towards both analyses of existing use/work activities and visioning of the future. Making scenarios is a creative process: they are hypotheses, or qualified guesses about the artifact and its use [START_REF] Bødker | Scenarios as springboards in design[END_REF]. They serve to open the dialogue about future possibilities and current constraints. [START_REF] Bødker | A conceptual toolbox for designing CSCW applications[END_REF][START_REF] Bødker | Scenarios as springboards in design[END_REF] propose to work from use-like situations towards scenarios that are constructions meant to stage acting in the future or to reflect on and illustrate problems with this action. Our use of personas and scenarios in design is described in [START_REF] Bødker | Preparing students for (inter-)action[END_REF].

While Carroll [START_REF] Carroll | Making Use: Scenario-Based Design of Human-Computer Interactions[END_REF] argues "…that use-scenarios can be the principle design representation of an artifact" (pp. 190) Bødker & Christiansen [START_REF] Bødker | Scenarios as springboards in design[END_REF] propose to capture and communicate innovative or prototypical examples of technologies, in a manner that seems quite parallel to personas. It is necessary in particular to enable a systematical exploration of new ideas, which is an integral part of any type of interaction design. Hence, we need a means for exploring an (even far-fetched) idea against personas and use scenarios in a lightweight yet systematic manner. Accordingly we see personas and techsonas as closely connected with scenario-based methods, and we have been inspired by previous work on future scenarios [START_REF] Bødker | Scenarios -setting the stage for reflection and action in user-centered design[END_REF] and inspiration cards [START_REF] Matthews | How do designers and user experience professionals actually perceive and use personas? CHI[END_REF].

The difference between personas, techsonas and scenarios can briefly be captured as follows. While a persona in design would easily be approached as "imagine a us-er…", a scenario as "imagine a situation…" the techsona gives the missing piece of "imagine a technology…". If focusing on a current use situation, personas summarize and crystalize findings regarding users' actions and motives, scenarios flesh out (specific) use situations, and techsonas summarize the action possibilities, or affordances of the technology. Like a persona, the techsona is hypothetical, yet rigorous and precise enough to be confronted with a variety of personas and scenarios.

Where personas are discovered through investigation, techsonas are invented as part of the design process. A good techsona embodies a simple (potential) design idea and provides enough description of its realization to confront the values, fears, capabilities, etc. of the users as expressed in the personas.

In the following we discuss what goes into a techsona and how can it be confronted with personas in a systematic manner. We use a model derived from activity theory that helps reasoning about the dialectics between human and technology.

A framework for mediation between personas and techsonas

Recently we have applied the Human-Artifact Model [START_REF] Bødker | The Human-Artifact Model: An Activity Theoretical Approach to Artifact Ecologies Human-Computer Interaction[END_REF] for design-oriented analyses of current and future artifacts and activities of use. In this context the model provides an emphasis on understanding interaction with technology as multi-layered and dialectical. The analytical scheme of the Human-Artifact Model (Figure 1) combines analyses of human experiences and artifacts, and addresses the tensions between human skills and capacity on the one hand, and the action possibilities and affordances offered by the artifact on the other [START_REF] Bødker | Scenarios -setting the stage for reflection and action in user-centered design[END_REF]. This is done on three levels reflecting the activity hierarchy: activity, action and operation. These levels provide three sets of analytical glasses: Motivation (by asking why?), goal-orientation (by asking what?) and operation (by asking how?).

Figure 1. The Human-Artifact Model (according to [7])

We have combined this approach to understanding the relationship between a practice and given artifacts with iteratively and concurrently developing personas [START_REF] Bødker | Preparing students for (inter-)action[END_REF]. Through this we confront the richness of personas with the more focused questions of "Why?", "What?" and "How?" coming from the Human-Artifact Model (Figure 1). In the following we illustrate how this is done similarly for the techsona. The model resembles Carroll et al's task-artifact cycle [10. However, while the task-artifact cycle reflects a causal relationship between the artifact and task, the dialectical relationship of the Human-Artifact Model is different: In practical terms, we need to analyze what happens and might happen to the human activity every time artifacts are changed and vice versa, but there no causal predictions available, such as 'because the artifact is changed in this way, the task will change in that'. Another difference between the Human-Artifact Model and the task-artifact model in terms of its focus on tasks, in short, is that tasks are mainly about actions, whereas the why level in particular is missing in the analysis. We refer to [START_REF] Bannon | Beyond the Interface, Encountering Artifacts in Use[END_REF] for a longer discussion of tasks.

Applying the Human-Artifact Model as a mediator between personas and techsonas emphasizes their dialectical relationship, and provides a structure for systematically exploring the tensions between the assumptions of the artifact and the capabilities and orientation of the user towards the artifact all the way from motivation to low-level operation.

Case: The Municipal E-service

The eGov+ project focused on studying and improving eServices for public authorities (see e.g. [START_REF] Bødker | Personas, people and participation-challenges from the trenches of local government[END_REF]). The main problem identified were that public authorities often see eServices as replacements of the ways in which citizens show up in the public office to receive e.g. a new passport or a driver's license. The project worked to identify alternative ways of designing and understanding citizen eServices and multiple student projects have been part of this.

Based on records of interviews and observations with citizens and municipal workers, the researchers developed a set of personas that the students were asked to use as starting points when carrying out an iterative design process (see more in [START_REF] Bødker | Preparing students for (inter-)action[END_REF]). Personas were used to summarize substantial research findings as communication devices within the project, from researchers/analysts to students/designers, similar to the ways personas is described in [START_REF] Chang | Personas: from theory to practices[END_REF][START_REF] Halskov | The emergence of ideas: the interplay between sources of inspiration and emerging design concepts[END_REF]. They were supplemented with scenarios to describe the current use situation as well as future use. Here is a condensed description of the persona Ahmed (from [START_REF] Bødker | Personas, people and participation-challenges from the trenches of local government[END_REF]):

"Ahmed is a citizen of the world. Yet, somewhat in contrast to this, he is motivated in his life by a wish to own his own home, to live in Denmark. He prefers face-to-face deals in general, and he does e.g. most of his shopping in this manner. He is heavily dominated by his concerns for his close family and their everyday life in Denmark (school, housing, etc.) and his wider family. This family is spread across the world and he makes contact with members whenever possible. Ahmed is used to visit the municipal office in question. He is used to cell phones whereas he does not use Internet banking. He travels with his family to meet relatives whenever possible."

Based on a more detailed description of Ahmed and five other personas, and a selection of scenarios [START_REF] Bødker | Preparing students for (inter-)action[END_REF] the students were asked to discuss technology suggestions, such as a passport machine coming from a newspaper article, to make rich descriptions of this and their own technological alternatives and relate them to the given personas-these rich descriptions are what we refer to as techsonas. A condensed example the techsona "friend-in-waiting": "The Friend-in-waiting allows citizens who come to the citizen services office to spend the time in the waiting area completing as much passport information as possible. It is used in a similar way as current paper forms, but has the advantages that existing information from municipal systems can be called up automatically. The Friend-in-Waiting is a pad that the front-desk staff hands to people when they enter the office. Hence the Friend-in-waiting does not require for citizens to bring their own device, if only they bring their social security card. The look-and-feel resemblance with paper forms combined with the use of touch displays is intended to help citizens who are otherwise not using computer technolo-gy. The Friend-in-Waiting is also a guide that leads citizens to a booth where pictures are taken and automatically linked to the ID of the citizen, and, if needed, to the citizen services staff responsible for handling the passport."

The students used The Human-Artifact Model as an interface between personas and techsonas, to analyze potential matches and mismatches between human beings and a technology as it was described, or tensions between persona and techsona. The Human-Artifact Model was also used to structure the development of a techsona, e.g. when the students were asked to develop a techsona based on the passport machine described in a newspaper (Figure 2). From there the students directed their attention towards the future and used the given personas personas to develop scenarios of use, and the Human-Artifact Model to structure their insights regarding the users and use situations. Since the goal was to inform design of a new technological solution used when acquiring a new passport, the personas were coded to focus on e.g. Ahmed's orientation towards a potential new artifact. The Human-Artifact Model supported this coding through its levels: Why would he use the artifact? What will he need to do? How is he used to achieve the goals the artifact realizes? Whether such artifact was useful was debatable, and up for the students to uncover.

Along the way, the students were asked to be skeptic towards their ideas and to provide alternatives. Hence the outcome of the process was four sets of alternatives illustrating four different sets of problems relating to a passport machine. These included the possible use of mobile technology while in the municipal office or away from it. One of these alternatives was the 'Friend-in-waiting'a tablet-based passport device to be used while in the waiting area of citizen services. This helped address a critical issue that had been uncovered in discussions of the passport machine, namely the match of the passport picture with the actual person, which, according to legislation, required physical presence at the office. But how would it help citizens like Ahmed? A techsona was created together with the video-based scenario and coded with the Human-Artifact Model for analysis towards the personas. This techsona was confronted with the personas and new issues occurred that were explored further through traditional means such as a video scenario (exemplified in Figure 3).

Friend'in'wai*ng,WHY

The$Friend*in*wai-ng$allows$ci-zens$who$come$to$the$ci-zen$services$office$ to$spend$the$-me$in$the$wai-ng$area$complete$as$much$ passport$informa-on$as$possible.$It$is$used$in$a$similar$way$as$ current$paper$forms,$but$has$the$advantages$that$exisng$informa-on$ from$municipal$systems$can$be$called$up$automa-cally.$The$Friend*in* Wai-ng$is$a$pad$that$the$front*desk$staff$hands$to$people$when$they$enter$ the$office.$Hence$the$Friend*in*wai-ng$does$not$require$for$ci-zens$to$bring$ their$own$device,$if$only$they$bring$their$social$security$card.$The$look*and* feel$resemblance$with$paper$forms$combined$with$the$use$of$touch$displays$ is$intended$to$help$ci-zens$who$are$otherwise$not$using$ computer$technology.$The$Friend*in*Waing$is$also$a$guide$that$leads$ ci-zens$to$a$booth$where$pictures$are$taken$and$automa-cally$linked$to$the$ ID$of$the$ci-zen,$and,$if$ needed,$to$the$ci-zen$services$staff$ responsible$for$handling$the$passport.

Friend'in'wai*ng,WHAT

The$Friend*in*wai-ng$allows$ci-zens$who$come$to$the$ci-zen$services$office$ to$spend$the$-me$in$the$wai-ng$area$complete$as$much$passport$ informa-on$as$possible.$It$is$used$in$a$similar$way$as$current$paper$forms,$ but$has$the$advantages$that$exis-ng$informa-on$from$municipal$ systems$can$be$called$up$automa-cally.$The$Friend*in*Waing$is$a$ pad$that$the$front*desk$staff$hands$to$people$when$they$enter$the$office.$ Hence$the$Friend*in*wai-ng$does$not$require$for$ci-zens$to$bring$their$own$ device,$if$only$they$bring$their$social$security$card.$The$look*and*feel$ resemblance$with$paper$forms$combined$with$the$use$of$touch$displays$is$ intended$to$help$ci-zens$who$are$otherwise$not$using$computer$technology.$

The$Friend*in*Wai-ng$is$also$a$guide$that$leads$ci-zens$to$a$ booth$where$pictures$are$taken$and$automa-cally$linked$to$ the$ID$of$the$ci-zen,"and,"if"needed,"to"the"cizen"services"staff" responsible"for"handling"the"passport.

Friend'in'wai*ng,HOW,',Handling

The$Friend*in*wai-ng$allows$ci-zens$who$come$to$the$ci-zen$services$office$ to$spend$the$-me$in$the$wai-ng$area$complete$as$much$passport$ informa-on$as$possible.$It$is$used$in$a$similar$way$as$current$paper$forms,$ but$has$the$advantages$that$exis-ng$informa-on$from$municipal$systems$ can$be$called$up$automa-cally.$The$ Friend*in*Wai-ng$is$a$pad$that$ the$front*desk$staff$hands$to$people$when$they$enter$the$ office.$Hence$the$Friend*in*waing$does$not$require$for$ci-zens$to$bring$ their$own$device,$if$only$they$bring$their$social$ security$card.$The$ look*and*feel$resemblance$with$paper$forms$combined$with$the$ use$of$touch$displays$is$intended$to$help$ci-zens$who$are$otherwise$not$ using$computer$technology.$The$Friend*in*Wai-ng$is$also$a$guide$that$leads$ ci-zens$to$a$booth$where$pictures$are$taken$and$automa-cally$linked$to$the$ ID$of$the$ci-zen,$and,$if$needed,$to$the$ci-zen$services$staff$responsible$for$ handling$the$passport.

Friend'in'wai*ng,HOW,',Adap*ve

The$Friend*in*wai-ng$allows$ci-zens$who$come$to$the$ci-zen$services$office$ to$spend$the$-me$in$the$wai-ng$area$complete$as$much$passport$ informa-on$as$possible.$It$is$used$in$a$similar$way$as$current$paper$forms,$ but$has$the$advantages$that$exis-ng$informa-on$from$municipal$systems$ can$be$called$up$automa-cally.$The$Friend*in*Wai-ng$is$a$pad$that$the$front* desk$staff$hands$to$people$when$they$enter$the$office.$Hence$the$Friend*in* wai-ng$does$not$require$for$ci-zens$to$bring$their$own$device,$if$only$they$ bring$their$social$ security$card.$The$look*and*feel$resemblance$with$ paper$forms$combined$with$the$ use$of$touch$displays$is$intended$to$ help$ci-zens$who$are$otherwise$not$using$computer$technology.$The$Friend* in*Wai-ng$is$also$a$guide$that$leads$ci-zens$to$a$booth$where$pictures$are$ taken$and$automa-cally$linked$to$the$ID$of$the$ci-zen,$and,$if$needed,$to$the$ ci-zen$services$staff$responsible$for$handling$the$passport.

TECHSONA:*Friend1in1wai4ng

The$Friend*in*wai-ng$allows$ci-zens$who$come$to$the$ci-zen$services$office$ to$spend$the$-me$in$the$wai-ng$area$complete$as$much$passport$ informa-on$as$possible.$It$is$used$in$a$similar$way$as$current$paper$forms,$ but$has$the$advantages$that$exis-ng$informa-on$from$municipal$systems$ can$be$called$up$automa-cally.$The$Friend*in*Wai-ng$is$a$pad$that$the$front* desk$staff$hands$to$people$when$they$enter$the$office.$Hence$the$Friend*in* wai-ng$does$not$require$for$ci-zens$to$bring$their$own$device,$if$only$they$ bring$their$social$security$card.$The$look*and*feel$resemblance$with$paper$ forms$combined$with$the$use$of$touch$displays$is$intended$to$help$ci-zens$ who$are$otherwise$not$using$computer$technology.$The$Friend*in*Wai-ng$is$ also$a$guide$that$leads$ci-zens$to$a$booth$where$pictures$are$taken$and$ automa-cally$linked$to$the$ID$of$the$ci-zen,$and,$if$needed,$to$the$ci-zen$ services$staff$responsible$for$handling$the$passport.

Artifact

Huma n

The meeting of persona and techsona

Underlying this work to establish and code the personas and techsonas lies the activity theoretical idea that the human users possess a set of learned and adapted action possibilities, and they are oriented towards certain goals and motives. All of these change throughout the human activity, and they are changeable when meeting and using a new artifact. A new artifact provides new possibilities for use, but it also talks back and cannot be used in any odd way the user may want. Similarly the user cannot necessarily make her capacities work with any new artifact.

Accordingly the meeting of the persona and the techsona is intended to help address the possibilities of a match between the human and the artifact side, as well as the problems. This is not an either/or, rather they are dialectical tensions: Where lie the possibilities that the Persona Ahmed (or one of the other personas) would use the 'Friend-in-Waiting', or the passport machine? What kinds of immediate problems do we find that may prevent such future use, or cause a need to adjust the concept before we move on to designing a more elaborate prototype? For instance, would Ahmed's preference for a personal handshake and face-to-face communication prevent him from picking up the 'Friend-in-Waiting'? Or is it sufficient that he meets a person when picking it up and leaving it? Would experience with modern multi-touch inter-faces on mobile devices help or hinder his handling of the pad? Or would the idea of picking up a device that is not his own be a barrier? etc. (Figure 3).

The sort of analysis illustrated in this example falls in a tradition that goes back to [START_REF] Bødker | A conceptual toolbox for designing CSCW applications[END_REF], where the person side and the technology side of activity are systematically confronted with each other through a number of different scenarios, based on systematic questions. The Human-Artifact Model has helped making it more operational in terms of helping students and other designers work in a structured manner with the levels and questions. What we are providing is an analytical tool that helps uncover potentials and problems of future use at an early stage, but obviously, ultimately, there are no firm answers, and this sort of analysis does not prevent problems from occurring in later prototypes or the final design.

Discussion & Conclusion

In this paper we have introduced techsonas to facilitate a focus on technological possibilities and alternatives in scenario-and persona-based design. We have used the Human-Artifact Model as a mediator between these. The above example illustrates how the analysis brings out critical details, such as in the example, the relationship between experiences with personal devices related to a shared tablet in the case of "friend-in-waiting". While we have limited evidence for this, we suggest that through this kind of analysis we may be able to recognize important details from less important ones, in techsonas, as well as in the personas.

In line with the original motivation for personas, the techsona makes it possible to address motivational issues together with the handling possibilities of the proposed technology. Hence, the techsona targets the role of proposed technology in relation to e.g. human fear and values, and the techsona in that manner is a way of filling the gap that we identified.

There may well be other ways of filling this gap, and we suspect that the Human-Artifact Model structured analysis is best suited for the design of IT, and less so for e.g. a media production where personas similarly are used to represent the target audience. Other theoretical frameworks may serve equally well as a mediator between techsonas on the one hand and scenarios and personas on the other hand. However, the activity theoretical rooting in dialectical thinking is in our opinion crucial to properly understand the relationship between personas and techsonas.

In our experiences with techsonas so far, we have seen indications that this analysis to some extent helps resisting the temptation of premature prototyping, i.e. the temptation to pursue and build a prototypes simply because it looks like a good idea, technically. The students who carefully used the techsonas seemed more critical to their own ideas. Obviously we have built our presentation around one, simplified example. Richer material and more cases are behind this presentation, and we are working to gather even more such material. In the meantime, we encourage the readers to start using and developing the ideas further.

Figure 2

 2 Figure 2 Creation of techsona through answering the questions of the Human-Artifact Model (The Passport Machine as read about in the newspaper)

Figure 3 .

 3 Figure 3. Human-Artifact Model as a mediator between persona and techsona

Acknowledgements

We thank members of the eGov+ project as well as Karen Johanne Kortbek, Lene Mejlby, Nikolaj G. Borchorst, Marianne Dammand Iversen, Peter Dalsgaard and Kim Halskov for their various contributions. This work has been supported by the Carlsberg Foundation and Aarhus University through the iCenter PIT.


Sheetal Agarwal
email: sheetaga@in.ibm.com

Jyoti Grover
email: jyogrove@in.ibm.com

Arun Kumar

Monia Puri

Meghna Singh
email: meghna.singh@in.ibm.com

Christian Remy
email: remy@ifi.uzh.ch

Visual Conversational Interfaces to Empower Low-literacy Users

Keywords: Diversity, HCI4D, Information Sharing, ICTD, User-Centered Design, Interactive Voice Systems, Smartphones, India

Mobile phones have come a long way from being plain voice calling devices to becoming multipurpose handy tools powered by ever increasing new applications available on-the-go. For many, the mobile phone of today has become the essential device one does not leave home without. However, for a large percentage of human population mobile phone apps are not of much use as they are not literate or IT savvy enough to be able to benefit from them. Recent advances in voice-based telecom information systems enable underprivileged and low-literacy users to access and offer online services without requiring expensive devices or specialized technical knowledge. However, voice applications are limited in their capability due to their time consuming nature. In this paper, we demonstrate an interaction modality that combines the power of voice communication with graphical interfaces in smartphones to break the barrier of illiteracy.

Introduction

Over the last couple of years, smartphones have proliferated at a large scale even in developing countries. Emergence of new local manufacturers has led to easy availability of smartphones in the market costing less than USD 100 [START_REF] Datta | New low-cost smartphone from Micromax[END_REF]. The plethora of mobile apps available today have brought applications in domains ranging from personal management to healthcare, collaboration to education, at users' fingertips. However, even though a lower upfront cost has helped extend the reach of smartphones, their utility still remains limited to voice calls for a large section of their users in developing countries. The primary reason for this is that the running cost of downloading and using several apps is high if an active internet connection is required. Furthermore, another barrier is the low level of literacy or English literacy that prevents many needy users from making use of their powerful handsets effectively.

On the other hand, advances in interaction experience of telephony voice user interfaces have seen good uptake by this underserved population in developing countries [START_REF] Agarwal | Content creation and dissemination by-and-for users in rural areas[END_REF][START_REF] Lau | Towards conversational interfaces to web applications[END_REF][START_REF] Sambasivan | Designing a Phone Broadcasting System for Urban Sex Workers in India[END_REF]. Traditionally, they have not had any access to online information systems due to reasons of affordability, local relevance, and illiteracy.

In this paper, we propose to marry voice user interfaces available on ordinary phones with the power of graphical user interfaces for Visual Conversational interfaces that achieve a two-pronged effect. First one is to enable textually illiterate people to harness the power of mobile apps available on their smartphone devices. Second is to exploit the rich user interface available through mobile apps to make existing applications usable by such users. Examples include even basic device based services such as SMS and address book among others that are currently not usable by low-literate. In this paper, we demonstrate the use of smartphones to enable a voice content sharing telephony application for textually illiterate users without the need for an Internet connection.

Voice Interface intermingled with visual interface

To demonstrate the synergy between voice user interfaces and smartphone based graphical user interfaces, we took the scenario of voice content sharing over telephony voice applications. The voice interface part of the application allows callers to identify and generate a link to online voice content of interest. This link is received by the caller as SMS which can then be shared with others. A corresponding mobile application makes use of intuitive icons that let the user call online voice application for accessing voice content. The application also allows navigation of local repository containing links received from others and enables commands for accessing those links without having to open the address book or SMS to punch in the code for those links.

Figure 1 shows screenshots of the application that enables users to share funny messages in a modulated voice, with their friends. On calling the application, the caller is asked to record any content. This content is then voice modulated to resemble a cat voice and is played back to the user. At this point the user can press the *7 key combination to save this recording and generate a link to this recording. If not, the user can simply continue using the application by recording more content. If the *7 key combination is pressed while the modulated content is being played back or a few seconds after it, the application logs the request to save it, generates a link for it and informs the user that he will get the link via an SMS shortly. The received link allows direct access to the stored voice content bypassing any voice menu navigation. To access a link, a feature phone user needs to dial the phone number embedded in the link and enter a numeric code (also embedded in the link) when prompted. When using the smartphone application the process is seamless and it requires only a tap from the user to access the link. On link access, the voice application fetches and plays the corresponding content and continues with the regular voice application interaction (i.e., the caller can record and share more such recordings).

This application enables textually illiterate users to make use of device features such as SMS and address book through the use of an intuitively designed mobile app. Also, it enables mobile apps to take the aid of voice interfaces in local language that these users are comfortable with.

A second example scenario where this concept can be applied is in the context of navigating Interactive Voice Response (IVR) menus of various organizations. Services such as Gethuman1 and Deepdial2 provide a mechanism to callers to navigate directly to a particular portion of an IVR's deep navigation menu without having to go through the entire process manually. A mobile application on the user's device could either prefetch or obtain via Short Message Service (SMS) on demand, an outline of the target IVR menu. On connecting a call, it could present a visual interface constructed from that IVR meta data. This would enable the caller to navigate the IVR menu through the visual interface rather than having to speak or punch in a digit everytime. Further, with the concept of voice links, the individual menu options of the IVR could be hyperlinked allowing the user to jump directly to a portion rather than wait for the entire navigation to take place sequentially. Augmenting IVR menus with corresponding visual interfaces was also explored by FonePal system [START_REF] Yin | The benefits of augmenting telephone voice menu navigation with visual browsing and search[END_REF] where an Instant Messaging (IM) client was used to present the visual interface.

System Design and Implementation

Even though the processing power and memory available on mobile phones has increased drastically in recent years, yet current speech recognition systems available on mobile devices are not powerful enough to perform sophisticated recognition tasks. Applications such as Siri3 virtual assistant from Apple and Nina4 from Nuance make use of cloud based speech recognition software to deliver their service. Since our target users may not have Internet subscription required to utilize such services, our system makes use of ordinary voice channel based interaction with server side voice application deployment and hosting.

To offer these telephony voice services coupled with visual mobile app interface, we make use of applications built over Spoken Web Application Framework [START_REF] Kumar | The spoken web application framework: user generated content and service creation through low-end mobiles[END_REF]. It is a platform that enables easy creation of new voice applications even without the need for any programming skills or even textual literacy. It has been deployed in several applications meant for serving the underprivileged that have language or affordability as a barrier to access or offer information services.

Figure 3 shows a typical deployment infrastructure where a smartphone user without a data connection invokes a mobile application. This mobile application provides its functionality locally and makes a call to a server based voice application deployment. The user could interact with the voice call even as supportive content or icons are displayed on the screen. The server side telephony infrastructure makes use of a Voice Gateway to convert Public Switched Telephone Network (PSTN) calls into voice-over-IP (VoIP). This is then delivered to the application platform which renders Voice XML (VXML) pages to interact with the user. Nuance Recognizer (or similar system) is used for Speech Recognition through its Automatic Speech Recognition (ASR) function. The database server manages all recordings, content and configuration, while SMS gateway and Email server provide a mechanism to send and/or receives SMS and email respectively. The mobile application can interact with server on any of these channels -voice, SMS, email. A web server is shown since in some scenarios, the stakeholders may like to see the reports and other function through a web based interface.

Since the voice interface is available over ordinary telephone call, it does not requiring any sophisticated device capabilities at the client's end.

Related Work

The concept of combining a conversational interface with a visual one has been applied in the context of visualization tools [START_REF] Sun | Articulate: a Conversational Interface for Visual Analytics[END_REF]. To create effective visualizations the Articulate system provides a semi-automated visual analytic model coupled with a conversational user interface. Using natural language processing and some heuristics it tries to create a suitable visualization as desired by the user. Employing conversational interfaces for software agents was employed as early as in the 90s [START_REF] Trower | Creating Conversational Interfaces for Interactive Software Agents[END_REF] and some of those concepts are visible in intelligent automated assistant systems of today such as CoCo system [START_REF] Lau | A Conversational Interface to Web Automation[END_REF] that uses conversational interface to automate tasks on the web as well as Siri and Nina. They are also being applied to other tasks such as information retrieval [START_REF] Lau | Towards conversational interfaces to web applications[END_REF].

Interplay of a conversational interface with animated personas has been studied by Oviatt et al. [START_REF] Oviatt | Toward adaptive conversational interfaces: Modeling speech convergence with animated personas[END_REF]. In their study with children, they learnt that children's speech con-verged with the text-to-speech (TTS) heard from the animated persona. The participating children adapted several acoustic-prosodic features of their speech based upon what they heard from animated personas. Further, children readapted when exposed to new personas.

However, the focus for such conversational systems primarily has been to enable multiple channels of communication with a software system or to study the influence of multiple channels of communications. Recent popular tools such as Siri or Nina, focus primarily on the conversational aspect of speech based interface and do not attempt to leverage the conversation interface with a corresponding visual interface.

This paper, on the other hand, proposed an interaction modality in which applications leverage conversational interfaces side by side and intermixed with visual interfaces to help several users overcome their accessibility problem. These could be textually-illiterate people or older adults unable to read or type properly especially on mobile phones. An entertainment application with such a visual conversation interface has been presented in [START_REF] Remy | Supporting Voice Content Sharing among Underprivileged People in Urban India[END_REF].

Discussion

Due to the increased proliferation of smartphones, it is only a logical step to leverage the power of visual interfaces and provide illiterates with additional means of communication. Friscira et al. [START_REF] Friscira | Getting in touch with text: designing a mobile phone application for illiterate users to harness SMS[END_REF] used a smartphone application to augment SMS messages with icons that let illiterate users make use of basic SMS messaging functionality. This marks just the beginning of a range of new possibilities that open up for researchers and practitioners. As we found out in a study focusing on the aspect of sharing information [START_REF] Remy | Supporting Voice Content Sharing among Underprivileged People in Urban India[END_REF], smartphone applications can contribute to the understanding of interactive voice application services and facilitate the interaction with such voice applications.

The example content sharing implementation we presented in this paper could be extended to allow for sharing to multiple contacts simultaneously by simply clicking on pictures of contacts stored in the address book. This would not only simplify sharing, it would also save time and therefore money for the caller of such a service, and make the process less error-prone by removing the need to enter all recipients' numbers via phone keyboard or voice input.

An area of growing interest in research in developing countries is that of job opportunities for the underprivileged [START_REF] Raza | Spread and sustainability: the geography and economics of speech-based services[END_REF][START_REF] White | Designing a Voicebased Employment Exchange for Rural India[END_REF]. Browsing such jobs by voice navigation only is cumbersome and time-consuming, eventually reducing effectiveness and success of such applications. Not only does this make such services more useful, the visualization might also contribute to the understanding of the hierarchical structure of voice systems.

Conclusion

We presented a new approach of creating visual conversational interfaces that utilize the voice interface of telephony voice applications along with rich graphical interfaces of smartphones to help low literate users overcome their handicap. While providing this, the application does not assume the presence of a data connection on the phone. In the presence of a data connection, these interfaces can be made a lot richer and capable than what is possible with just voice connectivity and local processing power. This mode of interaction (with or without data connection) is also applicable to other users with accessibility challenges such as older adults. As mobile phone platforms become more powerful to host voice applications locally or Internet usage charges become affordable for this population, voice applications could be much more seamlessly integrated with mobile applications similar to emerging voice-on-cloud based mobile assistant applications of today.

Figure 1 :

 1 Figure 1 : Screenshots of the Android application: a) home screen, b) list of recordings, c) single recording selected, d) screen during the phone call.

Figure 2 :

 2 Figure 2 : Researchers demonstrating the visual conversational interface to subjects from target population.

Figure 3 :

 3 Figure 3 : Infrastructure setup for providing conversational interface to mobile applications.

http://gethuman.com

http://deepdial.com

http://www.apple.com/ios/siri/

http://www.nuance.com/meet-nina/index.htm


Rahul Rajan
email: rahulraj@andrew.cmu.edu

Joey Hsiao

Deven Lahoti
email: deven.lahoti@gmail.com

Ted Selker
email: tselker@andrew.cmu.edu

"Roger that!" -The Value of Adding Social Feedback in Audio-Mediated Communications

Keywords: Audio-mediated, Conference calls, Considerate, Social feedback

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Communication is a type of social action [START_REF] Higgins | Achieving 'shared reality' in the communication game: A social action that creates meaning[END_REF]. It can be verbal and non-verbal in nature. From a suggestive glance to an admonishing tone, people rely on all sorts of cues to assess the situation and regulate their behavior. Particularly while collaborating, people orient themselves and coordinate in creating a shared reality. They engage in this process to seek understanding, and to be understood. Feedback is pivotal to this process, and it propels and directs further communications. It helps in creating a shared awareness and mutual understanding.

When the communication is mediated by technology there is a reduction in these social cues or feedback. This creates a sense of disengagement and psychological distance. It is interesting to note that both video and audio-only conferencing suffer from the attenuation of these cues, albeit in different measures [START_REF] Sellen | Remote conversations: the effects of mediating talk with technology[END_REF]. Hence, we find that the popularity of their use lies on a spectrum depending on the situation, the participants, the nature of the task, and the social setting. For instance, audio conference calls are widely used in business meetings [START_REF] Halbe | Whos there?[END_REF], whereas desktop videoconferences are more popular in personal settings [START_REF] Brubaker | Focusing on shared experiences: moving beyond the camera in video communication[END_REF]. Even so, the reasons for users choice and preference are nuanced and complex, involving multiple tradeoffs related to intrusion, amplification of inattention, and mobility. While there has been significant work and progress to preserve social cues in video communications [START_REF] Nguyen | Multiview: improving trust in group video conferencing through spatial faithfulness[END_REF], audio-mediated technologies have not received the same share of attention, and depend largely on visual cues like participant lists to buttress communications [START_REF] Yankelovich | Meeting central: making distributed meetings more effective[END_REF].

In this work, we explore how social cues can be restored on the audio channel, while addressing some of its most often cited drawbacks [START_REF] Tang | Why do users like video?[END_REF][START_REF] Sellen | Remote conversations: the effects of mediating talk with technology[END_REF][START_REF] Yankelovich | Meeting central: making distributed meetings more effective[END_REF]. These are predominantly social in nature, and focus on the process of interaction. These include the ability to accurately identify speakers, the notion of personal space for remote participants, and the issues of awareness about the presence of other collaborators. We design different types of audio cues, and experiment with feedback and feedforward techniques to better understand how these might support human communication. Our goal is to build a considerate agent that would know when and how to apply these techniques appropriately. Audio interfaces work well in spite of arguably have the lowest bandwidth for natural synchronous communications between multiple people. The difficulty in improving these communications with an agent further loading this narrow channel should then be maximally hard, making it an ideal place to demonstrate the utility of an agent being considerate and appropriate [START_REF] Selker | Understanding considerate systems -UCS (pronounced: You see us)[END_REF].

A second reason to experiment with augmenting the audio space is that many discussions and meetings involve documents and physical artifacts that occupy users attention. This makes display space expensive, and switching between display views task intensive. Besides, the visual channel is not the best medium to convey awareness information because human visual field is limited to the frontal hemisphere, and the foveal region in particular. This creates inherent limitations in the use of visual displays, wherein the user must see and attend to the display. Noticing visual changes also gets harder as tasks get more demanding, or if the display is cluttered.

Thirdly, people can perceive multiple audio channels simultaneously, and do so with considerable ease, especially while listening to music. In particular, we have the perceptual ability to hone in on a particular channel while filtering out the rest, commonly referred to as the "cocktail party effect" [START_REF] Arons | A Review of The Cocktail Party Effect[END_REF]. Thus, the audio channel can be used as an effective mode of transmitting background information (e.g., [START_REF] Gaver | Sound support for collaboration[END_REF][START_REF] Cohen | Out to lunch: Further adventures monitoring background activity[END_REF]). In addition, audio can be used for conveying temporal information like whether an activity is occurring right now, and when actions start and stop. It can also be used to indicate spatial and structural information, like where the actions are happening, the type of activity and the qualities of the action (e.g., [START_REF] Rigas | Communicating spatial information via a multimedia-auditory interface[END_REF][13][START_REF] Gutwin | Chalk sounds: the effects of dynamic synthesized audio on workspace awareness in distributed groupware[END_REF]).

To evaluate feedback techniques for improving communications, we choose the multiparty audio conference call with a shared-screen as our setting. Previous work demonstrates that considerate agent cues can reduce distraction, and help conference call participants equalize contributions [START_REF] Rajan | Considerate Audio MEdiating Oracle (CAMEO): improving human-to-human communications in conference calls[END_REF]. This paper moves further to show that considerate agents like CAMEO can more generally make call participants aware of others on the line and allow them to focus better on the conversation. In particular, we focus on speaker identification, audio presence, and entry & exit announcements. To evaluate speaker identification methods, we had participants listen to a pre-recorded five-person conference call and an-swer identity-related questions. We found that while speaker identification is hard, audio augmentations or spatially arranging the different speakers, could aid with identification success. We then show that by feed-forwarding simple audio cues, users were more assured, and less distracted about the presence of other collaborators. Finally, we had participants engage in a memory game while subjecting them to three kinds of entry/exit prompts. We show that by making a more natural, and less syntactic utterance, participants made fewer errors on average.

Related Work

There is a rich body of work on the use of audio for user interfaces, which provides the foundation for our work of supporting social cues in conversation. We briefly review how audio interface design has evolved, and the sounds and techniques others have used to provide audio feedback and guide user interactions. We then cover how audio has been used in distributed settings to allow people to coordinate better, and to increase shared awareness of remote events and activities.

Auditory Interface Design

Audio interfaces largely use two types of non-speech cues, namely, earcons and auditory icons. Earcons are synthetic tones whose timbre, pitch, and intensity are manipulated, to build up a family of sounds whose attributes reflect the structure of a hierarchy of information. Since earcons are abstract, they require training and need to be learned to be effective. Auditory icons are a more focussed class of audio cues, which are carefully designed to support a semantic link with the object they represent, making them easier to associate. Furthermore, sounds can be perceptually mapped to the events they indicate using symbolic, metaphorical and iconic methods.

Soundtrack [START_REF] Edwards | Soundtrack: an auditory interface for blind users[END_REF] was one of the first auditory interfaces to use earcons and synthetic speech. More recently, Rigas et al. [START_REF] Rigas | Communicating spatial information via a multimedia-auditory interface[END_REF] demonstrated the use of earcons to communicate information about the layout of a building. Four different timbres (piano, organ, horn, and clarinet) were used to communicate the sections of the building. Floors were communicated by musical notes rising in pitch. A single note was rhythmically repeated to indicate room number, and combination of timbres was used to indicate hallways. Users successfully located the rooms but were not able to interpret the different hallways, suggesting that combination of two timbres created confusion. Early in our work, we experienced how an overloaded audio dimension could easily be created by assigning multiple tracks of an orchestra to each participant. We focus on methods that prevent such overloading in a single audio dimension.

SonicFinder [START_REF] Gaver | The SonicFinder: An Interface That Uses Auditory Icons[END_REF] was the first interface to incorporate the use of auditory icons. A variety of actions made sounds in the SonicFinder, including the manipulation of files, folders, and windows. SonicFinder also made use of dynamic parameterized sounds to indicate temporal and structural activity, like file transfers producing a continuous filling up sounds, and different files producing different pitched sounds. In our work, we seek to explore when we can use the intuitive semantic mappings of auditory icons, over the arbitrary symbolic mapping of earcons.

A number of other works show how audio interfaces can improve interactions. gpsTunes [START_REF] Strachan | GpsTunes: controlling navigation via audio feedback[END_REF] focussed on using adaptive audio feedback to guide a user to their desired location. As the user gets closer to the target, the music gets louder followed by a pulsing track to indicate their arrival. Schlienger et al. [START_REF] Schlienger | Improving users' comprehension of changes with animation and sound: an empirical assessment[END_REF] evaluated the effects of animation and auditory icons on awareness. They showed that the auditory icons were commonly used to notify a change, and to focus attention on the right object just before it changed. AudioFeeds [START_REF] Dingler | AudioFeeds: a mobile auditory application for monitoring online activities[END_REF] explored how audio can be used to monitor social network activity, and PULSE [START_REF] Mcgookin | PULSE: the design and evaluation of an auditory display to provide a social vibe[END_REF] evaluated how audio cues can be used to communicate the local social vibes as a user walks around. This paper shows that such indicators might work well and not interfere with a conference call.

Activity Coordination in a Distributed Setting

SoundShark [START_REF] Gaver | Auditory icons in large-scale collaborative environments[END_REF] was an auditory interface extension of SharedARK, a multiprocess system that allowed people to manipulate objects and collaborate virtually. It used auditory icons to indicate user interactions and ongoing processes, to help with navigation, and to provide information about other users. Users could hear each other even if they couldn't see each other, and this seemed to aid in coordination. This work motivated the development of ARKola, a simulation of a soft-drink bottling factory [START_REF] Gaver | Effective sounds in complex systems: the ARKOLA simulation[END_REF]. Temporally complex sounds occupied different parts of the audible frequency spectrum, and the sounds were designed to be semantically related to the events they represented. Also, instead of playing sounds continuously, a repetitive stream of sounds were used to allow other sounds to be heard between repetitions. Gaver et al. observed that the sounds allowed the people to keep track of many ongoing processes, and facilitated collaboration between partners. Users were able to concentrate on their own tasks while coordinating with their partners about theirs, when sound was providing the background information. We seek to employ similar techniques to show how we might improve the process of audio communication itself.

The CSCW community has also paid attention to the use of audio in distributed workspaces. In a shared drawing environment, Ramloll and Mariani [START_REF] Ramloll | Do localised auditory cues in group drawing environments matter?[END_REF] played different sounds for different participants, and spatialized the sound in the 2D environment to help with location awareness. Participants complained that the spatial audio was distracting, but it provided them with information about others intentions which helped them with turn-taking. McGookin and Brewster [START_REF] Mcgookin | An initial investigation into non-visual computer supported collaboration[END_REF] looked into audio and haptic locating tools as well, while extending their single user GraphBuilder to a multiuser interface. They found that shared audio helped in mediating communication, and served as shared reference points, allowing users to refer to events they couldn't see. Our work seeks to extend this to situations where the fact of a persons presence is crucial to the outcome.

Shared Awareness in a Distributed Setting

The Environmental Audio Reminders (EAR) system [START_REF] Gaver | Sound support for collaboration[END_REF] transmits short auditory cues to people's office to inform them of a variety of events around their building . For example, the sounds of opening and closing doors are used to indicate that someone else has connected or disconnected from a user's video feed. They use stereotypical and unobtrusive sounds to make people aware of events in the workspace without interrupting normal workspace activities. We follow this approach attempting to discriminate in the more delicate domain of presence. ShareMon [13] used auditory icons to notify users about background file sharing events. To indicate the various actions involved, Cohen experimented with three types of sound mappings. For example, to indicate user login he used knock-knock-knock (iconic), "Kirk to enterprise" (metaphoric), and Ding-Dong (symbolic). To some degree, all three methods were intuitive and effective at communicating information, and users found them less disruptive than other modalities, like graphics and text-to-speech. In our work, we try to understand how these mappings affect users when they are used to interject ongoing communication.

The OutToLunch system [START_REF] Cohen | Out to lunch: Further adventures monitoring background activity[END_REF] attempted to recreate an atmosphere of "group awareness". It gave isolated or dispersed group members the feeling that their coworkers were nearby, and also a sense of how busy they were, by taking advantage of the human ability to process background information using sound. Each user had a theme that was mixed in with a seamless loop of solo guitar music, and would only play when the user was typing on their keyboard. With only six people in the group, the paper reports that users had no trouble associating a theme with the person it represented. We attempted to use a similar approach, but when convolved with conversation, the multitrack instrument sounds overload the channel and can be annoying. Similarly, there has been work in groupware systems to address the issue of awareness through audio. GroupDesign, a real-time multi-user drawing tool, used audio echo to represent user action on another users' interface [START_REF] Beaudouin-Lafon | Transparency and awareness in a real-time groupware system[END_REF]. In Thunderwire [START_REF] Hindus | Thunderwire: a field study of an audio-only media space[END_REF], an audio-only shared media space, the audible click of a microphone being switched on or off served to let participants know when people were joining or leaving the discussion. In Chalk Sounds [START_REF] Gutwin | Chalk sounds: the effects of dynamic synthesized audio on workspace awareness in distributed groupware[END_REF], Gutwin et al. used the granular synthesis method to create chalk sounds that were parameterized by the speed and pressure of an input stylus. Our goal is to extend such awareness without the need for users to break the flow of conversation by having to ask about who has joined or left the conference call, for instance.

AUDIO DESIGN FOR CAMEO

Synthesizing prior work on audio signals, we experiment with feedback techniques to support the social process of communication. Specifically, we incorporate them in the design of three CAMEO features -Speaker Identification, Audio Presence, and Entry/Exit announcements [START_REF] Rajan | Considerate Audio MEdiating Oracle (CAMEO): improving human-to-human communications in conference calls[END_REF]. We focus on the functionality and design of assistive audio cues for each of these features.

Adding audio cues on to an already overloaded communication channel with multiple speakers provided ample opportunity to be distracting and inconsiderate. We had to try many approaches and various kinds of audio feedback cues to find reasonable candidates to formally evaluate. We chose to work with Apple's GarageBand1 software. We pre-recorded two ten-minute teleconference calls between five people and used these as the base tracks in Garageband. This was then overlaid with earcons and auditory icons, which included instrument sounds from http://free-loops.com/, and nature sounds from http://www.naturesoundsfor.me/. In the following paragraphs, we summarize the lessons learnt from the broad explorations in the designs of each of these three features.

Speaker Identity

One of the major reasons people might find video conferencing attractive is because it elevates identifying and distinguishing between speakers to a separate channel with less crosstalk [START_REF] Sellen | Remote conversations: the effects of mediating talk with technology[END_REF][START_REF] Yankelovich | Meeting central: making distributed meetings more effective[END_REF]. The question of identity and presence can get even more muddled when people on the line are not familiar with each other, or their accents. So we experimented with different audio cues to support speaker identification in such difficult situations.

We focussed on designing earcons that were easy to perceive, remember and discriminate. We initially experimented with assigning background instrumental tracks to each participant. For example, the bass track might be assigned to participant one, and the rhythm guitars to participant two. When the participants spoke, the track assigned to them would start to play in the background. This, however, proved to be distracting as the instrumental tracks introduced too much crosstalk on the channel.

To reduce crosstalk, we experimented with simple tones instead of tracks, that pulse while the participant speaks. We were able to achieve good discriminability by using the following timbres : tambourine, bongos, and vibraphone for the 2nd, 3rd and 4th participants, respectively. For the 1st and 5th participants we used muted electric bass with tones that were an octave apart.

Once these qualitative design evaluations were done, the next part of the audio design was to determine the temporal nature of the cues, i.e when they should play and for how long. The cues were designed to play at the beginning of every utterance a participant makes while holding the floor. We found that this worked best when the cues began playing a second into the utterance, as opposed to right at the beginning of the utterance. This duration was long enough to ensure that a participant was contributing more than just back-channel feedback, like uh-huh. With regards to the duration, a cue that was a second long was distracting, whereas a cue that was 0.25 sec long was too subtle to discriminate from the other cues. Thus, the duration of the cues was set to be 0.5 sec.

An interesting side effect from designing the audio cues in this manner was that they also seemed to emphasize a participants hold on the floor, reinforcing personal audio space. These can be thought of as analogous to people's use of hand gestures while speaking. Thus, when a speaker is speaking loudly and at a rapid pace, the cues pulse rapidly too. If the speaker is speaking softly and at a slower rate, the cues pulse at a slower rate.

The timbre from Garageband are high quality, and occupy a large portion of the soundscape when mixed-in with the conference call. To push the auditory cues to the background we experimented with a number of filters and reverb effects. We found that using a high-pass filter, and the "small room" reverb effect worked most effectively.

Another technique to aid with the identifying speakers is to spatialize them in a 2D environment. We used stereo panning to place the 5 speakers at the -32, -16, 0, +16, +32 positions on the Garageband's Pan Dial. Like in experiments done by Ramloll [START_REF] Ramloll | Do localised auditory cues in group drawing environments matter?[END_REF], using stronger panning was actually distracting, and made the listener want to cock their head to the side where the sound was coming from, like when someone taps your shoulder.

Next, we describe the process of designing audio cues for the second CAMEO feature, i.e. to indicated presence of other collaborators on the line to a user.

Audio Presence

Feedback is very important for communication. Even the absence of feedback about whether the others on the line can hear you or not, can be distracting. For instance, without the addition of sidetone users have a tendency to attend to their displays to know if their call has been dropped or not. This can be disruptive to the conversation. Sidetone is a form of feedback thats picked up from the mouthpiece and instantly introduce into the earpiece of the same handset. It gives users the assurance that their signal is being registered by the phone system, and is therefore now incorporated into most phone devices. Similarly, the awareness that the other participants are on the line, and are listening is important confirmation which reduces uncertainty about the channel continuing to be functional. There are various methods to present such information visually [START_REF] Yankelovich | Meeting central: making distributed meetings more effective[END_REF], but the visual channel comes at a high bandwidth and attention cost as described above.

Initial ideas involved the use of background sounds to create a soundscape around which users could orient themselves. For example, if Ron is playing music in the background, or Joe is driving, it becomes immediately obvious when either of them go offline, even if they aren't talking. Their absence becomes conspicuous because of the sudden change in the soundscape. To test this idea in our system, we assigned different instrumental tracks to the participants, similar to the OutToLunch system [START_REF] Cohen | Out to lunch: Further adventures monitoring background activity[END_REF]. The tracks would play while the user was online, but they overloaded the conversational channel and were, therefore, highly distracting.

To soften the crosstalk, we were inspired by radio shows where people call in to chat with the host. They are usually located in different audio environments, and it's easy to tell them apart. We improvised on this by experimenting with different nature sounds like from a flowing stream, waves landing on the shore of a beach, and the chirping of birds. These blended so well into the background, though, that it was hard to notice when they stopped or started.

An alternative idea was to employ a roll cqall, i.e. to periodically announce the presence of the participants either by name or auditory icons. Announcing the names of participants would be the easiest to understand, but people may find it annoying to hear their names being called out periodically. Instead, auditory icons were created by sampling backchannel like participants laugh, and other characteristic sounds. These were inserted in the channel at periodic intervals when a participant had been silent for a while. However, they were too subtle and weren't noticed. We then experimented with recorded ambient environmental sounds of someone typing on a keyboard, clicking a mouse, opening and closing a drawer, and thumbing through papers. These auditory icons were found to be distinct, perceptible, and natural in a work environment.

In the final subsection, we consider multiple entry & exit announcements for CAMEO's Entry/Exit feature, which we describe below.

Entry/Exit

It can be hard to tell when participants get dropped from or reenter a conference call. Some existing conference call systems circumvent this by announcing entry and exit with a lengthy statement, which can be annoying. In this work we explore alternate iconic and metaphoric prompts to announce these events.

To avail of iconic mapping, we attempted to use the sounds of a door opening and closing to indicate entry and exit. We initially tried to superimpose the name of the participant with the sounds of the door opening and closing, but it was hard to discriminate between the door open and the door close sounds. We obviated this by appending the sounds to the name using different sequences. We usually hear the door open and then see the person enter, so an entrance is announced by the sound of a door opening followed by the name. On the other hand, when leaving we see a person head to the door and then hear the door close. So an exit is announced by the name followed by sound of the door closing.

We wanted to compare this to more metaphorical mapping approaches, like using fade and intonations. We modified the TTS in Audacity2 by using the amplitude fade-in and fade-out effects for entrance and exit, respectively. But this reduced the understandability of the name. For intonations, we chose to map entrance to a normal intonation, and exit to an upward intonation. This was partly due to convenience as Apple's TTS engine automatically intonates a word when it is punctuated with a question mark. For instance, "Armstrong?" is automatically intoned upwards and indicates that a participant named Armstrong has been disconnected from the conference call.

In the next section, we formally evaluate a final set of audio designs and protocols that came out of the preliminary explorations. We conducted three studies that we discuss below for each of the three features.

STUDY I: Speaker Identification

The goal of this study is to understand how the addition of audio cues to a conference call can help people differentiate between different speakers. To highlight difficulties in recognizing people in a conference call we chose five non-native english speakers (males, 22-28 years old), and had them remotely collaborate on a sub-arctic airplane crash scenario commonly used in team building exercises. According to the scenario, the five of them had just survived a plane crash in Northern Canada, and had managed to salvage some items. Their task was to list the items in order of importance, and to come to an agreement on this order as a group. The audio from each participant was recorded in separate files, which was then processed in Audacity. Garageband was used to create three separate versions: a simple downmixed version; one mixed with the speaker identification earcons discussed in the previous section; and another which arranged the speakers spatially in a 2D environment. We compare these three and see how well participants do on each. Our hypothesis was that the participants will do better with the spatialization and earcons aids, than without any audio cues.

Methods

The speaker identity study asked participants to listen to a segment of a prerecorded conference call while answering questions related to the conversation at hand, and speaker contributions.

Participants, Procedures, and Task Thirty-two people were recruited for the study (8 female, 24 male). Participants were between 20 and 30 years old, and all reported having no hearing impairments. Participants had the choice to take part in the experiments either remotely or in the lab This study had two stages, a training stage and a test stage. During the training stage, the participants were first asked to listen to the recorded introductions from the five speakers on the recorded conference call. They were presented with five colored buttons with the number and name of the different speakers. Upon clicking the button, they would hear a recording of the corresponding speaker saying their name and a fun fact. The next page allowed the participants to practice speaker tagging. They could click on the practice button which would cause the program to randomly play a short segment of the recorded conference call. The participant was asked to identify the speaker in the short segment by clicking on the speaker's corresponding button. After every attempt both the right answer and the selected answer were displayed. The participant could choose to go to the next screen whenever they felt confident of successfully differentiating between the different speakers. In the test stage, participants listened to a two minute and thirty second clip of the pre-recorded conference call. As they were listening, questions would appear about the conversation that had to be answered within five seconds.

Apparatus and Sounds Participants were provided with Logitech headsets. Remote participants were requested to find a quiet place and use headsets. They were provided with the address of the server where the experiment was hosted, and were asked to access it using the Chrome browser.

The earcon audio cues were obtained from Garageband. Speakers one to five were assigned muted electric base (low tone), tambourine, bongos, vibraphone, and muted electric base (high tone), respectively. Similarly, for creating a spatial 2D environment, speakers one to five were placed at the -32, -16, 0, +16, and +32 units on Garageband's pan knob (2D spatial positioning).

During the training stage when participants were introduced to the five speakers, their corresponding instrument or spatial location was also displayed, both visually and aurally.

Study Design We used a between group study, where half the participants answered the questions with the aid of musical instrument earcons, while the other half used 2D spatialization. For each group we also included a withinsubject condition to compare the test condition (with earcons) to a baseline (with no earcons). To balance out learning effects, half the participants started with the baseline, while the other half started with the test condition. Different two-and-a-half minute segments of the pre-recorded conference call were used in the within-subject study. The first segment had nine questions, while the second segment had eight questions.

To keep the test conditions same across study participants, and to isolate only the participant's perception of the audio cues, we used the same pre-recorded tracks in our evaluations. A limitation of this approach is that the audio cues are evaluated by third-party observers, and not by active participants of a meeting. We tried to account for this by asking questions that were of a "who said <something related to conversation>" nature, which is different from asking who just spoke. The aim was two-fold. First, to keep the participants engaged in the conversation, and to prevent them from simply matching audio cues to the speaker. Second, to cognitively load the user (as they might be while participating in a conference call) so that the distractive effects of audio cues might come to bear on the results.

Results

We present our results below in terms of participants being able to accurately identify the speakers on a conference call, and their response times. Attempt rate is the fraction of questions users answered in each condition. For accuracy, we report two metrics: Overall Accuracy, which includes questions that were not answered, and Attempt Accuracy, which only includes questions that were answered. Together, these metrics should account for distractions that audio cues might introduce causing participants to take longer than five seconds to answer a question. Spatialization vs. No audio cues Participants ability to identify speakers increased significantly, with greater than 20% improvement using spatial audio cues. They were able to do this almost half a second quicker on average when compared to the condition without audio cues (p<0.05, 1-tailed ttest, Table 1). Overall Accuracy: SEM=(0.024, 0.041); Attempted Accuracy: SEM=(0.046, 0.044); Response Time: SEM=(151.2, 163.7); N=16. Earcons vs. No audio cues With earcons, participants were also able to achieve an increase in accuracy of 30% on average over the condition with no audio cues (p<0.05). Participants also appeared quick to respond but the difference was not significant (p<0.1, 1-tailed t-test, Table 2). Overall Accuracy: SEM=(0.052, 0.024); Attempted Accuracy: SEM=(0.054, 0.054); Response Time: SEM=(169.0, 111.5); N=16. Summary We were able to show that speaker identification improved with the addition of either spatial cues, or earcons. A between group analysis did not reveal any difference between these two conditions. Furthermore, there was no significant difference in the number of questions that were attempted across the three conditions from which we might infer that the addition of audio cues was not notably distractive.

STUDY II: Audio Presence

The goal of the audio presence study is to investigate whether the addition of audio cues to a conference call can help reassure people that the other participants are still on the line, and haven't been disconnected. A different segment of the pre-recorded conference call described above was used in this study. Garageband was used to create two separate versions: a simple downmixed version with no audio cues added; and one mixed with the auditory icons for audio presence discussed in the previous section.

Methods

Participants were asked to listen to a segment of a pre-recorded conference call while answering some questions related to the conversation. The participants were also asked to indicate if they thought a participant had been dropped from the call.

Participants, Procedures, and Task Twenty people were recruited for the study (4 female, 16 male). Participants were between 20 and 30 years old, and all reported having no hearing impairments. Participants had the choice to take part in the experiments either remotely or in the lab The participant was asked to listen to a five-minute clip of the pre-recorded conference call. As they were listening, questions would appear about the conversation that had to be answered within ten seconds. Participants were also instructed to periodically ensure that everyone was online. They could do so by pressing the "nudge" button which simulated feedback from each participant stating that they were still there (like a ping test).

Apparatus and Sounds

The apparatus used by the participants is identical to the first study. The cues in this study were recorded using an iPhone, and processed in Audacity. As motivated in our exploration experiments above, these include auditory icons of ambient environmental sounds like someone typing on a keyboard, clicking a mouse, opening and closing a desk drawer, and shuffling through papers. These cues were then added to the segment of the pre-recorded conference call used for this test.

Study Design Our working hypothesis was that adding audio cues like keyboard sounds and mouse clicks acted to reinforce the presence of people who had not spoken in a while, but were still online. In other words, we wanted to show that like the sidetone, adding audio cues improves awareness about the presence of other collaborators.

We used a between group study, where half the participants were presented with audio cues (test condition), and the other half was not (baseline condition).

Participants had to answer eight multiple-choice questions while listening to the conversation. This was to simulate a real meeting where participants would be paying attention to the conversation, and not actively tracking the presence of other collaborators. Participants were told that because of some collaborators being in weak signal areas, there was a high chance that they might accidentally drop off the call. They were asked to virtually "nudge" the other participants if they suspected that one of them was not present.

Results

We investigate our hypothesis by comparing how often users "nudge" others to check if they are present, with and without the auditory icons discussed above. We found that the number of nudges was reduced by 37% in the condition where the auditory icons were used (p<0.05, 2-tailed t-test, Table 3). There was no significant difference in the attempt rate or error rate. # of Nudges: SEM=(0.72, 0.64); N=10. The goal of this study is to understand the effects that different conference call entry & exit announcements have on the participants, and meetings in general. We focus on three kinds of prompts, namely, speech, iconic and metaphoric. Our hypothesis is that the metaphoric prompts using different intonations will have the least impact on participants cognitive capability (i.e., their ability to follow game protocol in this particular study).

Methods

To bring out the effects, we designed and built a memory card game for four people that can be accessed remotely from the browser. Participants are paired off into two teams that take turns in choosing two cards from the sixteen that are shown face down on a GUI screen. If the two cards chosen by a team match, the team wins the turn. The team that matches the most number of pairs, wins the game. Participants, Procedures, Task We recruited 21 participants for this study (4 female, 17 male). Participants were between 20 and 30 years old, and collaborated remotely on the game. Six unique groups of four participants each were tested (some participants repeated).

When the participants join the meeting, the administrator would introduce them to the game, and the protocol they were to follow. During a team's turn, both team members are required to select a card. The selected card is revealed only to its selector. Thus, the first team member to click open a card has to communicate its content and position, based on which their partner picks the second card. The protocol specifically requires the team partners to alternate who gets to pick the first card at every turn. The protocol was designed in this way to encourage discussion.

After a practice round, the administrator would notify the participants that the experiments were going to begin. They were told that during the experiments, participants would randomly be dropped from the meeting. If they happened to be dropped from the conference call, they were requested to rejoin as soon as possible. During the course of such an event, a prompt would play to notify the rest of the participants that someone had left the conference call, while another prompt would play to indicate that they had joined back.

Apparatus and Sounds The apparatus used was identical to the first two studies. Mumble3 was used to host the conference call. All the participants were requested to download the Mumble client and follow the instructions that were provided.

A mac mini was used to run the python script that generated the prompts. The three entry and exit prompts that were used are speech-based, iconic, and metaphoric (Table 4). The prompts are dynamically created using Apple's textto-speech engine, and pre-recorded audio of a door opening and closing. The Python script was also set up to use the Mumble server's Ice remote procedure call interface to arbitrarily disconnect people every thirty seconds.

Study Design We used a within-subject study where each group played three rounds of the memory game, one for each of the three conditions. To balance out any learning effects, different sequences of the conditions were used for each group (Table 4).

Results

We wanted to investigate the effect that the different prompts would have on the participants ability to observe protocol, i.e. team members switching turns to pick the first card. We only take into account turns where both participants are online. We found that the metaphoric prompts had the lowest error rate at 15% in participants ability to maintain protocol compared to both the iconic and speech prompts (p<0.05, 2-tailed t-test, Table 5). The iconic prompts affected the participants as badly as the speech prompts did with error rates larger than 25%. Error Rate: SEM=(0.05, 0.05, 0.05); N=6. We also wanted to understand how the different prompts affected the game. We hypothesized that the shorter prompts would create less disruptions allowing the participants to finish the game quicker. There wasn't a significant difference in the durations, but the participants do appear to finish the games faster in the condition with the metaphoric prompts. The average durations are shown in Table 5. Duration: SEM=(34. 4, 22.3, 16.3); N=6.

Participant Preferences During the pilot experiments, participants strongly preferred the speech prompts to the metaphorical ones, which they found to be ambiguous. They were largely ambivalent about the iconic prompts. To help disambiguate the prompts in general, we began playing each of them at the start of their respective test conditions. This practice saw an increase in the number of participants who preferred the metaphoric prompts as they found it to be less distracting. They remained neutral with regards to the iconic prompts, although some of them claimed that it was hard to distinguish between the door opening and closing sounds when the line was noisy. This might explain poor participant performance under the iconic condition.

DISCUSSION

The three studies show that a constricted audio communication channel can be augmented with assistive social feedback cues, even in highly dynamic environments. In specific, we empirically showed that these cues allowed users to identify speakers more accurately, increased awareness about the presence of other collaborators, and improved participant performance.

To demonstrate the utility of the audio cues, we simulated particularly difficult and stressing situations. It is hard for the average person to distinguish between five people of the same gender with similar accents, or notice the quiet person in the room when engaged in conversation, especially when they aren't visible. Similarly, keeping track of multiple things while coordinating with others is difficult when there are a lot of distractions in the environment. As interactive systems weave themselves tighter into our social fabric, they need to be designed so as to accommodate such typically complex social situations. For instance, the first time we are introduced to a team that we are collaborating with, is when our understanding of their speech is most important; but it is also when their accents and behaviors are most difficult to interpret. We are very excited about being able to add audio to an already loaded channel and improve the perceived and real understanding of the situation.

Similarly, in an increasingly mobile and global workforce, a user might be in a noisy environment and have trouble distinguishing between some of the other collaborators on the conference call. In this case, the user could choose to add cues to some of the other collaborators, which would play only on their own channel. The sounds might also act as aids to users who might choose to associate meta-information (like the person's location or function) with an audio cue. Likewise, when to use speech, iconic or metaphoric prompts to announce events might be dependent on the situation. Developing an understanding of how these cues affect participants, and their applicability in different situations, allows us to build a vocabulary of actuators that a considerate agent like CAMEO would know when and how to use.

CONCLUSIONS and FUTURE WORK

The work described here focuses on domain-independent social feedback. We show how careful choice of its syntax -its sound and placement (like long utterances), and semantics -its direct and indirect relationship to the conversational channel (such as putting speech on top of a conversational channel), can deeply affect its goal of supporting social interactions. In particular, this paper focuses on how audio cues like earcons and auditory icons can appropriately provide feedback to stymie the disorienting effects of technology mediation. We show that earcons can improve accuracy on speaker identification, and is comparable to 2D spatialization of speakers. Auditory icons, like keyboard typing and mouse clicking, can act as feedback to reassure participants about the presence of others on the line. We also show that using metaphorical prompts (intonations) to announce events like entry and exit of participants reduces errors when compared to speech prompts.

With the proliferation of phones and tablets, more and more interactive systems are being used in social settings. The imperative now is for technology that celebrates situational awareness, and appropriateness. While this work builds a vocabulary of effectors to improve teleconference meetings, we look forward to the opportunities afforded by developing an understanding of how to accommodate system feedback in variety of other social situations. The choices of when and how a considerate agent should intrude on a communication channel is shown here to be delicate but tractable. We are excited about the possibility in the utility of such considerate agents across other interactive scenarios that would benefit from a system's ability to regulate and coordinate social feedback.

Table 1 .

 1 Accuracy metrics and average response times for speaker identification with and without 2D spatialization.

		Accuracy	Attempt	Accuracy	Response
		(Overall)	Rate	(Attempted)	Time (ms)
	Spatial	0.573	0.841	0.691	2187.4
	No cues	0.435	0.788	0.570	2663.0

Table 2 .

 2 Accuracy metrics and average response times for speaker identification with and without earcons.

		Accuracy	Attempt	Accuracy	Response
		(Overall)	Rate	(Attempted)	Time (ms)
	Earcons	0.538	0.772	0.703	2257.8
	No cues	0.386	0.819	0.475	2537.2

Table 3 .

 3 Average number of nudges, attempt rate, and error rate with and without auditory icons.

		# of Nudges	Attempt Rate	Error Rate
	Auditory icons 3.50	0.90	0.33
	No audio cues	5.63	0.81	0.35
	6 STUDY III: Entry & Exit

Table 4 .

 4 Entry & Exit prompts using different mappings in each test condition.

		Entry & Exit Prompts
	Speech	<participant name> has joined the conference <participant name> has left the conference
	Iconic	sound of door opening + <participant name> <participant name> + sound of door closing
	Metaphoric	<participant name> (said with normal intonation) <participant name> (said with raising intonation)

Table 5 .

 5 Average error rates in following the protocol and game duration across the three conditions.

		Error Rate	Duration (sec)
	Speech	0.29	262.3
	Iconic	0.26	258.6
	Metaphoric	0.15	222.0

http://www.apple.com/ilife/garageband/

http://audacity.sourceforge.net/

http://mumble.sourceforge.net/