Jeremy Birnholtz

Johnathon Schultz

Matthew Lepage

Carl Gutwin
email: carl.gutwin@usask.ca

A Framework for Supporting Joint Interpersonal Attention in Distributed Groups

Keywords: Awareness, Attention, Interaction, CMC, CSCW

Informal interactions are a key element of workgroup communication, but have proven difficult to support in distributed groups. One reason for this is that existing systems have focused either on novel means for gathering information about the availability or activity of others, or on allowing people to display their activities to others. There has not been sufficient focus on the interplay between these activities. This interplay is important, however, because mutual awareness and attention are the mechanisms by which people negotiate the start of conversations. In this paper, we present the OpenMessenger Framework, a system and design framework rooted in the assumption that individual behaviors occur in anticipation of and/or in response to the behavior of others. We describe both the system architecture, and specific examples of the novel implementations it enables. These include techniques for coupling gathering behaviors with display behaviors, and for integrating these into user workspaces via peripheral displays and gaze tracking.

Introduction

Informal interaction has repeatedly been shown by CSCW researchers to be a key attribute of modern work [START_REF] Allen | Managing the Flow of Technology[END_REF][START_REF] Kraut | Patterns of Contact and Communication in Scientific Research Collaboration[END_REF][START_REF] Whittaker | Informal Workplace Communication: What is It Like and How Might We Support It?[END_REF]. Significant efforts over the past 20 years have focused on supporting these interactions via improved awareness of others" presence and activities [START_REF] Dourish | Awareness and Coordination in Shared Workspaces[END_REF][START_REF] Harrison | Media space 20+ years of mediated life[END_REF], by improving people"s ability to interrupt at appropriate times [START_REF] Dabbish | Controlling Interruptions: Awareness displays and social motivation for coordination[END_REF], and by strategically displaying impending interruptions (e.g., [START_REF] Iqbal | Understanding and developing models for detecting and differentiating breakpoints during interactive tasks[END_REF][START_REF] Mccrickard | Introduction: design and evaluation of notification user interfaces[END_REF]). While this work has yielded many research prototypes, the most common tools used in everyday, real-world collaboration still offer only rudimentary support for initiating and concluding informal interactions [START_REF] Birnholtz | Adapt, Abandon: Understanding Why Some Young Adults Start, and then Stop, Using Instant Messaging[END_REF][START_REF] Tang | Approaching and leave-Taking: Negotiating Contact in Computer-Mediated Communication[END_REF].

One key reason for this is the persistent difficulty of supporting fluid transitions between passive awareness of one"s surroundings and engaged attention to a particular person or object. Many have demonstrated the ability of people in face-toface environments to track others" activities in their visual or auditory periphery, and adjust their own activities or shift their focus of attention accordingly (e.g., [START_REF] Frischen | Gaze cueing of attention: Visual attention, social cognition, and individual differences[END_REF][START_REF] Sudnow | Temporal parameters of interpersonal observation[END_REF]).

Background

When people are aware of or interact with others, we consider them to be managing their attention, which can vary both in intensity and focus, to activities and stimuli in the environment. As such, we define attention more broadly than a purely cognitivist interpretation that considers only a single point of focus [START_REF] Kastner | Mechanisms of Visual Attention in the Human Cortex[END_REF]. Broadening this definition allows us to account for transitions between more and less active attentional states, and is consistent with evidence from neuroscience on the mechanisms by which people sense and attend to the presence of others (e.g., [START_REF] Nummenmaa | Neural mechanisms of social attention[END_REF]).

At the same time, our treatment leaves aside debate on the relationship between attention and awareness, as discussed by Schmidt [START_REF] Schmidt | The problem with 'awareness[END_REF]. As Schmidt ultimately points out, however, the interesting practical questions in this domain are not about the underlying awareness and attention mechanisms, but rather the roles played by specific types of information in affecting behavior. We take as our baseline assumption that abstract awareness in the sense discussed by Dourish and Bly [START_REF] Dourish | Portholes: Supporting awareness in a distributed work group[END_REF] often leads to conscious and focused attention, and that the details of this transition from awareness to engagement are not well supported by today"s tools.

We look at these processes within the context of informal interaction and conversation initiation. This is not an exhaustive treatment, but represents a first order approximation [START_REF] Ackerman | The Intellectual Challenge of CSCW: The Gap Between Social Requirements and Technical Feasibility[END_REF] of a social problem; one that will enable a foothold in this key area.

Gathering, Display and Coupling

Attentional behaviors can be usefully framed using the terms gathering and display. Gathering refers to one person getting (either via passive monitoring or active seeking) information about what another is attending to, such as tasks, other people or the gatherer herself. Display refers to information about the focus of one"s attention that is available for others to gather. We separate these concepts for discussion, but note that they are highly interdependent.

These cues are used reciprocally; initiating interaction is a process of negotiating joint attention via sequences of actions informed by present and prior actions of others [START_REF] Birnholtz | Privacy in the open: how attention mediates awareness and privacy in open-plan offices[END_REF][START_REF] Frischen | Gaze cueing of attention: Visual attention, social cognition, and individual differences[END_REF][START_REF] Kendon | Conducting interaction: patterns of behavior in focused encounters[END_REF]. As such, one framework that can be useful in understanding this process is Clark"s framework of joint action. This framework allows for the situation of Schmidt"s observations about both passive and active states of awareness within the context of behavior aimed at a shared objective or goal. Joint action occurs when people act in the belief that they are part of a collective activity, in which their actions occur in response to coordination signals from another party. We argue that gathering and display are what Clark [START_REF] Clark | Using language[END_REF] calls component moves in joint action.

Consider again a case in which Alex wants to talk to Bill. Alex must get Bill"s attention, which he does using both gathering and display. Glancing quickly allows Alex to gather information about Bill"s likely availability; he then walks toward Bill, which serves to display Alex"s intent to start a conversation. Bill then can respond by using gaze and body position to display his own interest.

These individual actions can be interpreted within the framework of joint activity. If Bill is wearing headphones and does not look up as Alex approaches, for example, that could signal either that Bill is unaware of Alex, meaning that Alex needs to get closer and louder; or that Bill is aware that Alex is trying to talk to him, but signaling that he (Bill) is unavailable. This judgment on Alex"s part is based on information about Bill, the outcomes of recent actions, and knowledge of the context.

One key attribute of Alex"s approach for the purposes of attention management is the relationship between gathering and display. Alex"s movement toward Bill to gather information necessarily functions simultaneously as an act of display, because Bill can see him approaching [START_REF] Birnholtz | Privacy in the open: how attention mediates awareness and privacy in open-plan offices[END_REF]. We can therefore say that these instances of gathering and display are tightly coupled. In face-to-face interactions, people"s ability to notice others (e.g., [START_REF] Nummenmaa | Neural mechanisms of social attention[END_REF]) relies on this coupling. We perceive others" gathering because it is visible in ways that we can attend to.

Returning to Clark"s terms, we can define coupling as the extent to which a particular action is visible and noticeable to others. Table 1 illustrates coupling relationships between gathering and display. In face-to-face approaches (top left cell), physical proximity and eye gaze are effective ways to display attention because gathering and display are tightly coupled [START_REF] Birnholtz | Privacy in the open: how attention mediates awareness and privacy in open-plan offices[END_REF]. In contrast, many behaviors that are tightly coupled in face-to-face interactions have different relationships online [START_REF] Bailenson | Transformed Social Interaction: Decoupling Representation from Behavior and Form in Collaborative Virtual Environments[END_REF], as illustrated in the three remaining cells of Table 1. If Alex and Bill are spying covertly on each other via webcam, for example, gathering and display may be completely decoupled (bottom right cell). Alex"s gathering is not displayed at all, and vice versa. This helps to address a key question identified by Schmidt [START_REF] Schmidt | The problem with 'awareness[END_REF]: how do people regulate the obtrusiveness of their behavior? In face-to-face encounters, obtrusiveness is often a function of the salience of display. Staring at somebody"s screen or standing very close to them, for example, are very salient forms of display that are also obtrusive [START_REF] Birnholtz | Privacy in the open: how attention mediates awareness and privacy in open-plan offices[END_REF]. Online, however, it is possible to get detailed information about others without appearing obtrusive or even being visible at all. These examples of asymmetric coupling relationships have a significant effect both on how people regulate their behavior, and on the feasibility of joint action.

One example is the "appear offline" option on instant messaging (IM) clients used by those wishing to avoid interruptions [START_REF] Birnholtz | Adapt, Abandon: Understanding Why Some Young Adults Start, and then Stop, Using Instant Messaging[END_REF]. People using this option can gather information about others on their contact list without those others even knowing that such gathering is possible (because they cannot see that the gatherer is online). This sidesteps the key role that obtrusiveness ordinarily plays in attracting and negotiating attention and has significant consequences for joint action, because people cannot respond to actions of which they are unaware. In other words, it is the coupling of gathering and display in face-to-face interactions that helps facilitate joint action.

Gathering is Display; Display is Gathering

Despite Schmidt"s discussion of awareness as a duality of gathering and display, most systems and theoretical frameworks for addressing the problem have focused either on one or the other.

Early media space systems used cameras to provide video views of others in their offices [START_REF] Dourish | Portholes: Supporting awareness in a distributed work group[END_REF][START_REF] Fish | The VideoWindow system in informal communication[END_REF][START_REF] Gaver | Realizing a Video Environment: EuroPARC's Rave System[END_REF][START_REF] Harrison | Media space 20+ years of mediated life[END_REF]. Cameras, however, were thought by some to be invasive [START_REF] Boyle | The Language of Privacy: Learning from Video Media Space Analysis and Design[END_REF][START_REF] Clement | Considering privacy in the development of multi-media communications[END_REF]; and the systems did not support the subtleties of negotiating interaction [START_REF] Heath | Reconsidering the virtual workplace: flexible support for collaborative activity[END_REF]. These problems reflect the de-coupling of gathering and display in that one user could view (i.e., gather) video of another, without a clear display that this was taking place.

These early video experiences led many to experiment with the notion of a "virtual approach" (reviewed in [START_REF] Tang | Approaching and leave-Taking: Negotiating Contact in Computer-Mediated Communication[END_REF]). In our terms, this work can be characterized as an attempt to increase coupling between gathering and display by displaying to an observed party that gathering is taking place, and that conversation may be desired. The idea was that the approach would facilitate interaction more naturally by allowing for multiple levels of gathering, and by displaying activities to the observed parties.

As such, "approaches" often involved replicating the sequence of actions typically involved in initiating face-to-face conversations. Several systems allowed users to, for example, "glance" at others to discern their availability [START_REF] Isaacs | A desktop environment supporting impromptu and planned interactions[END_REF], and then follow a series of progressively more informative steps eventually resulting in conversation. At each step, the observed party would have to respond in kind (e.g., with a "glance" of their own) to proceed with the interaction.

More recent systems such as Community Bar [START_REF] Mcewan | Supporting social worlds with the community bar[END_REF] allow for a continuum of awareness states. In Community Bar, these must be manually updated via independent "focus" (how much information is seen about others) and "nimbus" (how much information is revealed to others) sliders. These terms come from work by Benford and Fahlen [START_REF] Benford | A spatial model of interaction in large virtual environments[END_REF] and Rodden [START_REF] Rodden | Populating the application: a model of awareness for cooperative applications[END_REF] that aims to distinguish between being the object of somebody"s attention (their nimbus) and focusing on somebody (one"s focus). While this distinction is a useful one, the Community Bar implementation is problematic in that it renders the coupling relationship between gathering and display dependent on the combined status of independent users" sliders being manipulated in parallel. That is, Alex could choose to reveal more information to Bill (via a nimbus slider) even as Bill is reducing the amount of information he sees about Alex (via a focus slider).

From a joint action standpoint, designs that foster multiple levels of awareness and sharing are an improvement in that they allow for coordinated activity. These systems were critiqued, however, for requiring lockstep and seemingly artificial sequences of behavior. By this, we mean that real-world gathering behaviors (e.g., glancing, walking up to somebody) are easily noticed and responded to because, unlike pointing at buttons in an interface or watching a PC window for notification of an incoming virtual "glance," they are the actions that naturally occur to assess availability, respond to somebody to avoid appearing rude, and/or to start a conversation [START_REF] Frischen | Gaze cueing of attention: Visual attention, social cognition, and individual differences[END_REF][START_REF] Kendon | Conducting interaction: patterns of behavior in focused encounters[END_REF].

We argue that effective support for a joint action approach to attention requires consideration of the interplay between gathering and display. Specifically: 1) acts of gathering must be coupled to displays or notifications, and 2) these displays must be easily noticed and responded to via subsequent acts of gathering, that must also be displayed. Gathering must be displayed and displays must be gatherable, ad infinitum.

Approaching is Interacting

A second key question in this area concerns the specific information that causes people to adapt their behavior in response to the actions or reactions of others. There are useful lessons from face to face interaction that can be considered here.

Goffman [START_REF] Goffman | The presentation of self in everyday life[END_REF] argues that human behavior around others is performative; it is often intended to convey information or impressions to others. Sudnow [START_REF] Sudnow | Temporal parameters of interpersonal observation[END_REF] discusses the importance of glances in assessing others" behavior and availability. He notes that people in public settings know that others may glance at them and act accordingly, such as by putting on headphones or adjusting posture to appear busy [START_REF] Birnholtz | Privacy in the open: how attention mediates awareness and privacy in open-plan offices[END_REF]. Sudnow [START_REF] Sudnow | Temporal parameters of interpersonal observation[END_REF] refers to these as "glanceable states," in that status can be discerned via a glance.

In some ways, the glanceable state is reflected in current interaction tools, such as the IM contact list. The intent of the list is to concisely summarize who is online and available. There has also been work aimed at improving this information by automatically updating status information (i.e., availability) via sensors (e.g., [START_REF] Begole | Lilsys: Inferring Unavailability Using Sensors[END_REF][START_REF] Fogarty | Predicting Human Interruptibility with Sensors[END_REF]).

One problem with the IM contact list, however, is that recent work has focused either on the problem of interruptions [START_REF] Mcfarlane | The scope and importance of human interruption in human-computer interaction design[END_REF] and developing systems that allow for better timing of interruptions (i.e., better gathering of information; [START_REF] Avrahami | IM waiting: timing and responsiveness in semisynchronous communication[END_REF][START_REF] Dabbish | Controlling Interruptions: Awareness displays and social motivation for coordination[END_REF]) or on techniques for unobtrusive notification of impending interruption (i.e., better display; [START_REF] Bartram | Moticons: detection, distraction and task[END_REF][START_REF] Booker | Usability testing of notification interfaces: are we focused on the best metrics?[END_REF][START_REF] Mccrickard | Introduction: design and evaluation of notification user interfaces[END_REF]. Considering these problems in isolation ignores a key component of our joint action argument: acts of gathering and display occur in response to each other. Approaching somebody does not occur prior to interacting; it is part of the joint action of initiating conversation. That is, approaching is interacting.

People do not respond the same way to interruptions from all others, such as work collaborators vs. social friends, [START_REF] Davis | Using Relationship to Control Disclosure in Awareness Servers[END_REF], and also may behave differently when they know their behavior is being monitored by others [START_REF] Forsyth | Group dynamics[END_REF]. Moreover, not all interactions result from interruptions; many result from serendipitous mutual attention [START_REF] Kendon | Conducting interaction: patterns of behavior in focused encounters[END_REF]. Thus, our second argument is that each act of gathering and display, however preliminary from the standpoint of starting verbal conversation, must be considered as component behaviors in an interaction (or joint action) to which others should be able to respond.

A real-world approach progresses from distant observation -characterized by less detailed gathering and less salient display -to closer observation -characterized by more detailed gathering and more salient display due to physical closeness. We advocate a similar progression online, consisting of multiple types of interactive behavior that enable both gathering and display to take place. As with the face-to-face approach, we emphasize that what is important is not reciprocal instances of identical behavior (i.e., a glance must be followed by a glance), but rather a general correlation structure between behaviors such that the amount of detail that can be gleaned from a particular gathering behavior roughly correlates with the salience of the display behavior with which that gathering is coupled.

The OpenMessenger Framework

Supporting a joint-action approach to attention management requires mechanisms for coupling gathering and display behaviors, and for treating these as interaction. In this section of the paper, we present the OpenMessenger1 Framework (OMF), a software framework and application for addressing these issues. We aim to make two contributions: 1) an extensible software framework for experimental exploration of issues related to awareness, and 2) an implementation example with novel gathering and display mechanisms.

Supporting Gathering and Display at the Framework Level

To effectively support joint action in attention management, we need a conceptual architecture that supports gathering and display of information, and the coupling of these behaviors to each other. This is accomplished in OMF with abstractions called sensor managers, monitors, awareness events, and views. Data about user activities is captured from hardware sensors by sensor managers, and is then analyzed and distributed to other OM clients by monitors. Transmission of the data is via awareness events passed from clients to the OM server, and then to all clients in a pre-defined group (see Figure 1). Event information is made perceptually available to users in a view that could be a visual, auditory, or tactile display.

Monitors

To allow people to gather useful information about others while still restricting the flow of potentially sensitive sensor data (as in [START_REF] Begole | Lilsys: Inferring Unavailability Using Sensors[END_REF][START_REF] Fogarty | Sensing from the Basement: A Feasibility Study of Unobtrusive and Low-Cost Home Activity Recognition[END_REF]), and avoiding user inundation with information, OMF uses monitors to distill raw sensor data. OMF monitors are C# objects on the client that subscribe to one or more sensor managers and process the raw sensor data for publishing to the server as awareness events. These events are sent to the server, which broadcasts them to all connected clients (including the sender, though the sender does not use them). Windows Communication Framework (WCF) is used for network communication (based on TCP sockets). When an awareness event is received, the server stores it in the server event cache. With this cache, event data can be immediately sent to newly connected clients.

As an example, the monitor for a keyboard activity sensor might release keystroke frequency, but not reveal which keys were pressed. This allows for conveying information about activities without releasing potentially sensitive data, and also takes data that may not be meaningful in raw form (e.g., sound level) and processes it to provide useful information (e.g., presence of sound above a conversational threshold).

Importantly, monitors are abstractions of sensor managers. While raw sensor data generally comes only in one form, data can be used in many ways. For example, a microphone sensor could indicate sound above a threshold, while another monitor could use speech recognition on the same data to determine if there is a conversation going on. As with sensor managers, subtypes of the Monitor class are singletons.

Social Monitors

Monitors, as described so far, operate similarly to previous systems when the user is focused on objects in the task or environment [START_REF] Fogarty | Presence versus Availability: The Design and Evaluation of a Context-Aware Communication Client[END_REF]. Our joint-action perspective, however, requires a way for users to dynamically detect and respond to attention from other users. Thus, a unique feature of OMF is special support for situations when the user"s focus of attention is on another user. This is accomplished via social monitors, a sub-type of the monitor class that communicates this information.

Social monitors are necessary because attention to another person is distinct from attention to a task element or interface object. As discussed above, attention to people is interactive; it often occurs often in expectation of and/or in response to another act of gathering or display. From an implementation standpoint, social monitors are unique in that they affect the intensity of attention that one user is paying to another, which we define later as focus level. Social monitors determine when one user is attending (e.g., via mouse or other input data) to another user"s representation (e.g., an on-screen avatar) on a view. This is accomplished via data about attentional focus (e.g., an x,y coordinate pair from the mouse or eye tracker monitor) and knowledge about the arrangement of avatars on the view. When one user is determined to be attending to another, focus levels are adjusted accordingly (see below).

Sensor and Monitor Examples

Example OMF sensor managers and monitors are shown in Table 1. This is not an exhaustive list of possibilities; most of these specific examples were chosen based on our own experience with the first OM application and based on prior work on predicting availability with sensor data [START_REF] Avrahami | IM waiting: timing and responsiveness in semisynchronous communication[END_REF][START_REF] Begole | Lilsys: Inferring Unavailability Using Sensors[END_REF][START_REF] Fogarty | Presence versus Availability: The Design and Evaluation of a Context-Aware Communication Client[END_REF]. That work suggests that keyboard activity, idle time, and the presence of sound can be very helpful in determining availability. Gathering active window titles and screen snapshots are sources that were used in the initial version of OM, but were seen by some users as too invasive [START_REF] Birnholtz | Gradual Initiation of Interaction for Distributed Workgroups[END_REF]. We therefore added a monitor for this data source that only detects screen changes, rather than transmitting the actual screen image. The eye tracker and mouse monitors are unique to the OMF in that they are social monitors, described below.

OMF also provides a simple extension mechanism for adding new monitors for additional analyses, or new sensor managers for new data sources. This requires some coding by the application programmer -creating a new subtype of Monitor or SensorManager, and linking APIs for new hardware sensors to these classes -but the process is simple and existing classes can be used as templates.

Views

Gathering and display behaviors are rooted in the perceptual availability of awareness information. Gathering cannot occur if this information cannot be perceived, and display has no purpose if the information is unavailable to others. Awareness information in OMF is made perceptually available via a view. OMF provides a basic visual view based on earlier OM systems [START_REF] Birnholtz | Gradual Initiation of Interaction for Distributed Workgroups[END_REF]. However, application programmers can easily develop new views within the .NET environment (e.g., using XAML or Windows Forms), or can build displays that use sound or other feedback mechanisms. Views are client-specific (i.e., they are not WYSIWIS), and multiple views can run simultaneously on the same client. A standard data structure for each user simplifies the storage of view-based information such as an avatar image, user name and ID, and the most recent awareness event data received pertaining to that user.

Focus Level: Degree of Attention and Coupling

As noted above, people do not respond the same way to interruptions from different people, and some interruptions result from serendipitous mutual attention. Alex may be available to Bill but not to Cathy, for example. Support for variation in treatment of contacts, and for awareness of mutual attention requires the capacity to share different information with certain contacts. Moreover, as Alex receives attention from Bill, he should be able to easily respond by displaying his own attention to Bill (indicating interest or availability) or to a task or another user (indicating that he is busy). In OMF, this is accomplished via the focus level mechanism, which represents the level of attention that one user is paying to another.

It is through focus levels that acts of gathering are coupled to acts of display, and that the correlation structure of gathering and display behaviors is maintained. First, the focus level of one user on another determines what information the first can gather about the second. Each awareness event includes a threshold focus level for event data display. This means that if Alex releases an awareness event with threshold level X, Bill will only see this event if his focus level on Alex is greater than or equal to X. More detailed information is therefore assigned a higher focus level.

Second, the focus level of one user on another is used to notify the observed user (i.e., display to them) that gathering is taking place. Notification increases in salience as focus level increases. That is, Alex receives increasingly salient notifications as Bill"s focus on him increases. Our OMNI system described below uses visual salience, but this could be accomplished with sounds or other cues as well. The key is that more detailed gathering behaviors correlate with more salient displays. Focus level is represented in the system as an integer (range: 0-5, with 0 as minimum) for each user"s level of focus on each other user. Focus level is updated dynamically via social monitors, which detect one user"s focus on another. The 0-5 range is based on experience with the initial OM version, suggesting that six levels provides enough variance for multiple means of gathering, and for displaying varying levels of interest without distracting the user. Further, focus is uni-directional. That is, Bill"s focus on Alex does not vary directly with Alex"s focus on Bill. Rather, each client stores a list of the current client"s focus level on all other users. As such there are two discrete focus level variables that describe Alex and Bill"s focus on each other. Thus, OMF is not a strict reciprocity-based system [START_REF] Tang | Approaching and leave-Taking: Negotiating Contact in Computer-Mediated Communication[END_REF].

Information displayed at each focus level in the current OM is listed in Table 3. This sequence is preliminary, but based on our experience with sharing similar information in previous OM versions. The focus level mechanism could also be exploited in more powerful ways by providing multiple information streams at each level, or by providing progressively more detailed information from a single sensor.

Focus Level

Data/Monitor (see

Changing Focus Level

Focus level is increased via explicit indicators of attention from social monitors (e.g., mouse and eye-gaze data) and decreased over time via attenuation. With the mouse social monitor, for example, if Alex"s cursor hovers over Bill"s avatar for a predetermined length of time the social monitor produces a focus change event that is distributed to all clients.

Testing of earlier OM versions [START_REF] Birnholtz | Gradual Initiation of Interaction for Distributed Workgroups[END_REF][START_REF] Birnholtz | Sharing Awareness Information Improves Interruption Timing and Social Attraction[END_REF] suggests that these trigger mechanisms must be carefully designed. For example, users tended not to use OM"s awareness features if it took more than a second for information to appear. We therefore increase focus level after one second of sustained attention to an avatar, and subsequently increment focus level by one after each second of continued attention. This continues until the observer stops focusing on the target, or focus level reaches its maximum.

Focus then decreases over time at a linear rate when the observer stops focusing on the target. Decrease is slower than increase: 15 seconds to decrease from level 5 to 0, compared with 5 seconds to increase from level 0 to 5.

As with views, the mechanisms for changing focus levels can be extended in OMF. For example, it would be simple for an application programmer to add a social monitor that changes focus levels through explicit keystrokes or commands, or that changed the way that focus level increases and decreases.

Event Filtering

As noted above, monitors generate events to be displayed only to users with at least a threshold focus level on the user generating the event. This presents the problem of how to broadcast events: should they be shared with all users, but displayed only to those viewing at the appropriate focus level (receiver-filter), or shared only with users at a particular focus level (sender-filter)? We chose the receiver-filter strategy to reduce the complexity of the system in terms of server query overhead. Given relatively low numbers of connected clients and sensors, broadcasting the data to all clients at once involves fewer operations and is simpler than a query-based model. This might be reconsidered, however, in an environment with a substantially greater number of clients or sensors. This means more network traffic, but this is acceptable since overall bandwidth requirements are low.

Text Chat

To support conversation, the OMF provides basic text chat. This is supported by the framework via the MessageEvent, a type of client-generated OM event characterized by a short text message, a sender and a receiver. The chat window implementation is similar to other text-based chatting systems, so we do not provide a detailed description here. However, OMF chat is novel in that conversations are a potential data source -that is, the "In Conversation" monitor keeps track of who is talking to whom (Table 1), and can use this to help determine attention and focus level.

An OMF Implementation Example: Eye Tracking and OMNI

To illustrate the capabilities of the OMF, we here describe our current implementation of a system called OMNI, which combines a social monitor that reads data from a head-mounted eye tracker, and a projected peripheral-vision awareness view (Figure 3). OMNI is an OMF implementation example intended for information or other office workers who work primarily at a desk but benefit from frequent informal interaction with potentially remote colleagues (e.g., designers [START_REF] Birnholtz | Privacy in the open: how attention mediates awareness and privacy in open-plan offices[END_REF], engineers, or researchers [START_REF] Allen | Managing the Flow of Technology[END_REF][START_REF] Kraut | Patterns of Contact and Communication in Scientific Research Collaboration[END_REF]). Those with a different work environment may benefit from a different OMF implementation. Elements of this system have been reported previously [START_REF] Birnholtz | Awareness Beyond the Desktop: Exploring Attention and Distraction with a Projected Peripheral-Vision Display[END_REF], but this is the first explication of OMNI at the system level.

The EyeTracker Social Monitor

One problem that the OMF enables us to address is the difficulty of mapping natural face-to-face behaviors (e.g., glancing) to online interfaces. Joint action requires the capacity for easy response, and one way we accomplish this is by using an eye/head tracker to the user"s visual focus. Eye tracking is particularly appropriate for this context because interpersonal attention is often conveyed via the eyes. Looking at somebody implies interest, and looking away implies disinterest [START_REF] Argyle | Gaze and Mutual Gaze[END_REF][START_REF] Frischen | Gaze cueing of attention: Visual attention, social cognition, and individual differences[END_REF][START_REF] Kendon | Conducting interaction: patterns of behavior in focused encounters[END_REF].

We use an ASL H6 head-mounted eye tracker and a Flock of Birds magnetic head tracker in our current system. We acknowledge this is a cumbersome and expensive device, but note that eye tracking technology is rapidly falling in cost as it becomes possible to track unobtrusively and inexpensively via webcams (e.g., [START_REF] Chau | Real Time Eye Tracking and Blink Detection with USB Cameras[END_REF]). While our specific implementation necessarily relies on the details of our hardware, the general approach to eye tracking that we describe here is not device-dependent; it could be adapted to any device that provides (x,y) coordinate pairs on a defined plane in space.

The eye tracker"s API libraries allow for the definition of planes in the real-world space. Each eye-tracker data point consists an (x,y) coordinate pair on a particular plane. Augmented by basic knowledge of the size, location, and resolution of the display that are defined as parameters, we determine when a user is visually attending to elements of the display; and in particular, when they are looking at another person"s avatar. When this occurs and fixation is prolonged, the social monitor increases the focus level on the observed party.

The OMNI Awareness View

One problem with existing systems is the difficulty of noticing when others gather information. Where social and peripheral attention processes mean that people constantly monitor their real world surroundings for the presence and gaze of others [START_REF] Nummenmaa | Neural mechanisms of social attention[END_REF], limited screen space and window occlusion can limit attention to avatars or notifications online [START_REF] Mccrickard | Introduction: design and evaluation of notification user interfaces[END_REF], and these displays can be distracting as well [START_REF] Birnholtz | Awareness Beyond the Desktop: Exploring Attention and Distraction with a Projected Peripheral-Vision Display[END_REF].

The OpenMessenger Notification and Interaction (OMNI) view is intended to exploit the properties of human visual perception in concert with the eye tracker social monitor and OMF. In the OMF, OMNI is classified as a view. The OMNI display consists of avatars representing contacts currently logged in, in addition to basic awareness information about these contacts. The display is projected onto a surface behind and above the user"s primary monitor (or displayed on a large screen in the same space), so that it appears in the periphery of the user"s visual field. Avatars are arranged in a semi-circle around the user"s primary monitor, with their spatial arrangement initially determined by the order in which users login to the OM server, but this could be changed by the user to reflect natural groupings based on external factors such as project roles or relationship types.

OMNI has two primary uses: 1) displaying others" gathering behavior, and 2) facilitating gathering information about others.

Displaying. For displaying others" behavior, two visual parameters are manipulated: physical distance and motion. Physical distance of a contact"s avatar from the user"s body in the real world (presumed to be at the center of the display, though this could be altered) is inversely related to OM focus level. Contacts with a higher focus level on the current user (i.e., those that are gathering information) appear closer to the user. Distance between a user and a contact with focus level zero is defined by the vertical height of the display (i.e., the maximum possible distance given screen size constraints). Distance decreases by 20% of the overall distance with each increase in focus level until the avatars are touching at focus level five.

In line with findings from [START_REF] Bartram | Moticons: detection, distraction and task[END_REF], we use motion to attract user attention in the event of change. As a contact increases their focus on the current user, the contact"s avatar gradually moves toward the user on the OMNI display. Movement rate correlates directly with the rate of increase in focus level and occurs at a linear rate. As focus level decreases, the avatar moves away from the center at a rate slower than the approach. Movement away occurs more slowly to minimize distraction in the event that a contact looks back at an avatar soon after looking away.

Gathering. When the current user wishes to gather information about a contact, they focus on that contact"s avatar (using a mouse, eye tracker, or other device with a social monitor). As they maintain focus on this avatar, more monitor information is displayed (see Figure 2) about that contact, reflecting the increase in focus level.

At this point, our social monitor and view architecture becomes quite powerful in supporting joint action and the notion of approach as a form of interaction. Consider the case where Alex wishes to interact with Bill. Alex looks at his OMNI display and sees that Bill is online. He then fixates on Bill"s avatar to get more information (Figure 2) about him. As Alex gets more information (and his focus level on Bill increases), Alex"s avatar will move toward Bill"s on Bill"s OMNI display. Bill notices this and looks at Alex"s avatar, thus increasing Bill"s focus level on Alex. As Alex"s avatar moves toward Bill, more information about Alex (from his monitors) will appear next to Alex"s avatar on Bill"s display. If Bill continues to look at this information, (i.e., as Bill"s focus level on Alex increases), Bill"s avatar on Alex"s display moves toward Alex. Alex knows that Bill has noticed him, and can watch his display to see if Bill continues to attend to him. With sustained attention, they could opt to start a conversation or simply be aware of each other"s interest.

Discussion

By focusing on both gathering and display of awareness information as interdependent components of the joint activity involved in negotiating mutual attention, OMF provides an extensible and general framework for experimenting with the management of interpersonal attention. It is the product of four years of development efforts, including three versions of OM. Here we discuss the implications of some key design decisions and assess the OMF"s technical properties.

Sensor Data for Both Gathering and Display

We began by noting that prior systems often considered gathering and display in isolation, but that in reality these behaviors often correlate or even overlap. One unique and important attribute of the OMF is that data from sensors are used for both gathering and display. This stands in contrast to previous systems, which have largely treated sensor data primarily as a way to gather information about others (e.g., [START_REF] Begole | Lilsys: Inferring Unavailability Using Sensors[END_REF][START_REF] Fogarty | Sensing from the Basement: A Feasibility Study of Unobtrusive and Low-Cost Home Activity Recognition[END_REF]). OMF"s sensor/monitor/focus level architecture is agnostic about who is "starting" an interaction or who is "observing" vs. "being observed." It uses sensor data to determine the focus of attention of both the observed party and the observer.

The implications of this approach are illustrated by the Eye Tracker sensor. If the eye tracker detects Alex looking at Bill"s avatar, this single awareness event triggers both an increase in the amount of information Alex sees about Bill (gathering), as well as a notification to Bill that this is taking place (display). In this way, gathering and display are coupled more naturally, and it is technically possible to support the interactive aspects of awareness discussed by Schmidt [START_REF] Schmidt | The problem with 'awareness[END_REF].

More research and experimentation, however, are needed to implement mechanisms for interaction via gathering and display in ways that will not confuse or overwhelm users. This is the focus of our current and future research.

Approaching, Interacting and Privacy

Our second key point was that approach is a form of interaction, and people should be notified when others gather information about them. While the OMF supports notification for joint action purposes, this does not mean that notification will occur every time information is accessed. From a privacy standpoint, OMF"s architecture supports but does not strictly enforce notification. Using Boyle"s [START_REF] Boyle | The Language of Privacy: Learning from Video Media Space Analysis and Design[END_REF] distinction between confidentiality and solitude, we think about monitors as protecting the confidentiality of detailed sensor data, while focus levels protect solitude by reducing the amount of information that is displayed at any one time.

The combination of these mechanisms means that there are two potential cases where gathering could occur without notification. The first of these results from possible variation in views between users. It is possible that a user could implement or select a view that, for design or other reasons, does not display instances of certain types of gathering by other users. In these cases, the user would not be notified when these types of gathering took place. Second, receiverside filtering of awareness events means that data are sent to clients regularly, but notification is provided to an observed user only when the data are accessed via a view. As data is stored on the client, it is theoretically possible that the user could hack the system for access in a way that does not result in notification. While these issues merit consideration, we do not feel they present a major threat to privacy because they seem unlikely to occur often in trusted groups of collaborators, and because the OMF provides other privacy controls that regulate what information is shared. Technical Assessment of OMF Generality. The OMF is general in that it can theoretically handle any type of awareness-based attention management in which users gather information about the displayed activity or status of their collaborators, and adjust or update their own activities based on this information. The framework can easily be extended to incorporate data from any software or hardware sensor providing a data stream that can be parsed by an OMF monitor. This could include information to support nonverbal or semi-synchronous communication, or those that provide additional verbal/synchronous channels (e.g., audio, video).

The framework architecture also supports easy development of novel mechanisms for increasing or decreasing focus. While our sample implementation increases focus level when another user"s avatar is the focus of attention, this could be extended to increase focus when users are focused on the same object (e.g., a document), or even when certain users are focused on each other.

Flexibility. The ideas behind OMF have been tested primarily with synchronous or semi-synchronous interaction, but the design of the framework is such that the architecture and general implementation approach could be adapted for use in other systems -other awareness / messaging systems, but also in a variety of other applications (e.g., shared editors, workflow systems, or document sharing).

The framework is flexible enough to be used in a variety of contexts, although we have primarily developed and tested it in a desktop/office environment. The general architecture, however, means that a group of users could be using individual clients with very different sensors, sensor managers, monitors and views; but these could all smoothly exchange data via the common structure of awareness events and focus level. In future we plan to adapt the system to mobile devices and sensors.

Performance. The nature of the OMF and current implementations means that the computation and bandwidth requirements for the framework are small. In particular, sensing, monitoring, and distribution of information functions of the OM framework are lightweight. We will consider performance further in future work.

Limitations and Future Work

There are several limitations that provide opportunities for substantial future work. One clear limitation in our assessment is the lack of a field or laboratory evaluation, and this is part of our immediate future plans. We do not focus on this here, but instead point to several other key questions. First, we provide an operational and conceptual framework for supporting joint attention management, but leave aside substantial questions about how sensor data should be parsed, interpreted and aggregated as the number of sensors and granularity increase. Parsing sensor data has received substantial attention in recent years (e.g., [START_REF] Begole | Lilsys: Inferring Unavailability Using Sensors[END_REF][START_REF] Fogarty | Predicting Human Interruptibility with Sensors[END_REF][START_REF] Olguin | Capturing individual and group behavior with wearable sensors[END_REF]) and we aim to incorporate and build on these techniques. Relatedly, we currently use a naïve approach to increasing focus levels that assigns equal weight to different modes of input. We aim to develop a more sophisticated model, defined within OMF, that reflects the nuance of cues and modes of social attention.

Conclusion

We have presented a joint action approach to interpersonal attention management in supporting awareness and informal interaction. In our approach, actions are assumed to occur in anticipation of or in response to acts by others. Our OpenMessenger Framework provides operational solutions for the problems of: 1) discerning focus of attention, and treating this differently when focus is on another person; 2) allowing for joint action both during and prior to conversational interaction; and 3) allowing for awareness of other users" presence and behavior. The software and examples are available at: http://collabtech.hci.cornell.edu/projects/openmessenger.php so that other researchers can further test and explore the idea of joint attention.

Figure 1 .

 1 Figure 1. Information flow in the OpenMessenger Framework

Figure 2 .

 2 Figure 2. Screen shots of two OMNI displays in use, with rows denoting what users Alex and Bill see. In Alex's view, as Alex focuses on Bill, a halo appears around Bill's avatar along with a line to indicate that he is being focused on. On Bill's screen Alex moves closer to Bill's avatar as Alex's focus level on Bill increases. (A) is the remote user's avatar, (B) is the local user, and (C) is sensor information about the remote user.

Figure 3 .

 3 Figure 3. A user seated in front of an OMNI display, with four remote contacts. The user is focused on the leftmost contact, as indicated by the line.

Table 1 . Coupling of Gathering and Display

 1

			Alex Gathers
			Displayed	Not Displayed
	Bill Gathers	Displayed Not Displayed	Face-to-Face Approach Some IM conversations	Some IM conversations Spying, covert looking

Table 2 : Example monitors and associated sensor managers in OMF.

 2

	Monitor	SensorManager Used (API)	Analysis
	• Typing Activity	Keyboard (DirectInput)	• # keys pressed since last event
	• Keypresses		• Record keys since last event
	• Screen Activity	Screen (Windows API)	• Compare successive screenshot frames
	• Current Window		• Record title of current focus window
	• Screen Contents		• Record screen image
	• Sound Presence	Microphone (DirectSound)	• Compare microphone level to thresholds
	• Sound Level		• Track level over time
	• Number of Speakers		• Analyze sound for voices
	• Overall Presence	Keyboard, Screen, Microphone • Time since last sensor change
	• Visual Presence	Webcam (DirectX)	• Compare successive frames for changes
	• Visual Changes		• Analyze frame for objects (e.g., people)
	• Number of People		• Record webcam image
	• Mouse Focus Target Mouse (UI toolkit)	• Determine avatar mouse is pointing at
	• Eye Focus Target	Eye Tracker (Custom API)	• Determine avatar the user is looking at
	• In Conversation	Chat Window (OMF events)	• Determine who the user is conversing with

Table 1

 1

		)
	0	Overall Presence
	1	Keyboard Activity, Idle Time
	2	Screen Activity
	3	Current Window Title
	4	Microphone Activity
	5	Current Gaze Focus

Table 3 . Example focus levels and corresponding monitors. Note that information display is cumulative, such that each level includes the lower levels as well.

 3

Note that the word "open" in OMF refers to open-plan offices, which were our inspiration for this work. We will happily share OMF source code, but "open" does not imply open-source.

Commentaire [J1]:Not sure this should be deleted.

Acknowledgements

We thank all who have helped develop OMF, especially Sigurd Teigen, Jeesung Na, Maryam Mustafa, Suchi Agicha, Oleg Krohkin, and Yang (Leon) Liu. This project is supported in part by the USDA Cooperative State Research, Education and Extension Service (Hatch Project #NYC-131439), the US National Science Foundation (#IIS-0942659), and the Institute for the Social Sciences at Cornell University.


Piotr Chynał
email: piotr.chynal@pwr.wroc.pl

Jerzy M Szymański

Remote Usability Testing Using Eyetracking

Keywords: Eyetracking, Usability, Remote Usability Testing, Human-Computer Interaction

In the paper we present a low cost method of using eyetracking to perform remote usability tests on users. Remote usability testing enables to test users in their natural environment. Eyetracking is one of the most popular techniques for usability testing in the laboratory environment. We decided to try to use this technique in remote tests. We used standard web camera with freeware software. Our experiment showed that such method is not perfect, but it could be a good addition to the standard remote tests, and a foundation for further development.

Introduction

In modern usability testing, remote tests are becoming more and more popular. Because of lack of time and money, companies are looking for alternatives for standard tests with users. Those tests require a place to test, gathered users, moderator and equipment. This situation lead to idea of remote usability testing. Its main goal is to test users in their natural working place, without any sophisticated equipment. Users in their own environment are behaving more naturally, like they would normally do while using the given website [START_REF] Moha | Towards a Platform for Usability Remote Tests via Internet[END_REF]. Moreover we do not need to gather all the users at one time, we can work with them when they have the time to take part in the test. Also we can have participants from different cities or even countries that would normally not visit our laboratory. Furthermore comparisons of the results of standard laboratory and remote testing have shown that participants find the same usability issues on tested pages with both methods [START_REF] Brush | A Comparison of Synchronous Remote and Local Usability Studies for an Expert Interface[END_REF], [START_REF] Oztoprak | Field versus Laboratory Usability Testing: a First Comparison[END_REF], [START_REF] Tullis | An Empirical Comparison of Lab and Remote Usability Testing of Web Sites[END_REF].

So far remote usability tests are evaluated using standard methods such as remote surveys and video conferences [START_REF] Andreasen | What Happened to Remote Usability Testing? An Empirical Study of Three Methods[END_REF], [START_REF] Petrie | Remote Usability Evaluations with Disabled People[END_REF]. We can also use some traditional laboratory usability testing methods in remote environment [START_REF] Scholtz | Adaptation of Traditional Usability Testing Methods for Remote Testing[END_REF]. One of the most popular tool for standard usability testing is eyetracking [START_REF] Duchowski | Eye tracking methodology: Theory and practice[END_REF], [START_REF] Mohamed | A history of eye gaze tracking[END_REF]. The biggest drawback of eyetracking technology is its cost. The equipment is very expensive and companies which perform such tests usually charge a lot of money for such tests. We tried to perform remote usability test with eyetracking, using low cost hardware, such as ordinary web camera and free software.

Tools used in experiment

The main equipment used during this research was a casual web cam Logitech Quick Cam Pro 9000. Software used in this experiment was: 1. Piotr Zieliński Opengazer1 .Net port made by Przemysław Nibyłowicz2 . This application enables gazetracking using ordinary webcam. It is freeware open source software. After selecting feature face points on the video image, user calibrates the program by looking at the appearing squares. Next, when calibration is finished, line of gaze is tracked by the program. We slightly modified this application, so it stores the data of all the points that users is looking at in the text file (Fig. 1).

Fig. 1.

Application after the calibration process. Blue square shows where the participant is looking and in the background we have a window that shows the coordinates of the gaze.

The points in the application have coordinates from -1 to 1 (float variable), so to store them as actual points on the screen, we are transforming them as shown below:

int okoX = (int)((eventArgs.EyeX + 1.0f) * (w / 2)); int okoY = (int)((eventArgs.EyeY + 1.0f) * (h / 2));
We add 1.0 to the float value (eventArgs.EyeX or eventArgs.EyeY) and then we multiply this value by height (h) or width (w) of the screen divided by two. Then the value is parsed to integer. Furthermore we needed to transform those points so they could be used in the heatmap generator application (point 2 below). We needed to transform the screen parameters to image parameters:

int imageX = (int) ((1024*okoX)/w); int imageY = (int) ((768*okoY)/h);
We received new coordinates for the image by calculating the given point with the image size and screen size. 2. JavaScript application for creating heatmaps 3 . Created by Michael Dungan, released under the MIT license. This application creates heatmaps on images taking mouse movement as an input data (Fig. 2). However it has the functionality to import points, so we used it to generate heatmaps for our test. It takes as parameters the offset of the picture and a "mousemove mask" for smooth rendering, so we omitted this last parameter and put "3" as a default value (center). Additionally we needed to run the browser in full screen mode, so the coordinates of the points calculated in Opengazer .Net port application would be adequate.

Fig. 2 Application for creating heatmaps from mouse input as well as from imported points (Source for koala bear picture: Microsoft Windows 7 sample images).

3. Real VNC 4 -simple application for desktop sharing.

4. Skype 5 -popular application that enables talking through microphone over the internet. 5. CamStudio 6 -simple application for recording actions from computer screen.

Experiment

The goal of our experiment was to try to perform remote usability eyetracking test and analyze if our method is suitable for future development. We tested it with five users, on different computers but with the same webcam (Logitech Quick Cam 9000).

During the test, first thing that we did was to connect to user"s computer via Skype and Remote VNC. After that we instructed the user how to position the camera and how to calibrate the eyetracking application. This was the toughest part for all the users, because the calibration process in this application requires a lot of patience. Application sometimes crashes, and calibration often needs to be repeated few times until it is correctly set. When the results of calibration were satisfactory we asked our user to start CamStudio and perform some simple actions on google.com web page, such as to log in. While user was working with the page we were able to see where he is looking, because small blue square was showing that position (Fig. 3). It is a very helpful thing for the moderator, because he can observe at which elements the user is looking during the test and have more control over what user is doing. After the experiment we took the coordinates of the user's gaze on the screen (stored in a text file) and we imported them to the heatmap application. It allowed us to create some simple heatmaps for our test (Fig. 4). To sum up the experiment, our method of remote usability eyetracking test can provide the moderator with:

1. Verbal remarks and comments of the tested users, as in standard remote test.

2. Observation of user actions and where user was looking during the test via Remote VNC. 3. Recordings from the point 2 provided by CamStudio. 4. Text file with the coordinates of the points on the screen on which the user was looking during the test. Those points can be used to generate some visual reports.

Conclusions and future work

Our experiment has shown that it is possible to perform remote usability eyetracking tests. Moreover we obtained a lot of valuable data that could be processed for usability analysis. However our solution has some drawbacks that need to be addresses. First of all user has to perform many actions before the test, such as setting up the camera and calibrating the eyetracking software. In future we will try to improve this process, so the users will not need to perform so many operations. Secondly, using a web cam and Opengazer .Net port we need to perform calibration many times to obtain eyetracking data, which precision can be compared with professional eyetrackers in the laboratory environment. So far we managed to obtain the precision which is about two times worse than in professional eyetrackers, so there is still room for further development here. We will also try to improve calibration process, so it would be faster and more effective. Last thing is that we needed to put a lot of effort into creat-ing a single heatmap for our test. We need to create an application that could quickly transform received points into heatmaps, gaze plots and other useful reports.

In conclusion our method needs a lot of improvements, but it is definitely a good starting point for creating a remote usability eyetracking testing methodology and a platform for such tests. Remote user testing is the future of usability tests, so introducing new methods and techniques to improve them is a very beneficial thing.

Fig. 3 .

 3 Fig. 3. Google.com website during the test, small blue square shows the position of the participants gaze.

Fig. 4 .

 4 Fig. 4. Simple heatmap for google.com website created with JavaScript heatmap generator.

http://www.inference.phy.cam.ac.uk/opengazer/

http://netgazer.sourceforge.net

https://github.com/xxx/heatmap

http://www.realvnc.com

http://www.skype.com

http:///www.camstduio.org

Acknowledgements

This work has been partially supported by the Polish Ministry of Science and Higher Education within the European Regional Development Fund, Grant No. POIG.01.03.01-00-008/08.