Katherine Vogt
email: [kvogt@elon.edu

Lauren Bradel
email: [lbradel1@cs.vt.edu

Christopher Andrews

Chris North
email: north@cs.vt.edu

Alex Endert
email: aendert]@cs.vt.edu

Duke Hutchings
email: dhutchings]@elon.edu

Co-located Collaborative Sensemaking on a Large High-Resolution Display with Multiple Input Devices

Keywords: Visual analytics, sensemaking, co-located, CSCW, large highresolution display

This study adapts existing tools (Jigsaw and a text editor) to support multiple input devices, which were then used in a co-located collaborative intelligence analysis study conducted on a large, high-resolution display. Exploring the sensemaking process and user roles in pairs of analysts, the two-hour study used a fictional data set composed of 50 short textual documents that contained a terrorist plot and subject pairs who had experience working together. The large display facilitated the paired sensemaking process, allowing teams to spatially arrange information and conduct individual work as needed. We discuss how the space and the tools affected the approach to the analysis, how the teams collaborated, and the user roles that developed. Using these findings, we suggest design guidelines for future colocated collaborative tools.

Introduction

As analysts sort through the growing amounts of data every day, tools that can display the information in a useful manner without overwhelming their sensemaking process are a beneficial component of their workflow. Visual analytics works to improve analysts' experience in their work and productivity. As such, exploring what collaborative visual analytics may contribute to this challenge has become a key area of research within visual analytics [START_REF] Thomas | Illuminating the Path: The Research and Development Agenda for Visual Analytics[END_REF]. Through the support of collaboration within the analytic process, designers can improve the effectiveness though leveraging various social and group dynamics [START_REF] Heer | Design considerations for collaborative visual analytics[END_REF]. Various design guidelines and structured collaborative techniques exist [START_REF] Heuer | Structured Analytic Techniques for Intelligence Analysis[END_REF], but, as the cul-ture of intelligence analyst working within an agency can be described as "competitive", where sharing of knowledge may adversely affect their job security, collaboration occurs at a much less formal level, if at all [START_REF] Chin | Exploring the analytical processes of intelligence analysts[END_REF]. The study presented here does not present a formal collaborative method, but places the users in a setting where information and knowledge is inherently shared through the use of a shared workspace.

Large, high-resolution workspaces (such as the one shown in Fig. 1) are beneficial to intelligence analysis in that they allow for spatial information organization to act as an external representation or memory aid [START_REF] Andrews | Space to think: large high-resolution displays for sensemaking[END_REF]. This advantage was shown to help individual intelligence analysts in their task, in that they were able to spatially organize and reference information. This work explores how such a workspace, allowing for these spatial strategies, can impact the strategy and workflow of a team (of 2) users working collaboratively on an intelligence analysis task. In this environment, we provide users with a social setting in which to perform their analysis, and a shared representation in which to organize their thoughts. We analyze their process in terms of their activities and roles exemplified during their task, their use of space, and level of collaboration.

In such co-located settings (versus remote settings), it has been shown that teams experience a greater quality of communication because of subtle physical interaction cues and a stronger trust that develops with the shared experience [START_REF] Waltz | The Knowledge-Based Intelligence Organization[END_REF]. Also, given that analysts often work with large collections of electronic documents, it is worthwhile to explore how the design of tools on large, high-resolution displays could facilitate collaboration during analysis. Further, if this environment supports collaborative work, then the ability to make sense of documents develops great potential. To investigate the collaborative use of a large, high-resolution display environment, we have completed an exploratory study of two visual analytic tools: Jigsaw [START_REF] Stasko | Jigsaw: supporting investigative analysis through interactive visualization[END_REF], and a simple multi-window text editor. The study we present involves synchronous, co-located collaborative sensemaking. Here, we define co-Fig. 1. Study setup, two users with their own input devices in front of the large display located work as multiple users working each with his or her own input devices (mouse and keyboard) on the same computer display.

Related Work

Design tensions exist in collaborative tools between "individual control of the application, and support for workspace awareness" [START_REF] Gutwin | Design for individuals, design for groups: tradeoffs between power and workspace awareness[END_REF]. Some previous groupware tools have had difficulty achieving a balance between these extremes, either supporting the group through consistent view sharing ("What You See Is What I See" -WYSIWIS) or the individual through relaxed view sharing [START_REF] Stefik | Beyond the chalkboard: computer support for collaboration and problem solving in meetings[END_REF]. However, Gutwin and Greenberg feel that a solution to this tension exists, stating that "the ideal solution would be to support both needs show everyone the same objects as in WYSIWIS systems, but also let people move freely around the workspace, as in relaxed-WYSIWIS groupware" [START_REF] Gutwin | Design for individuals, design for groups: tradeoffs between power and workspace awareness[END_REF]. Single display groupware provides an interface to achieve this balance.

Single Display Groupware (SDG) concerns face-to-face collaboration around a single shared display [START_REF] Stewart | Single display groupware[END_REF]. Early SDG systems include Liveboard [START_REF] Elrod | Liveboard: a large interactive display supporting group meetings, presentations, and remote collaboration[END_REF], Tivoli [START_REF] Pedersen | Tivoli: an electronic whiteboard for informal workgroup meetings[END_REF], and the Digital Whiteboard [START_REF] Rekimoto | A multiple device approach for supporting whiteboard-based interactions[END_REF]. When compared to co-located multi-display groupware, SDG resulted in increased collaborative awareness [START_REF] Wallace | Investigating teamwork and taskwork in single-and multi-display groupware systems[END_REF]. Stewart et al. continued to investigate SDG systems in subsequent work ( [START_REF] Stewart | When two hands are better than one: enhancing collaboration using single display groupware[END_REF][START_REF] Stewart | Single display groupware: a model for copresent collaboration[END_REF]). They proposed that the multi-user nature of SDG systems on early displays with limited screen size "may result in reduced functionality compared with similar single-user programs" [START_REF] Stewart | Single display groupware: a model for copresent collaboration[END_REF], although this concern can be alleviated by increasing the physical size (and resolution) or the SDG display.

SDG systems using multiple input devices have been found to increased interaction between participants and keep participants "in the zone" [START_REF] Stewart | When two hands are better than one: enhancing collaboration using single display groupware[END_REF]. Providing a separate mouse and keyboard to each participant has been shown to decrease communication and allow for users to complete more work in parallel than if they were restricted to a single mouse and keyboard [START_REF] Birnholtz | An exploratory study of input configuration and group process in a negotiation task using a large display[END_REF]. However, multiple input devices provide the benefit of allowing reticent users to contribute to the task [START_REF] Robinson | Collaborative Synthesis of Visual Analytic Results[END_REF][START_REF] Rogers | Equal Opportunities: Do Shareable Interfaces Promote More Group Participation Than Single User Displays? Human-Computer Interaction[END_REF]. As a result of our desire to keep participants in the "cognitive zone" [START_REF] Green | Building and applying a human cognition model for visual analytics[END_REF], given the cognitively demanding nature of sensemaking tasks, we chose to implement multiple input devices for our set-up.

The sensemaking process has been illustrated by Pirolli and Card (Fig. 2) to outline the cognitive process of "making sense" of documents throughout their investigation in order to produce a cohesive and coherent story of interwoven information found across document sources [START_REF] Pirolli | The Sensemaking Process and Leverage Points for Analyst Technology as Identified Through Cognitive Task Analysis[END_REF]. This process can be broken down into two board categories: foraging and sensemaking. The foraging loop involves extracting and filtering relevant information. The sensemaking loop represents the mental portion of sensemaking where a schema, hypothesis, and presentation are iteratively developed. The analyst is not restricted to a single entry point to this loop, and instead can enter at the top or bottom before looping through the various steps [START_REF] Pirolli | The Sensemaking Process and Leverage Points for Analyst Technology as Identified Through Cognitive Task Analysis[END_REF]. The sensemaking process has been studied and observed on large, high-resolution displays as well as multiple monitor set-ups for individual users [START_REF] Andrews | Space to think: large high-resolution displays for sensemaking[END_REF][START_REF] Stasko | Jigsaw: supporting investigative analysis through interactive visualization[END_REF][START_REF] Plaisant | Evaluating Visual Analytics at the 2007 VAST Symposium Contest[END_REF]. Paul and Reddy observed, through an ethnographic study concerning collaborative sensemaking of healthcare information, that collaborative sensemaking should focus on the following factors: prioritizing relevant information, the trajectories of the sensemaking activity, and activity awareness [START_REF] Paul | Understanding together: sensemaking in collaborative information seeking[END_REF].

Collaborative sensemaking has also been studied in terms of web searches [START_REF] Paul | CoSense: enhancing sensemaking for collaborative web search[END_REF][START_REF] Morris | WeSearch: supporting collaborative search and sensemaking on a tabletop display[END_REF], as well as remote collaborative sensemaking for intelligence analysis [START_REF] Pioch | POLESTAR: collaborative knowledge management and sensemaking tools for intelligence analysts[END_REF]. Furthermore, collaborative sensemaking has been observed in co-located tabletop settings [START_REF] Tobiasz | Lark: Coordinating Co-located Collaboration with Information Visualization[END_REF][START_REF] Isenberg | Collaborative Brushing and Linking for Co-located Visual Analytics of Document Collections[END_REF][START_REF] Isenberg | An exploratory study of co-located collaborative visual analytics around a tabletop display[END_REF], although, to the best of our knowledge, co-located collaborative sensemaking applied to intelligence analysis has not been investigated on large, high-resolution vertical displays.

User performance on simple tasks, such as pattern matching, has been shown to improve when using a large, high-resolution vertical display when contrasted with a standard single monitor display [START_REF] Ball | Move to improve: promoting physical navigation to increase user performance with large displays[END_REF]. In addition to quantitative improvement, users were observed using more physical navigation (e.g. glancing, head/body turning) than virtual navigation (e.g. manually switching windows or tasks, minimizing/maximizing documents) when using large, high-resolution displays, such as the one shown in Fig. 1.

Andrews et al. expanded the benefits of using large, high-resolution display to cognitively demanding tasks (ex. sensemaking) [START_REF] Andrews | Space to think: large high-resolution displays for sensemaking[END_REF]. We chose to use these displays to explore collaborative sensemaking on large vertical displays, especially the user roles that develop throughout the sensemaking process and how the sensemaking process is tackled by teams of two. Fig. 2. Adapted from sensemaking loop, Pirolli and Card [START_REF] Pirolli | The Sensemaking Process and Leverage Points for Analyst Technology as Identified Through Cognitive Task Analysis[END_REF] We have conducted an exploratory study examining the collaborative sensemaking process on a large, high-resolution display. Teams of two were asked to assume the role of intelligence analysts tasked with analyzing a collection of text documents to uncover a hidden plot against the United States. The teams were provided with one of two tools, Jigsaw or a multi-document text editor, with which they were asked to conduct their analysis. While each team was told that they were expected to work collaboratively, the nature of that collaboration was left entirely up to the participants.

Participants

We recruited eight pairs of participants (J1-J4 used Jigsaw, T1-T4 used the text editor). All pairs knew one another and had experience working together prior to the study. Six of the eight pairs were students and the other two pairs consisted of research associates and faculty. There were four all male groups, one all female, and three mixed gender. Each participant was compensated $15 for participation. As a form of motivation, the solutions generated by the pairs of participants were scored and the participants received an additional financial award for the four highest scores. The rubric for evaluating the participants' verbal and written solutions was based on the strategy for scoring Visual Analytics Science and Technology (VAST) challenges [START_REF] Plaisant | Evaluating Visual Analytics at the 2007 VAST Symposium Contest[END_REF]. The participants earned positive points for the people, events, and locations related to the solution and negative points for those that were irrelevant or incorrect. They also received points based on the accuracy of their overall prediction of an attack.

Apparatus

Each pair of users sat in front of a large display consisting of a 4x2 grid of 30" LCD 2560x1600 pixel monitors totaling 10,240x3,200 pixels or 32 megapixels (Fig. 1). The display was slightly curved around the users, letting them view the majority, if not all, of the display in their peripheral vision. A single machine running Fedora 8 drove the display. A multi-cursor window manager based on modified versions of the IceWM and x2x was used to support two independent mice and keyboards [START_REF] Wallace | Virtually shared displays and user input devices[END_REF]. Thus, each user was able to type and use the mouse independently and simultaneously in the shared workspace. This multi-input technology allowed two windows to be "active" at the same time, allowing participants to conduct separate investigations if they chose. A whiteboard, markers, paper, and pens were also available for use. These external artifacts were provided as a result of a pilot study where participants explicitly requested to use the whiteboard or write on sheets of paper. Each participant was provided with a rolling chair and free-standing, rolling table top holding the keyboard and mouse so that they could move around if they chose to do so. The desks and chairs were initially positioned side-by-side in the central area of the screen space.

Analytic Environment

During this exploratory study, four of the pairs (J1-J4) examined the documents within Jigsaw, a recent visual analytics tool, while the other four (T1-T4) used a basic text editor, AbiWord [START_REF] Lachowicz | [END_REF], as a contrasting tool. This text editor allows highlighting individual document sections and the ability to edit existing documents or create text notes. Teams using this text editor were also provided with a file browser in which they could search for keywords across the document collection. Jigsaw [START_REF] Stasko | Jigsaw: supporting investigative analysis through interactive visualization[END_REF][START_REF] Kang | Evaluating visual analytics systems for investigative analysis: Deriving design principles from a case study[END_REF] is a system that has been designed to support analysts; it visualizes document collections in multiple views based on the entities (people, organizations, locations, etc.) within those documents. It also allows textual search queries of the documents and entities. The views are linked by default so that exploring an entity in one visualization will simultaneously expand it in another. This feature is controlled by the user and can be turned on or off within each view. We were not able to change Jigsaw's source code to allow windows to be linked separately for each participant, therefore all Jigsaw views were connected unless the linking feature was disabled by the participant teams. Jigsaw can sort documents based on entity frequency, type, and relations. This information can be displayed in many different ways, including interactive graphs, lists, word clouds, and timelines. Jigsaw also comes equipped with a recently added Tablet view where users can record notes, label connections made between entities, identify aliases, and create timelines. As a result of the complexity of the visualizations available in Jigsaw, pairs using this visual analytics tool were given a thirty minute tutorial prior to the start of the scenario, while pairs using the text editor only required a five minute tutorial.

Task and Procedure

After a tutorial on Jigsaw or the text editor with a sample set of documents, each pair was given two hours to analyze a set of 50 text documents and use the information gathered to predict a future event. This scenario comes from an exercise developed to train intelligence analysts and consists of a number of synthetic intelligence reports concerning various incidents around the United States, some of which can be connected to gain insight into a potential terrorist attack. This same scenario was also used in a previous study evaluating individual analysts with Jigsaw [START_REF] Kang | Evaluating visual analytics systems for investigative analysis: Deriving design principles from a case study[END_REF]. Following the completion of the scenario, each participant filled out a report sheet to quantitatively assess their individual understanding of the analysis scenario, then verbally reported their final solution together to the observers. Finally, individual semi-structured interviews were conducted where each participant commented on how they solved the scenario, how this involved collaboration, and their sense of territoriality.

Data Collection

During each scenario, an observer was always present taking notes. Video and audio of every scenario, debriefing, and interview was recorded. The video was coded using Pe-CoTo [35]. We also collected screenshots in fifteen second intervals and logged mouse actions and active windows. The screenshots played two roles in our analysis. Their primary role was to allow us to "play back" the process of the analysis so that we could observe window movements and the use of the space. We also examined the screenshots computationally, counting background pixels to compute the amount of whitespace at each moment in time. This allowed us to quantify how much of the display was actually used by each team over the course of their analysis. Furthermore, we applied the previously described point system in order to evaluate the accuracy of their debriefing, providing a way to quantitatively measure their performance. The scores can be seen below in Table 1.

There was no significant difference between overall performance between the Jigsaw and Text Editor tool conditions. 4 Analysis

User Activities

Each group exhibited a variety of activities depending on their amount of progress to achieve a satisfactory solution. After analyzing the video, interviews, and solution reports, we have concluded that five major activities that were used by the participants, which together formed a strategy for analyzing the data. These were not usually explicitly identified by the participants, but rather tasks that the participants naturally took on in order to uncover the underlying terrorist plot. The five activities are extract, cluster, record, connect, and review and will be described in greater detail below. Although each group exhibited the execution of each activity (one exception with cluster which we will discuss later), the groups used different methods to implement that activity that were often based on the interface condition (Jigsaw or text editor) of the group [Table 2].

Extract. The groups had no starting point or lead to begin with -just fifty text documents and the knowledge that there was a terrorist threat to the nation. Therefore, they needed to familiarize themselves with the information presented within the documents and then extract that which seemed important. In Jigsaw, the visualizations allowed for participants to begin this process by looking at frequently occurring entities and the other entities and documents to which they connected. With the text editor, these features were not available therefore the participants were forced to open and read each document. They then all used color-coded highlighting to distinguish entities and/or important phrases. The coloring scheme was decided upon by the participants, whereas Jigsaw maintains a set color scheme for entities. In the text editor groups, the subjects opened documents in consistent locations to read them, but soon moved the opened documents into meaningful clusters (see next activity). The Extract activity required little display space to complete in either study condition. This activity was done together in some groups with both participants simultaneously reading the same document and in parallel in others with each participant reading half of the documents, often split by document number.

Cluster.

With the text editor, all of the groups found a need to cluster and organize the documents. The groups clustered by grouping the document windows by content in the space (they resembled piles), using whitespace between clusters as boundaries. The clusters eventually filled the display space, allowing the participants to view all documents at once in a meaningful configuration. Even when only one partner organized the documents into clusters, the other partner could easily find documents relevant to a certain topic due to their agreed upon clustering scheme (e.g. chronological order, geographical as shown in Fig. 1). Most text editor groups used the multi-mouse functionality to simultaneously organize the display space. Three of the four groups eventually re-clustered their documents after some analysis. The cluster activity as defined above (spatially arranging document windows) was not present in any of the Jigsaw groups, because Jigsaw organizes the enti- ties and documents through its various functionalities. Many Jigsaw groups, however, clustered relevant entities within their Tablet views, giving spatial meaning to the information recorded.

Record. Recording important information proved to be a useful strategy for all groups. Through interviews the participants revealed that this not only served as a memory aid, but also a way to see what events, dates, people, and organizations related. In the scenarios with the text editor, with two of the groups using the whiteboard and three using scrap paper (one used both), all groups found a need to use an external space to record important information regardless of how much of the display was filled by clusters. This allowed them to preserve the cluster set-up and keep the documents persistent. Three of the Jigsaw groups used the Tablet view to take notes and one group used paper and the whiteboard. Thus all participants devoted a separate space to keep track of pertinent information. Groups also recorded important information verbally to alert their partner to a potential lead, allowing their partner to create a mental record.

Connect.

In order to make connections and look for an overall plot, the Jigsaw participants would often loop through the list view, document view, and the Tablet, connecting the information they discovered. Two groups worked on this separately and two did this together. With the text editor, participants searched for entities and reread their notes. In comparison to their discourse during the other activities, the groups were more talkative when making connections. Text editor group T1 cleared a screen to use as a workspace for their current hypotheses. They opened relevant documents in their workspace and closed irrelevant documents or documents from which they had extracted all information. In all text editor cases, the meaning conveyed by clustered documents on the display was helpful in drawing connections.

Review. This appeared to be a very important element in the groups' analyses. Often when one or both partners reread a document for the second, third, or even fourth time, it Fig. 1. Geographical clustering of documents on the large display screen, done by group T4 (T4-B, the forager, arranged the space while T4-A, the sensemaker, instructed document placement)

took on a new meaning to them after they understood the greater context of the scenario. This element of review could also help the participants as they worked to Connect. Two of the Jigsaw groups chose to search for unviewed documents to ensure that they had encountered all potentially important information. Two of the text editor groups began closing windows after they had reread them. Sometimes this was because the document was considered irrelevant. For example, group T3 moved unrelated documents to what they called the "trash window". They later reread all of the trash window documents and closed those which still seemed irrelevant. The Review activity also included discussing current and alternative hypotheses.

While the activities listed in the table can be loosely defined in this sequential order, the order is certainly not set nor were they visited only once within each scenario. Rather, there was often rapid but natural movement between these activities and their methods depending on the current needs of the analysis. In particular, the middle three activities were present many times throughout the study. Extract was only necessary during the first part of each scenario and review was usually only seen after a significant portion of the first activity had been completed.

Comparison between sensemaking loop and activities

The processes we observed closely reflect the Pirolli and Card [START_REF] Pirolli | The Sensemaking Process and Leverage Points for Analyst Technology as Identified Through Cognitive Task Analysis[END_REF] sensemaking model (Fig. 2) which was developed for individual analysts. We have found that it may also generally be applied to collaborative pairs, although the loop is utilized differently because of the roles that developed. Extract and cluster relate to steps two through seven. The Evidence File and Schema steps were combined by the pairs due to the available display space. They were able to sort evidence into a meaningful schema by placing documents in different areas of the display. Record is very similar to schematizing and connect is a part of developing hypotheses. Review does not directly map to one stage of the sensemaking loop, but rather it is the equivalent of moving back down the loop, analyzing previous work, and returning to the shoebox and evidence file. Note that External Data Sources is not mentioned here because the participants were only presented with fifty documents so we are assuming that prior analysis has moved through this step. The cumulative Presentation directly links to the debriefing following the scenario.

While the activities described above and the sensemaking loop hold parallel ideas, we do want to distinguish the two concepts. The overall strategy we propose has been condensed to five activities as a result of the collaboration and space. Additionally, we have given the idea of review new emphasis. This is a very important element in the sensemaking process, but is not explicitly identified in the sensemaking loop.

All of the activities, excluding Cluster, were present in both scenarios. This is notable considering the vast differences of the scenarios based on tool type. Since the activities we observed correspond to the Pirolli and Card sensemaking model [START_REF] Pirolli | The Sensemaking Process and Leverage Points for Analyst Technology as Identified Through Cognitive Task Analysis[END_REF], with the primary difference in user behavior being the tool-specific methods adopted to fulfill those activities, we propose that these activities are very likely to be universal.

User Roles

All groups divided the responsibilities of the collaborative sensemaking task. The roles could be observed during the study because of actions and conversation, but they were also evident during the interviews following the study. Five of the eight groups established clearly defined collaborative roles, while the remaining three worked more separately and individually (Fig. 5). This appeared to be because the three groups were going through the steps of the analysis independently, but in parallel. Therefore various team-related roles and responsibilities in the analysis were less likely to develop.

For the five groups who established clearly defined roles, the two broad roles we identified through this analysis are sensemaker and forager. These high-level roles were primarily established after a considerable amount of the investigation had been completed, normally after the half-way point of the study session. Primarily, the sensemaker tended to be the dominant partner, often dictating what the forager did. This role included standing, writing on the whiteboard, using their hand to point to information (instead of using their cursor), and rarely using their cursor, instead requesting the forager to perform various activities. The forager's role consisted of questioning the current hypotheses, finding information, and maintaining a better awareness of where the information was located. For Overall Score example, the sensemaker would request actions such as "can you open [a particular document]?" and the forager would perform the action. These two roles closely match the two primary sub-loops (Fig. 2) in the Pirolli and Card model [START_REF] Pirolli | The Sensemaking Process and Leverage Points for Analyst Technology as Identified Through Cognitive Task Analysis[END_REF]. The first loop, foraging, involves sorting through data to distinguish what is relevant from the rest of the information. The second loop, sensemaking, involves utilizing the information pulled aside during the foraging process to schematize and form a hypothesis during the analysis. Thus, the sensemaker was more concerned with the synthesizing of the information, while the forager was more involved in the gathering, verifying, and organizing of the information. While the sensemaker and forager each spent the majority of their time at their respective ends of the loop, they did not isolate themselves from the rest of the sensemaking process.

To illustrate the distribution of responsibilities prompted by the roles adopted, we will explain in detail two of the pairs where the participants formed distinct roles. These are the two groups in which the roles are most clearly defined, and are therefore the most interesting to talk about.

In group T1, the team with the second-highest score, both participants spent the first hour foraging (i.e., exposing, clustering) for information while taking a few breaks to engage in sensemaking activities (i.e., connecting). Participant T1-A (the subject who sat on the left) at times led T1-B's (the participant who sat on the right) actions by initializing activities or finalizing decisions. At the 68-minute mark, participant T1-B moved to the whiteboard (never to return to the computer input devices) and established a clear, dominant role as sensemaker while T1-A continued to forage for information. Specifically, T1-A organized the documents, searched, and provided dates, locations, relevant events, etc. but T1-B drew a picture connecting the relevant events working to form a hypothesis and requested information from T1-A. T1-B began focusing on Record and Connect, but they both engaged in the Review activity together. The Review activity was interspersed throughout the scenario as pieces of information inspired participants to revisit a document. During the interviews, T1-B revealed that he wanted to build a chart or timeline to organize their thoughts better. Although interviewed separately, they seemed to have similar views on their roles. T1-B stated, "I basically just tried to stand up there and construct everything while he finds evidence," while T1-A said, "I was just trying to feed him the data, that was my skill, find it, and he can put it in a flow chart." The other pair is group T4, the group with the highest score, where T4-A was the sensemaker and T4-B the forager. Again, the sensemaker is the participant (T4-A) who built a picture on the whiteboard, meaning he drove Record and Connect. In fact, T4-A barely touched his mouse after the first fifteen minutes of the scenario. He only had 104 mouse clicks while T4-B had 1374. They worked through Extract and Cluster together, but T4-A verbally dictated the clustering while T4-B controlled it with the mouse. While T4-A worked on the whiteboard, T4-B fed him details as needed. As T4-A stated, "We ended up splitting the tasks into organization and story-building… I would say I built most of the story." Both participants worked on the Review activity, but during this T4-B questioned T4-A's hypotheses which forced him to justify and support his thoughts.

Multi-Mouse Role in Sensemaking

The presence of multiple mice allowed low-level actions (such as highlighting and window dragging) to happen more quickly because they could be performed simultaneously. The mice also gave the users equal control of the workspace.

Groups with distinctly developed user roles tended to use the mice unevenly after the roles became established. In the extreme case, as stated above, participant group T4 essentially used one mouse throughout the sensemaking scenario. T4-B sat in front of the display while T4-A instructed her interactions with the display. She became the computer's driver while her partner, T4-A, was the navigator. This is not a new method of interaction [START_REF] Pickens | Algorithmic mediation for collaborative exploratory search[END_REF], however, it is interesting that T4-A abandoned his own mouse in favor of instructing his partner.

Teams that did not develop clearly defined roles tended to use the mouse simultaneously instead of taking turns. This also occurred in teams with clearly established roles before these roles were assumed. Simultaneous mouse interaction varied based on how the pairs collaborated. If they worked closely together, they used their mice to complete actions in tandem, such as highlighting documents or arranging them on the screen. However, if they worked more independently, the simultaneous clicking represented individual investigations being conducted. [Figure] shows that there are periods of oscillating mouse activity as well as stretches of just one person interacting with the display. These extended peaks represent times that the participants came together to discuss their current findings, hypotheses, or questions, and one participant would find relevant information on his or her screens. Communication became an important converging point between the participants.

Collaboration Levels

The amount of time spent working closely together appears to have impacted the scores. We applied the video coding code set from Isenberg et al. [START_REF] Isenberg | An Exploratory Study of Co-located Visual Analytics around a Tabletop Display[END_REF] to determine how much time was spent closely coupled (collaborating together) versus loosely coupled (working individually). Closely coupled is defined by Isenberg et al. active discussion, viewing the same document, or working on the same specific problem [START_REF] Isenberg | An exploratory study of co-located collaborative visual analytics around a tabletop display[END_REF]. Loosely coupled is defined as working on the same general problem, different problems, or being disengaged from the task. Upon graphing this data (Fig. 6), two clusters appear separating the high-scoring groups from the low-scoring ones. The high scoring cluster worked closely over 89% of the time spent on the scenario. The low scoring cluster only worked closely in between 42% and 67% of the time. All but one group at least collaborated closely during the remaining half hour of the scenario in order to synthesize their hypotheses. The correlation coefficient between the amount of time spent collaborating closely and score is .96105, suggesting that there is a strong correlation between these variables. This reinforces the result from [START_REF] Isenberg | An exploratory study of co-located collaborative visual analytics around a tabletop display[END_REF] that strongly links collaboration levels with performance.

Design Implications

Viewing all documents simultaneously appeared to be an effective strategy, given the added space provided by the large display. All 50 documents comfortably fit into userdefined clusters with whitespace serving as a partition between clusters. No Jigsaw groups chose this approach, instead relying on the specialized views available. Visual analytics tools designed for large displays should take this into consideration by allowing users to open many documents and flexibly rearrange the clusters as needed. This may not be feasible after the document collection becomes large enough, in which case a tool such as Jigsaw would be valuable in narrowing down the document collection. We recommend that developers combine these two analysis approaches to perform well on all document collection sizes.

Because the highest scoring groups had clearly defined user roles while the lowest scoring groups did not, we recommend that co-located collaborative visual analytics tools support the division of responsibilities. One way to achieve this would be to implement specialized views for foragers and sensemakers.

Some sensemakers stood and used a physical whiteboard to record their thoughts. All text editor groups used the whiteboard or paper to record their thoughts. One Jigsaw group used the whiteboard while the rest used Jigsaw's Tablet view. From this we can see a clear need for tools that integrate evidence marshaling and sensemaking into the analytic process. The Tablet view in Jigsaw and other integrated sensemaking environments such as the Sandbox in the nSpace suite [START_REF] Wright | The Sandbox for analysis: concepts and methods[END_REF] are one approach. Another approach, suggested by the studies conducted by Robinson [START_REF] Robinson | Collaborative Synthesis of Visual Analytic Results[END_REF] and Andrews et al. [START_REF] Andrews | Space to think: large high-resolution displays for sensemaking[END_REF] as well as our observations of the text editor group would be to integrate sensemaking tools right into the document space. As we observed in this study, the users of the text editor already were arranging documents into structures based on their content. A logical continuation of this would be to integrate sensemaking tools and representations into this space directly, so that the sensemaking is done directly with the documents, allowing the user to maintain the context of the original source material.

We have also considered some frustrations expressed by the users while developing design implications. One issue involved the presence of the taskbar on only one of the eight monitors, an issue recognized in the past (for example GroupBar [START_REF] Patrick | The TaskBar Evolved[END_REF]). It became difficult and inconvenient for the users to locate windows in the taskbar, especially with over fifty windows opened simultaneously. For future visual analytics tools, we recommend implementing a feature that allows easier location of documents. This could be done through a better search feature, such as flashing document windows to make locating them easier.

Conclusion

We have conducted a study which explores an arrangement for co-located collaborative sensemaking and applied it to intelligence analysis, an application that, to the best of our knowledge, has not yet been seen for this specific set-up and application. We extracted five common activities which the participants used in their overall strategy during collaborative sensemaking. While the activities were common with all groups, the execution of the activities varied based on the tool (Jigsaw or text editor). These activities reflected many of the steps in the Pirolli and Card sensemaking loop. The participants also moved through the loop by using the roles of sensemaker and forager so that the two major areas of sensemaking could be performed synchronously. The groups that adopted these roles tended to score higher. Taking all of these findings into account, we have developed design implications for systems that use multiple input devices collaboratively on a large, vertical display.

The application of co-located collaboration to other visual analytics tools should be further investigated in order to develop a more accurate set of guidelines for designing colocated collaborative systems on large displays. We are also interested in studying the impacts of spatially arranged data on co-located collaborative analysis.

Fig. 4 .

 4 Fig. 4. Screenshot of one of the scenarios, group J2, using Jigsaw, illustrating one way in which the users partitioned the display to conduct invidual investigations

Fig. 5 .

 5 Fig. 5. Overall teams scores vs. strongly developed user roles. Groups from left to right: J4, J3, T2 & J2 (overlapped), T3, J1, T1, T4

Fig. 6 .

 6 Fig. 6. Jigsaw (dark blue) and Text Editor (light green) scores versus collaboration levels

Table 1 :

 1 Overall team scores grouped by tool used comparing aggregated performance

	Jigsaw		Text Editor
	J1	11	T1	13
	J2	-1	T2	-1
	J3	-2	T3	10
	J4	-7	T4	14

Table 2 :

 2 Five sensemaking activities and their methods for corresponding tool.

	Activity	Tool	Method
	Extract	Jigsaw	Look over frequently occurring entities and related documents
		Text Editor Read all of the documents, together or separately, and highlight
	Cluster	Jigsaw	(did not occur with this tool as Jigsaw automatically color codes
			and groups the entities by type)
		Text Editor Group document windows by related content
	Record	Jigsaw	Tablet (3 groups), whiteboard & paper (1 group)
		Text Editor Whiteboard, paper
	Connect	Jigsaw	Loop between list, document view, and Tablet (or pa-
			per/whiteboard), together or separately
		Text Editor Search function; reread paper, whiteboard, and documents
	Review	Jigsaw	Reread, search for unviewed documents (2 groups)
		Text Editor Reread, possibly close windows after reviewing

Acknowledgments. This research was supported by National Science Foundation grants NSF-IIS-0851774 and NSF-CCF-0937133.


David Warnock
email: warnockd@dcs.gla.ac.uk

Marilyn Mcgee-Lennon

Stephen Brewster
email: stephen@dcs.gla.ac.uk

The Role of Modality in Notification Performance

Keywords: Multimodal interfaces, accessibility and usability, technology in healthcare

The primary users of home care technology often have significant sensory impairments. Multimodal interaction can make home care technology more accessible and appropriate, yet most research in the field of multimodal notifications is not aimed at the home but at office or high-pressure environments. This paper presents an experiment that compared the disruptiveness and effectiveness of visual, auditory, tactile and olfactory notifications. The results showed that disruption in the primary task was the same regardless of the notification modality. It was also found that differences in notification effectiveness were due to the inherent traits of a modality, e.g.olfactory notifications were slowest to deliver. The results of this experiment allow researchers and developers to capitalize on the different properties of multimodal techniques, with significant implications for home care technology and technology targeted at users with sensory impairments.

Introduction

With an ageing population throughout Europe and similar trends emerging in other countries [25] it is apparent that preparations need to be made to support a larger aged population. While the post-war baby-boom is often cited as the primary reason for the predicted surge in people over retirement age, modern medicine has also increased life expectancy, suggesting that this trend will continue. As the population ages many will develop health problems that will require care, which is usually provided by family or paid carers. Increased demand for care services will make care at home uneconomical, which could lead to people being moved from their own homes into dedicated care homes where the quality of life is often perceived to be lower [START_REF] Kane | Long-Term Care and a Good Quality of Life: Bringing Them Closer Together[END_REF].

Home care technology can prevent this by helping care recipients to look after themselves, reducing their dependence on carers. This will enable them to continue living in their own homes with dignity and independence. A home reminder system is one type of home care system which can deliver alarms, warnings and information to the user. There are many applications for such reminders in the home: personal care reminders might remind the user to take medication or eat dinner; lifestyle reminders could remind users about events and appointments; and home management reminders might remind users to lock doors and windows at night.

Users of home care technology are very likely to have one or more sensory impairments such as sight or hearing loss [12]. Multimodal interaction uses one or more of the senses in order to convey information; as such it is ideal for creating more accessible systems for people with sensory impairments. Multimodal systems have the potential to allow interaction in a more appropriate way by altering the interaction modality based on factors such as user activity, social context, sensory impairment, preference, message complexity, message urgency and message sensitivity. For example, a potentially embarrassing toileting reminder might normally be delivered as a speech message, but could instead be delivered by private tactile notifications in a social setting.

In order for the designers of home care systems to use a range of modalities in this manner they need to know about the relative properties, capabilities and shortcomings of a variety of multimodal technologies. This paper empirically investigates the properties and effects of notifications delivered through different modalities in order to inform the design of multimodal technology for the home.

Related Work

Notifications can provide valuable information, aid multitasking, deliver timesensitive data and alert the user to important events. They can also be disruptive and irritating. McBryan & Gray [START_REF] Mcbryan | A Model-Based Approach to Supporting Configuration in Ubiquitous Systems[END_REF] describe a hypothetical scenario where notifications are delivered to a user, Fred, via his mobile phone. The mobile phone notifications irritate Fred when he is at home, so the notification system re-routes messages to be delivered via his television. Context sensitivity alongside a choice of modalities have stopped Fred from disabling his phone to stop the notifications. If the phone is legitimately unavailable, or Fred is not responding to the phone notifications, important messages can still be delivered using an alternative modality.

Many researchers have advocated using a range of modalities to provide more appropriate and acceptable interactions [START_REF] Arroyo | Interruptions as multimodal outputs: which are the less disruptive? Proceedings[END_REF][START_REF] Arroyo | Self-adaptive multimodal-interruption interfaces[END_REF][START_REF] Hoggan | Audio or tactile feedback: which modality when?[END_REF][START_REF] Latorella | Effects of Modality on Interrupted Flight Deck Performance: Implications for Data Link[END_REF][START_REF] Mcbryan | A Model-Based Approach to Supporting Configuration in Ubiquitous Systems[END_REF][START_REF] Mcgee-Lennon | Audio Reminders in the Home Environment[END_REF][START_REF] Warnock | A Subjective Evaluation of Multimodal Notifications[END_REF]. However, no guidelines or models exist that will help developers to include them, and most research in this area is directed towards the office or high-pressure environments such as aeroplane cockpits. In order to develop such guidelines, research is needed that will reveal how disruptive certain modalities are along with their general effectiveness as notifications.

A multimodal reminder system should not be disruptive in the home environment. If a notification interrupts the user it could create stress [START_REF] Cohen | Aftereffects of stress on human performance and social behavior: A review of research and theory[END_REF][START_REF] Mark | The cost of interrupted work: more speed and stress[END_REF], generate annoyance and anxiety [START_REF] Bailey | The effects of interruptions on task performance, annoyance, and anxiety in the user interface[END_REF], cause the user to make mistakes in the interrupted activity [START_REF] Kapitsa | Task performance under influence of interruptions. Operator functional state: the assessment and prediction of human performance degradation in complex tasks[END_REF], alter the perceived difficulty of a task [START_REF] Bailey | The effects of interruptions on task performance, annoyance, and anxiety in the user interface[END_REF] and cause a person to speed up their activities [START_REF] Burmistrov | Do Interrupted Users Work Faster or Slower? The Micro-analysis of Computerized Text Editing Task[END_REF][START_REF] Cellier | Interference between switched tasks[END_REF][START_REF] Mark | The cost of interrupted work: more speed and stress[END_REF]. Monk et al. [START_REF] Monk | Very brief interruptions result in resumption cost[END_REF] found that interruptions as short as 0.25 of a second were just as disruptive as interruption 5 seconds long. Berg et al. [START_REF] Berg | Circumstances and consequences of falls in independent community-dwelling older adults[END_REF] investigated the cause of slips, trips and falls in older adults and found that most were considered 'avoidable' and blamed on 'hurrying too much'. Home care developers must be aware of these adverse affects in order to minimise their impact.

The most comprehensive study into modality disruptiveness was performed by Arroyo, Selker & Stouffs [START_REF] Arroyo | Interruptions as multimodal outputs: which are the less disruptive? Proceedings[END_REF]. Their work investigated the disruptive properties of five notification modalities: heat, smell, sound, vibration and light. Disruption was measured subjectively by the participants. The authors reported that their results were not statistically significant, providing only anecdotal evidence for differences between notification modalities.

A later experiment by Arroyo & Selker [START_REF] Arroyo | Self-adaptive multimodal-interruption interfaces[END_REF] that considered heat and light based notifications attempted to demonstrate that disruptiveness differs with modality. Subjects were asked to play a text-based adventure game with disruptiveness measured primarily by the number of mistakes made post-notification. The data reported by the authors did not reveal significant differences between the disruptiveness of the modalities, although the authors asserted that light was superior to heat in their conclusions.

Latorella [START_REF] Latorella | Effects of Modality on Interrupted Flight Deck Performance: Implications for Data Link[END_REF] performed a study into the properties of multimodal notifications in aeroplane cockpits. Latorella reported that there were subtle differences between the visual and audio alerts, but that there were also interaction effects between the modalities of the task and the notification. Latorella concluded that the modality of the notification had an effect on response time, errors in the notification task, and errors in the primary task post-notification.

Arroyo, Selker & Stouffs [START_REF] Arroyo | Interruptions as multimodal outputs: which are the less disruptive? Proceedings[END_REF] suggested that previous exposure to a modality may reduce the negative effects of the notification. Other research has shown that training can reduce the disruptiveness of notifications [START_REF] Cades | Mitigating disruptions: Can resuming an interrupted task be trained? Human Factors and Ergonomics Society Annual Meeting Proceedings[END_REF], as can familiarity with the primary task [START_REF] Cellier | Interference between switched tasks[END_REF] which may support their conclusion. This would suggest that common methods of notification (such as an audio beep) should be less disruptive than uncommon methods (such as an olfactory notification).

There is a conflict in some of the literature regarding the relationship between modality and disruptiveness, and the study by Arroyo, Selker & Stouffs [START_REF] Arroyo | Interruptions as multimodal outputs: which are the less disruptive? Proceedings[END_REF] did not provide knowledge useful to multimodal system developers. A comprehensive study is needed that will investigate how disruptive and effective notifications delivered in different modalities really are; this will help developers to include these modalities when developing multimodal systems.

Design & Method

An experiment with a within-subjects repeated-measures design was carried out in order to evaluate the disruptiveness and effectiveness of notifications in different modalities. Participants attempted to complete a primary task while notifications instructed the participant to carry out a brief secondary task. The primary task was a card-matching game and the secondary task was to receive a notification and acknowledge it by pressing one of three buttons. The noti-Fig. 1. A card-matching game in progress. The player has already matched 6 cards and is attempting to match the 'umbrella' card. The timer at the bottom of the screen shows the player has 32 seconds remaining.

fications were presented in visual, audio, tactile and olfactory modalities. The independent variable was the modality of the notification and the dependent variables were the disruption in the primary task and the speed and accuracy of responses in the secondary task. The experiment was carried out with 27 participants (14 male and 13 female). The participants included 20 people in the 18-30 age group, 4 people aged 31-45 and 3 aged 46-60.

Primary Task

A primary task was desired that encouraged the type of cognitive workload experienced by a person in their own home. This would provide a more realistic picture of how multimodal notifications might interfere with home life. McGee-Lennon, Wolters & McBryan [START_REF] Mcgee-Lennon | Audio Reminders in the Home Environment[END_REF] used a digit span test to evaluate serial recall, and Arroyo & Selker [START_REF] Arroyo | Interruptions as multimodal outputs: which are the less disruptive? Proceedings[END_REF] used a proof-reading test for their primary task. While these tasks have been repeatedly employed in experiments to build a mental workload, the type of workload they encourage is more likely to be encountered in an office environment than at home. A later experiment by Arroyo & Selker [START_REF] Arroyo | Self-adaptive multimodal-interruption interfaces[END_REF] used a computer game, which was more likely to produce the desired type of cognitive workload; however their game was complex, time-consuming and difficult to interpret.

The task chosen was a simple card-matching game called 'Concentration' (also known as Memory or Pairs). In concentration, pairs of cards are presented face-down to the player. The player can then turn over two cards per turn in an attempt to find the pairs and remove them from the game (see Fig. 1). Psychologists have used this task in experiments to better understand how the brain processes information; in particular, the game has been used to evaluate how and why adults can outperform children despite the accepted view that children have superior visual-spatial memory skills [START_REF] Baker-Ward | Age differences in visual-spatial memory performance: do children really out-perform adults when playing Concentration[END_REF][START_REF] Gellatly | The Development of Skill at Concentration[END_REF][START_REF] Schumann-Hengsteler | Children's and adults' visuospatial memory: The game concentration[END_REF]. Schumann-Hengsteler [START_REF] Schumann-Hengsteler | Children's and adults' visuospatial memory: The game concentration[END_REF] suggests that the game of Concentration may not be entirely visual-spatial, arguing that adults have the ability to re-encode the information on the cards. A picture of a boat can be remembered by the picture itself, the verbal label 'boat', or simply by understanding what a boat is. In doing this, adults makes more efficient use of their memory which provides a greater performance benefit than the superior visual-spatial memory of a child player.

Simple icons were displayed on the cards to allow for the 're-encoding' phenomenon as described by Schumann-Hengsteler [START_REF] Schumann-Hengsteler | Children's and adults' visuospatial memory: The game concentration[END_REF]. While there are various ISO standard pictogram sets, they are generally used for sign-making and most lack easily identified verbal labels. Instead, the game will used simple A-Z icons taken from an online speech therapy resource website1 as shown in Fig. 2.

A voluntary pilot study hosted on the project website 2 was run to evaluate the primary task and identify a suitable configuration. The experiment needed to be performed quickly enough that multiple trials and conditions could be completed, but there also had to be enough cards to ensure that the task was sufficiently challenging. The pilot consisted of 37 anonymous participants who each played three games of concentration with 16, 24 and 36 cards. The 24 card game provided an average completion time of 63.9 seconds, suggesting that a 24 card game with a 60 second time limit would allow many players to complete the game. This configuration was used in the experiment as shown in Fig. 1.

Concentration is a simple leisure activity that might well be carried out at home, it is a well-known game with very simple rules and it can quickly build a mental workload. The pilot study demonstrated that it was effective for the needs of the experiment and also allowed for the validation of the experimental measures as discussed in Sect. 3.4.

Secondary Task

Gillie & Broadbent [START_REF] Gillie | What makes interruptions disruptive? A study of length, similarity, and complexity[END_REF] demonstrated that the complexity of the secondary task and its similarity to the primary task are the main factors in determining how disruptive a notification is. To isolate the effects of modality, the secondary task should be simple and as different from the primary task as possible.

The secondary task in our study required a participant to press one of three buttons in response to a notification. These buttons are large, physical, coloured buttons placed directly in front of the participant and fixed to the desk. Physically separate buttons encouraged the participants to look away from the screen and remove their hand from the mouse; this separation was intended to re-enforce the concept that participants must 'stop' the primary task to deal with the secondary one. Similar to the experiment carried out by McGee-Lennon, Wolters & McBryan [START_REF] Mcgee-Lennon | Audio Reminders in the Home Environment[END_REF], the buttons were labelled with the terms "Heating", "Lights" and "Telephone" to provide home-related context as shown in Fig. 3. This was used to provide additional semantic links between the notifications and buttons where the delivery method could support it.

Notifications

To evaluate the differences between notifications in different modalities, a wide range of unimodal notifications were designed for the experiments. These included common notification techniques such as text and speech along with less common notification modalities such as olfaction and abstract visual display. Eight unimodal notifications were developed for the experiment as shown in Table 1.

The text and pictogram notifications were delivered directly into the game window to the top of the play area. No additional hardware was required for this configuration. The abstract visual display was created with a short-throw projector positioned to project a coloured light against the wall adjacent to the participant. The projector was deliberately aligned so that the projection lay in the peripheral vision of the participant, as shown in Fig. 4a.

In all the audio conditions the notifications were delivered through a pair of Sennheiser HD 25-1 II closed-back headphones as shown in Fig. 4b. These headphones helped to prevent background noise from causing interference.

Tactile notifications were delivered via an Engineering Acoustics Inc. C2 vibrotactile actuator4 powered by a small amplifier. This was secured to the top of the wrist on the participant's non-dominant hand with a stretchable bandage as shown in Fig. 4c in order to simulate notifications delivered from a watch. The device has a very low latency and was able to create precise tactile messages.

The olfactory notifications were delivered using a Dale Air5 Vortex Active smell device which has the capacity for 4 different smells. Scents are stored on 1-inch disks, which are blown by a fan to deliver the smell. Delivery times are much longer than the other devices (as the smells took longer to reach the nose), so to ensure the smells were delivered in a reasonable time frame the delivery device was placed directly in front of the participants as shown in Fig. 4d.

The notifications used in the study were all powered by off-the-shelf technology, so any commercial product providing interaction in these modalities would be likely to offer it at a similar quality. The notifications used in the experiment are defined in Table 1 which also shows how the unimodal notifications were grouped by sensory apparatus.

Measurements

Other experiments into disruptiveness have examined errors introduced into the primary task and change in activity rate post-notification [START_REF] Arroyo | Interruptions as multimodal outputs: which are the less disruptive? Proceedings[END_REF][START_REF] Arroyo | Self-adaptive multimodal-interruption interfaces[END_REF]. Latorella [START_REF] Latorella | Effects of Modality on Interrupted Flight Deck Performance: Implications for Data Link[END_REF] observed that activity rate and error rate disruptions have distinctly different properties.

Other experiments investigating the properties of concentration have used various techniques to measure performance. Schumann-Hengsteler [START_REF] Schumann-Hengsteler | Children's and adults' visuospatial memory: The game concentration[END_REF] measured the number of missed opportunities, Baker-Ward & Ornstein [START_REF] Baker-Ward | Age differences in visual-spatial memory performance: do children really out-perform adults when playing Concentration[END_REF] measured the number of perfect matches, and Gellatly, Jones & Best [START_REF] Gellatly | The Development of Skill at Concentration[END_REF] observed player strategy.

Another measure was created by combining perfect matches and missed opportunities which we called superfluous views. This is a measure of how many decisions (i.e. cards clicked) were not productive. When a card was viewed, that card was marked as 'seen'. Every subsequent viewing of that card which failed to match it to another card was considered a superfluous view. A high number of superfluous views suggested that the participant found it difficult to remember which cards were where. This metric was expressed as superfluous views per turn to prevent it being influenced by player speed or game completion. The activity rate of the participants was measured as turns per second.

These measures of performance were tested as part of the pilot study described in Sect. 3.1. Superfluous views per turn was found to provide the most accurate, reliable and stable measure of error rate in the primary task, while turns per second was confirmed to be an adequate measure of activity rate.

Notification performance was measured by response accuracy and response time. Response accuracy simply checked that that the button pressed corresponded to the notification delivered. Response time was measured as the time between starting notification delivery and the participant pressing a button. This did not reflect the true response time as it is difficult to separate the delivery time from the processing time, particularly for olfactory notifications. Delivery time was considered to be an inherent trait of the modality, and as such separating it from the overall response time would not be representative of real-world performance.

Hypotheses

In order to explore the relationship between the effectiveness and disruptiveness of notifications and the modality used to deliver them, the following hypotheses were tested by the experiment.

H1: Error rate in the primary task will vary with the notification modality (measured as superfluous views per turn). Based on evidence provided by Arroyo, Selker & Stouffs [START_REF] Arroyo | Interruptions as multimodal outputs: which are the less disruptive? Proceedings[END_REF], it was hypothesised that people will be more disrupted by notifications that they do not encounter regularly. As olfactory notifications are very uncommon, it is likely they will prove to be the most disruptive. H2: Activity rate in the primary task will vary with the notification modality (measured as turns per second). Other work has shown that introducing notifications has a significant impact on the speed at which the primary task is performed [START_REF] Kapitsa | Task performance under influence of interruptions. Operator functional state: the assessment and prediction of human performance degradation in complex tasks[END_REF]. It is hypothesised that some modalities will have a more significant impact on the speed at which the primary task is performed than others. H3: The accuracy of notification responses will vary with notification modality (measured as the percentage of responses correct, incorrect and missed). It is hypothesised that inherent differences between the delivery methods will have a significant impact on how easy the notifications are to interpret. H4: The speed at which participants respond to notifications will vary with the notification modality (measured as the average response time). Latorella [START_REF] Latorella | Effects of Modality on Interrupted Flight Deck Performance: Implications for Data Link[END_REF] showed that audio notifications were responded to quicker than visual ones.

Procedure

At the beginning of each trial participants were given an information sheet and consent form, then asked to take a short demographic survey to collect gender and age information. Participants were also asked to self-assess their sensory ability on a 21-point Likert scale. Each participant then completed a control condition (which had no notifications) and 4 experimental conditions. During the visual and audio conditions each participant would only see one of the delivery methods described in Sect. 3.3. Participants were counter balanced to ensure even coverage of the different delivery methods. With 27 participants in the visual condition, 9 received textual notifications, 9 received pictograms and 9 received abstract visual notifications (and similarly for audio). As the tactile and olfactory conditions both use a single delivery method the same configuration was used for all 27 participants. The control condition always came first, while the experimental conditions were delivered in a random order. As such the experimenter and participant were both blind to the order or type of experimental conditions.

Each experimental condition consisted of a training segment and five games (except the control condition, where training was not required). At the start of each condition a screen described the type of notification and the hardware was configured as required (see Sect. 3.3). Each participant was given the opportunity to make minor comfort adjustments (such as volume) before each condition began.

Participants were trained by introducing each notification in turn and associating them with the correct button. Notifications were then delivered randomly until the participant had correctly acknowledged 6 sequential notifications. The participant was informed and corrected if a mistake was made. This training helped to ensure that each participant had fully understood the links between the notifications and buttons at the start of each experimental condition.

With the training complete, the participant would then play five games of concentration. In each game 3 notifications were delivered at random points with buffers between them to prevent overlap. A large buffer at the end of the game helped to ensure that quick players could not finish before all the notifications had been delivered. At the end of every game the participant was provided with the opportunity to rest before the next game started.

Once all the conditions had been completed participants were paid and offered the opportunity to ask questions. The experiment required around 50 minutes per participant.

Results

H1:

Error rate in the primary task will vary with the notification modality (measured as superfluous views per turn).

Fig. 5 shows a comparison between error rate and the notification modality. A within-subjects one-way ANOVA was performed on superfluous views per turn for each modality, which showed significant differences between the means of the conditions (F (4, 101) = 16.57, p < 0.0001). Post hoc Tukey tests revealed that the significant differences were between the control and the experimental conditions; there were no significant differences in error rate between any of the conditions with notifications. The data did not support the hypothesis, suggesting that there is no connection between task disruption and the modality of the notification. H2: Activity rate in the primary task will vary with the notification modality (measured as turns per second).

Control

Fig. 6 shows the relationship between activity rate and notification modality. A within-subjects one-way ANOVA showed a significant effect of notification modality on the speed of the player (F (4, 101) = 8.76, p < 0.0001). Post hoc Tukey tests identified two groups: one containing the visual and audio conditions and the other containing tactile and olfactory. These two groups were significantly different from each other. Olfactory and tactile notifications introduced a temporal disruption, slowing the players, while visual and audio conditions appeared to prompt the player to speed up resulting in a higher rate of activity. The evidence supports the original hypothesis that that notification modality will influence the rate of activity in the primary task.

H3: The accuracy of notification responses will vary with notification modality

(measured as the percentage of responses correct, incorrect and missed).

The number of correct responses, as shown in Fig. 7, was evaluated with a KruskalWallis one-way ANOVA which found a significant effect of modality on correct responses (χ 2 (3) = 41.2, p < 0.0001). The audio and visual modalities had a high correct acknowledgement rate with means of 90% and 96% respectively. Tactile and olfactory did not perform as well, scoring 76% and 70% respectively. Modality was also found to have an effect on responses incorrect (χ 2 (3) = 24.96, p < 0.0001) and missed (χ 2 (3) = 31.77, p < 0.0001). The visual notifications had the lowest number of incorrect responses at 2.5%, while the tactile condition had the highest at 16.7%. More olfactory notifications were missed than any other modality. For visual and audio, very few of the notifications were missed. The evidence in this case strongly supports the hypothesis that there is an effect of modality on notification response accuracy.

H4:

The speed at which participants respond to notifications will vary with the notification modality (measured as the average response time).

This hypothesis was evaluated with a within-subjects one-way ANOVA over the average response time for each modality, which found a significant effect of modality on response time (F (3, 75) = 322.09, p < 0.0001). Average response time was calculated using the response times for both correct and incorrect acknowledgements and is shown in Fig. 8.

Discussion

The study presented here show that the introduction of notifications was disruptive to the primary task, causing a significant jump in the error rate. However, there were no significant differences between the modalities, implying that the notifications were all equally disruptive. Activity rate was not significantly different from the control condition for any of the experimental conditions, despite the increased time demand on the participant. However, the visual and audio conditions did result in significantly higher activity rate than the tactile and olfactory conditions.

Arroyo, Selkers & Stouffs [START_REF] Arroyo | Interruptions as multimodal outputs: which are the less disruptive? Proceedings[END_REF] had tested disruption subjectively, but found no significant differences in any objective measures. This study agrees with their results, suggesting that there is no relationship between notification modality and error rate. Cellier & Eyrolle [START_REF] Cellier | Interference between switched tasks[END_REF] suggested an interrupted person takes a 'mental snapshot' of the task at hand when interrupted and then uses that information to resume the task later. The results of this experiment suggest that the modality of the notification does not affect the process of pausing and resuming a task and that as a result, the modality of the notification does not influence the error rate in the primary task.

Anecdotal evidence suggests that disruption could be greater for modalities which the user is unfamiliar with [START_REF] Arroyo | Interruptions as multimodal outputs: which are the less disruptive? Proceedings[END_REF][START_REF] Arroyo | Self-adaptive multimodal-interruption interfaces[END_REF]; this is backed up by research showing that disruption effects can be reduced through experience or training [START_REF] Cades | Mitigating disruptions: Can resuming an interrupted task be trained? Human Factors and Ergonomics Society Annual Meeting Proceedings[END_REF][START_REF] Gillie | What makes interruptions disruptive? A study of length, similarity, and complexity[END_REF][START_REF] Miyata | Psychological issues in support of multiple activities[END_REF]. This experiment did not measure familiarity, but the difference between visual and auditory activity rate and tactile and olfactory activity rate could be explained by this. Given the ubiquity of visual and audio devices and the relative scarcity of tactile and olfactory ones, it seems reasonable to assume that participants were more 'familiar' with visual and auditory notifications. This might provide an increased ability to intercept and process these notifications. However, further experiments would be needed to evaluate this.

The effectiveness of the notifications seems to be attributed to their individual properties. The olfactory notifications, which took the longest to deliver, showed a very long response time. Figure 7 shows olfactory and tactile notifications, which lacked semantic links to the buttons, had the lowest number of correct responses. Tactile notifications, which are targeted and attention-grabbing, had a lower 'miss' rate compared to olfactory notifications but were more likely than olfactory to be misinterpreted. This suggests that the individual properties of each modality are directly linked to their effectiveness as notifications.

The visual and auditory notifications performed well, as they were quick to deliver and provided a semantic link to the buttons (with the exception of the Earcons). The tactile notifications showed an unexpected 'response lag', as shown in Fig. 8. In addition, many of the participants verbally expressed that they had forgotten the button/notification associations at the start of the tactile condition, which did not occur with the other notifications. Hoggan & Brewster [START_REF] Hoggan | Designing audio and tactile crossmodal icons for mobile devices[END_REF] carried out a similar experiment but did not report any problems with tactile training or recall, suggesting that this issue could be the result of poor notification design. However, the tactile notifications had more correct responses than the olfactory ones, which could simply be the participants expressing a lack of confidence in their ability to correctly identify the notifications. This could also explain the increased response time.

The olfactory notifications could be said to have performed the poorest as they have the longest response time, the lowest number of correct responses and also caused a decrease in player speed compared to visual or audio notifications.

Smell technology is rarely examined in multimodal research; smells are difficult to control, they linger after delivery and the technology relies on limited-lifespan chemical components. For these reasons it is often unfairly dismissed as completely impractical, yet in reality artificial smells are common in the home from air fresheners to perfumes and scented candles. This suggests that people might also be willing to accept smell-based technology in their home.

If a home care system was equipped with the same modalities used in this experiment, it would be able to adapt to a range of situations. For example, a reminder notification to lock doors may be considered non-urgent in the afternoon and be displayed textually. Once night falls it becomes more important, so audio notifications might be used instead to alert the target user. Another example would be an elderly person living alone who must take certain medicines daily. To remind the person to take their medicine, a message could be sent to their TV screen; however, if the reminder system detects a social situation, any potentially embarrassing notifications could be re-routed through a private tactile device or an ambient visual display. If this person also had a visual impairment then Auditory Icons or Earcons could be substituted for speech in the same situation.

The study presented here has shown that there is no 'best modality'; all of the unimodal notifications examined by the experiment could be effective notifications. We agree with other researchers [START_REF] Arroyo | Interruptions as multimodal outputs: which are the less disruptive? Proceedings[END_REF][START_REF] Mcbryan | A Model-Based Approach to Supporting Configuration in Ubiquitous Systems[END_REF][START_REF] Mcgee-Lennon | Audio Reminders in the Home Environment[END_REF][START_REF] Warnock | A Subjective Evaluation of Multimodal Notifications[END_REF] who have argued that a range of modalities should be used to provide more appropriate and effective interactions. Notification modalities should be chosen to suit the situation based on factors such as user preference, message sensitivity, social setting, message complexity, message urgency and user impairment.

Conclusion & Future Work

The study presented here has shown that there are a wider range of modalities available to home care developers than are commonly used. The results of the study will help developers to include these modalities, allowing them to create home care technology that is more appropriate, acceptable and effective. This knowledge will also help to expand upon models of interruptibility in the home, such as that developed by Vastenburg, Keyson and Ridder [START_REF] Vastenburg | Considerate home notification systems: A user study of acceptability of notifications in a living-room laboratory[END_REF]. Their model considers message urgency and user engagement to evaluate notification acceptability, which is very much a step in the required direction. However, their model does not include modality or social acceptability, and it does not make provisions for sensory impairment. New models are required that will satisfy the needs of home care patients. In addition, more research is needed in order to fully understand the acceptability of different notification modalities when delivering messages of varying urgency or importance.

In conclusion, the results of this experiment show that the modality of the notification will not affect the primary task, but it is the notification itself which is disruptive. In addition, the results highlight how effective and practical different notification modalities can be. This will help to guide future development of multimodal technology by encouraging developers and researchers to take advantage of a wider range of modalities, allowing the adoption of techniques which are more effective, appropriate and practical.

Fig. 2 .

 2 Fig. 2. Icons used in the primary task.

Fig. 3 .

 3 Fig. 3. Button colour & labelling.

Fig. 4 .

 4 Fig. 4. Experimental configurations.

Fig. 5 .

 5 Fig. 5. Graph of Error Rate and notification modality. No notifications were delivered in the control condition. Superfluous views per turn has a theoretical range of 0-2.

Fig. 6 .Fig. 7 .Fig. 8 .

 678 Fig. 6. Graph of turns per second and modality. No notifications were delivered in the control condition.

Table 1 .

 1 Modal Specification

			Message
	Delivery Method Heating	Lights	Telephone Justification
	Visual
	Text	"Heating" "Lights" "Phone" A simple one-word message dis-
					played in a large bold font above
					the game.
	Pictogram	IEC-60878	IEC-60878	ISO-7001	Taken from two international stan-
		Ther-	Light	Telephone	dards; IEC-60878 and ISO-7001.
		mometer
	Abstract Visual Yellow	Green	Blue Light Projector used to shine a coloured
		Light	Light		light against the wall. The colour
					of the light matched the colour of
					the correct button.
	Auditory
	Voice	Spoken	Spoken	Spoken	Created using the same synthetic
		"Heating"	"Lights"	"Phone"	voice that was used by McGee-
					Lennon, Wolters & McBryan [22].
	Earcon	Acoustic	Clarinet Marimba The Earcons had the same rhythm
		Grand			and varied in the sound of the in-
		Piano			strument; taken from an experi-
					ment by McGee-Lennon, Wolters
					& McBryan [22].
	Auditory Icon	Gas	Light	Phone	Auditory Icons at 1 second each
		Ignition	Switch	Dialing	taken from an online sound effect
			Click (x2)	Beeps	archive. 3
	Tactile
	Tacton	multiLP textLP	voiceLP	Tactons varied in rhythm and
					were taken from an experiment by
					Brewster & Brown [6].
	Olfactory
	Aromacon	Dale Air	Dale Air	Dale Air	Smells were selected based on in-
		"Dark	"River-	"Rasp-	formation from an experiment by
		Choco-	side"	berry"	Brewster, McGookin & Miller [7].
		late"

Speech Teach UK, http://www.speechteach.co.uk

The MultiMemoHome Project, http://MultiMemoHome.org

PacDV Sound Effects, http://www.pacdv.com/sounds/

Engineering Acoustics, http://www.eaiinfo.com

Dale Air, http://www.daleair.com

Acknowledgments

This work was funded by the EPSRC (grant number EP/G069387/1). We would like to thank the anonymous reviewers for their feedback and insights, those who participated in the study and Miss M. Vernigor for modelling the experiment.