Giannis Drossis
email: drossis@ics.forth.gr

Dimitris Grammenos
email: gramenos@ics.forth.gr

Ilia Adami
email: iadami@ics.forth.gr

Constantine Stephanidis

3D Visualization and Multimodal Interaction with Temporal Information Using Timelines

Keywords: Interactive timelines, multimodal interaction, 3D information visualization, full-body interaction, virtual environment, usability evaluation

This paper reports on the design, development and evaluation of TimeViewer, a system allowing the storing, visualization and multimodal interaction with temporal and semantic information using timelines in 3D environments. We focus on the appliance of 3D interactive timelines for temporal information visualization, extending their capabilities with the novel concept of a time-tunnel. Our work is intended to investigate beyond the current state of the art interaction with 3D environments using a variety of input modes, such as touch and full-body kinesthetic interaction. Qualitative results were elicited through an evaluation with 16 users in order to assess the users' opinion of the system, the efficiency of the offered interactions and direct future research in the area.

Introduction

Timelines are a widespread concept for temporal information representation (i.e., events that took place during a time period) and are employed in various environments and contexts, ranging from exhibition spaces and museums to educational textbooks. In general, timelines aim to present information regarding specific temporal periods along with an overview of related occurring events in condensed, yet rich, form. The contents of a timeline can vary from historical events (e.g., [START_REF] Allen | Causation, and History[END_REF][START_REF] Zhao | TimeSlice: Interactive Faceted Browsing of Timeline Data[END_REF]) to numerical data representation (e.g., [START_REF] Marcus | TwitInfo: Aggregating and Visualizing Microblogs for Event Exploration[END_REF][START_REF] Matkovic | Event Line View: Interactive Visual Analysis of Irregular Time-Dependent Data[END_REF]), while the only limitation is that the various items should be temporally related.

In this context, this paper reports on the design, development and evaluation of TimeViewer, a system allowing the modeling, storing, visualization and multimodal interaction with timelines.

TimeViewer extends the typical visualization techniques offered by the majority of the available interactive timelines, which orders the various events in a 2D space to denote their temporal relation by providing an innovative alternate view that aims to highlight the time dimension and facilitate a more immersive exploration of the avail-able information. This view employs a "time tunnel" metaphor, i.e., a long corridor along which events are placed in chronological order, using distance to represent time. On the tunnel sides, next to each event, exists an extendable 'showcase' comprising various multimedia objects (e.g., texts, images, videos and 3D models).

TimeViewer supports a variety of multimodal interaction techniques, even in combination, so as to offer natural interaction in a wide range of hardware set-ups. Apart from common desktop-based interaction techniques (i.e., point and click or multitouch gestures), TimeViewer supports non-instrumented full-body kinesthetic interaction targeted to navigation and manipulation of 3D virtual environments. To assess TimeViewer in terms of usability and user experience and also compare the various supported interaction methods, an evaluation study was conducted with 16 participants.

2

Background and Related Work

Timeline-related concepts

Events are a common concept of timelines and represent any type of incident that took place at some point in time. Information concerning events is represented in chronological order and usually includes at least a title/short description, as well as when it happened. Event occurrences may be (or considered) instant, e.g., the birth of a person, or have a duration, e.g., the construction of a monument.

Periods are time frames during which incidents share some common characteristics or hold a distinct meaning.

Finally, Categories define groups of semantically relevant events whose common denominator can be their type, their context or a specific attribute (e.g., the director in the case of a movie). Categories are mainly used to facilitate interaction by minimizing the amount of the displayed information through filtering out certain objects.

Interactive timelines

Interactive timelines may be broadly classified into three distinct categories according to the type of information they present, as well as the aspect of information they focus on: (a) Historical Events Representation (e.g., [1, 25 and 20]), (b) Temporal Data Representation (e.g. [START_REF] Marcus | TwitInfo: Aggregating and Visualizing Microblogs for Event Exploration[END_REF][START_REF] Matkovic | Event Line View: Interactive Visual Analysis of Irregular Time-Dependent Data[END_REF]) and (c) Semantic Timelines (e.g., [START_REF] Jensen | Visualizing Complex Semantic Timelines[END_REF][START_REF] Stab | Sematime -timeline visualization of time-dependent relations and semantics[END_REF]). The first category includes timelines that represent historical events, aiming to provide an overview around a specific topic. Timelines representing temporal data consist of numerous values of one or more variables that change over time; therefore, the second category can be considered as enriched graph visualization techniques, which primarily focus on the analysis of data distribution over time. Finally, the third category, semantic timelines, represents events with metadata that can be interrelated with others, primarily focusing on displaying the relationships between events rather than providing detailed information about the events per se.

Faceted navigation is a common practice used for temporal data visualization in timelines [2, 16, 23 and 25], implemented through the adoption of taxonomies to classify data in multiple ways and allows the application of filters to information. Faceted display is implemented through hierarchical trees [START_REF] Kr | Visualizing an Overview of Event Sequences[END_REF][START_REF] Zhao | TimeSlice: Interactive Faceted Browsing of Timeline Data[END_REF], separate quantitative controls [START_REF] Ronkainen | Tap input as an embedded interaction method for mobile devices[END_REF] and separate toggle / filtering controls [START_REF] Kumar | Metadata visualization for digital libraries: interactive timeline editing and review[END_REF].

Existing 3D timelines include Beedocs [START_REF]Beedocs 3D Timeline[END_REF] and Kullberg's Dynamic Timelines [START_REF] Kullberg | Dynamic Timelines Visualizing Historical Information in Three Dimensions[END_REF], both of which place events on a two dimensional plane. The major drawback of both is that they do not support a formal way to acquire input from a knowledge source, while Kullberg's approach [START_REF] Kullberg | Dynamic Timelines Visualizing Historical Information in Three Dimensions[END_REF] additionally fails to provide rich interactivity and an overview. Furthermore, another issue of these approaches is the lack of content categorization and time segmentation. In all aforementioned referenced cases user input is provided through a mouse, a keyboard or a touch surface.

Multimodal / Natural interaction

Multimodal user interfaces support interaction techniques which may be used sequentially or concurrently, and independently or combined synergistically, aiming to increase the systems' usability by the application of different modalities that each time suit the system's needs.

Gestures can be defined as a form of non-verbal communication in which visible body actions communicate particular messages; gestural interaction is not only widely studied in the literature, but is actually one of the most popular approaches that users prefer due to its intuitiveness [START_REF] Nickel | Pointing Gesture Recognition based on 3D-Tracking of Face, Hands and Head Orientation[END_REF][START_REF] Sangsuriyachot | Novel Interaction Techniques by Combining Hand and Foot Gestures on Tabletop Environments[END_REF][START_REF] Yoo | 3D User Interface Combining Gaze and Hand Gestures for Large-Scale Display[END_REF]. Furthermore, hand gestures can be used to augment systems and allow supplementary interactions [START_REF] Hilliges | Interactions in the air: adding further depth to interactive tabletop[END_REF] when combined with other means of interaction, such as single-or multi-touch. Apart from single-hand gestures, both users' hands may be used in combination to enrich the set of available gestures [START_REF] Fikket | Gestures for large display control[END_REF]. Finally, the use of feet gesturing [START_REF] Sangsuriyachot | Novel Interaction Techniques by Combining Hand and Foot Gestures on Tabletop Environments[END_REF][START_REF] Valkov | Traveling in 3D Virtual Environments with foot gestures and a multitouch enabled World in Miniature[END_REF] and foot tapping [START_REF] Crossan | Foot tapping for mobile interaction[END_REF][START_REF] Ronkainen | Tap input as an embedded interaction method for mobile devices[END_REF] are occasionally conceded as supplementary helpful interaction tools.

In addition to hand gestures, body movement can augment and facilitate natural interaction. Body movement indicates the pose of a user's body as mentioned in [START_REF] Jaimes | Multimodal human-computer interaction: A survey[END_REF], which can be tracked and used for selective interaction with the environment. Papadopoulos et al. [START_REF] Ch | NuNav3D: A Touch-less, Body-driven Interface for 3D Navigation[END_REF] use defined body poses recognition in order to allow navigation in 3D environments.

Motivation and Design Requirements

The overall goal of the presented work was to design, develop and assess an integrated approach for modeling, storing, retrieving, and visualizing temporal information using multimodal, natural, interaction techniques. In brief, according to the envisioned design requirements the system should: 1. allow expressing, storing and retrieving event-related data augmented with semantic information, and thus be able to retrieve knowledge from a formal data model;

2. support the arbitrary grouping (categorization) of events in a non-restrictive way; thus, apart from temporal categorization, the system should offer a contentindependent semantic categorization mechanism; 3. encompass natural user interaction techniques, such as body movement, hand and leg gestures, which can be used independently or in combination. Any single interaction technique should allow users to manipulate all parts of the system in a consistent way in all supported alternative views; 4. provide alternative, appropriate and complementary, ways (views) to represent information among which the user should be able to dynamically switch 'on-the-fly'; 5. provide suitable display modes for the presentation of event information, both for fundamental data such as titles and for extended details such as descriptive text, images, videos and 3D models.

Brainstorming sessions with HCI experts took place in order to define the appropriate design, regardless of the content displayed and the timespan examined. Interviews were conducted with real users with no technological background, after short experimentation with prototypes, in order to assess the fundamental functionalities they would expect to be offered from such a system. The most significant finding of these interviews involved the segmentation of the process of exploring a timeline in two sections: the users expected the system to offer firstly an overview of the examined time region and secondly a technique to examine details of interest on demand.

From the early design stages it was decided to instantiate the envisioned system through a 3D user interface, thus adding some additional requirements, as interaction complexity in 3D space radically increases due to the additional available degrees of freedom. In this respect: 1. interaction should be precise and rich, while remaining as simple as possible; 2. user disorientation should be prevented; 3. interaction for temporal exploration should offer means to travel both easily through prolonged and precisely through short time frames; 4. the system should be robust and tolerant to user behavior (e.g., gestures, movement) that may be beyond the foreseen 'repertoire' of actions, avoiding unexpected 'reactions' or states; 5. the system should be able to render both in 2D and stereoscopically, allowing to explore and assess in practice the potential benefits and shortcomings of each rendering approach.

System Overview

This section provides an overview of the characteristics and design rationale of the TimeViewer system, which was created based on the aforementioned rationale and requirements. In order to cover the need for semantic information to represent knowledge, timelines are described and stored as an ontology using the Web Ontology Language (OWL) [START_REF] Owl | Web Ontology Language Overview[END_REF], a widely used formal data model used for knowledge information representation in many state of the art systems. Data stored in the ontology can be retrieved through the use of SPARQL queries (SPARQL Protocol and Re-source Description Framework Query Language) executed either at initialization or at runtime. Information retrieval is performed by a dedicated module which can easily be substituted by another implementation which uses an alternate way to represent knowledge.

Visualization Approach

A principal characteristic of TimeViewer is the provision of an immersive display of temporal information supplementary to a view that favors the provision of an overview. In this direction, TimeViewer supports two distinct co-existing modes (i.e., views) for visualizing information respectively: the 'Tunnel' (Figures 3 and4) and 'Classic2D' (Figures 1 and2). Users can freely and seamlessly switch between the two modes at any time. In this respect, a key design goal was to preserve the consistency between alternative representations of the same information regarding both the 'look' and 'feel' of corresponding interactive items. For example, multimedia elements are displayed and manipulated in the same way in both modes, while, although the actual position and scale of events may occasionally change, their appearance and the way that users interact with them remains identical.

In contrast to the display differentiations required for each view of TimeViewer, the fundamental information such as its title holds the same display in both modes for the sake of consistency. The timeline's title is placed at the upper side of the display in order to inform users about the subject that the TimeViewer presents. Next to the title, resides an interactive label showing the category of events presented. Upon selection, a drop-down list appears containing all the available event categories. As event categories in TimeViewer are mutually exclusive, only one can be active at a time. The placement of the related interactive object next to the timeline's title is intended to aid towards understanding the functional role of categories, i.e. filtering and hiding information. Furthermore, using a drop-down menu allows supporting a large number of choices using minimum display estate. Similarly to simplified faceted navigation [START_REF] Andre | Continuum: Designing Timelines for Hierarchies, Relationships and Scale[END_REF][START_REF] Plaisant | LifeLines Visualizing Personal Histories[END_REF][START_REF] Kr | Visualizing an Overview of Event Sequences[END_REF][START_REF] Zhao | TimeSlice: Interactive Faceted Browsing of Timeline Data[END_REF], events not belonging to the selected category are just "dimmed" (Figure 1) and not completely removed from the visualization, as on the one hand, they constitute integral parts of contextual information regarding the rest ones, while, on the other hand, hiding and showing events would result in significant changes to the visualization of the timeline and eventually to confuse the users.

To avoid potential visualization problems (e.g., overcrowding, empty spaces) that arise when some periods contain too many or too few events, in both modes, the visual space taken up by a period can be appropriately scaled (up or down) out of proportion. Furthermore, color coding is also supported to highlight the time period each event belongs to.

Events may be 'spontaneous' or have a duration and can be accompanied by multimedia information, including text, videos, images and 3D models. Each multimedia type affords different interactions: images may be zoomed, videos may be played, paused and stopped, texts may be read aloud, and 3D models may be magnified and rotated. All multimedia types may be accompanied by a descriptive label.

Classic2D Timeline Visualization mode. The goal of this mode is to provide an overview of the available information in a manner familiar to the user, easy to understand and convenient for navigation (Figure 1). Thus, in accordance to the convention followed by similar timelines, the horizontal axis is used to represent time. Periods are also represented along this axis. Time and period representation is placed in the upper side of the virtual world and acts as an upper boundary, below which all events are displayed. The vertical axis is used for the visualization of events, taking care so that closely occurring events do not overlap. More specifically, in the case where any two events overlap, the one that occurs later is placed below the other, thus providing an additional visual clue for perceiving the temporal order of closely occurring events.

Fig. 1. Enabling the category 'Computer Systems' in Classic2D Mode

Time legends appear over time periods, which can adapt to the current level of detail (i.e., zoom level), in order to avoid overcrowding the scene with redundant information; when not viewing the maximum detail level, respectively, the legends are scaled up in order to remain clearly visible. Finally, a button resides at the bottom of the display, which allows the user to switch to the Tunnel mode at any time.

When the timeline is presented at a small scale, events are displayed as rectangular boxes containing a title and a time/date. Events not belonging to the currently selected category are dimmed, indicating that they are not available for interaction. When an events' size increases, its representative item is added to the container box. Upon selection, the event is brought to the center of the display, scaled up and its container is transformed in order to match the need for hosting additional information: it extends so as to host two additional elements, which are placed at the sides of the box inside an expanded case (Figure 3). The item in the center may be examined in further detailaccording to its type -, whereas the next and previous ones are placed on the side. Apart from examining the centered item in detail, the users may proceed to the next or previous items so as to explore all the available information regarding the specific event. In case no further multimedia elements exist, the boxes collapse, indicating that the user has no other items to further explore. At last, events that extend over a span of time are treated as the 'spontaneous' ones (placed according to their start time) differentiated only by the display of their start and end time as a time of occurrence.

Fig. 2. Magnifying an event's image

Tunnel Visualization mode. This mode works complementarily to Classic2D and is targeted to supporting the task of sequential, exhaustive, exploration of the presented information, allowing the user to physically experience the time dimension as well as the unambiguous display of each event. The time dimension is visualized as a "tunnel" i.e., a long corridor, along which all information is integrated. Events are placed on the tunnel floor, while periods are placed on the side corners of the tunnel roof. The tunnel walls serve a dual role: they considerably constraint navigation in 3D space, also offering rich orientation cues, and can host contextual information, such as event-related items.

All basic timeline concepts are supported, but are transformed in an equivalent way suitable to fit the needs of detailed event information demonstration. In essence, the virtual tunnel is a sum of individual tunnel parts (slices). Each distinct part may either represent an event or be just a connecting segment used to connect the various events. The distance between events remains constant and linear, as time inside each period is also linear, depicting the time that intervenes between their occurrences. However, in the case where two or more events take place in a very short time, the events' slices would overlap as their extent is fixed. Therefore, TimeViewer adopts the idea of "freezing" time in a region around the event, equal to its slice, in order to allow the straightforward sequential placement of events occurring even in small time intervals. Thus, the proposed approach is capable of supporting any time interval between two events.

Periods are represented as horizontal signs along both sides of the tunnel roof which are color-coded consistently to the Classic2D mode. Additionally, to further emphasize the organization of events in periods, the lightning of tunnel sections belonging to a specific period follows the respective color code.

An event's title is displayed on a board residing on the tunnel floor with an indication of the event's occurrence time on the side. In addition to the event's title, a 'cavity' with a showcase on its front is placed on the one side of the walls of the tunnel. The cavity includes a glass showcase housing the event's representative item, highlighting the visual representation of the event and providing an image to which the users may be familiar to. The showcase is placed inside the wall and protrudes, when the user virtually passes near an event. As indicated by a handle placed at the outer side of the glass, the showcase may be dragged outwards so as to display additional information regarding the event. Upon dragging the multimedia elements that host the event's additional information extend in a row, covering the space that the expanded showcase occupies. Events that span over time are presented using two separate boards, one representing the beginning being fully functional, and the ending one being supplementary and only displaying a link to the event's start. The display of the end of an event with duration is altered, as its title and date is grayed out, unambiguously designating its semantic difference. Events that do not belong to the currently selected category collapse on the axis that the virtual tunnel extends. However, their presence is still evident, due to the fact that the tunnel's union with each event's collapsed slice is specially designed so as to be different in comparison to the rest of the tunnel's walls, allowing the user to identify the presence of an event at that spot (Figure 3). The user's perception of the time currently inspected is enhanced by the presence of a minimap in the form of a railway wagon, which is placed at the tunnel's roof, covering the whole timeline extent. The minimap rendered is the Classic2D representation along with a frame that provides live feedback regarding the equivalent position in the other view of TimeViewer, while also acting as a gateway to Classic2D view.

Supported Interaction Methods

TimeViewer is designed to support diverse interaction devices and techniques including desktop-based (e.g., touch screen) and full-body kinesthetic interaction.

Touch Screens. The user's touch point on the screen is projected to the virtual world, allowing the selection and dragging of items. Item selection is accomplished by either clicking or dwelling upon (i.e., touching for some time) an item. Item lists (e.g., multimedia elements related to an event) can be scrolled. Navigation in the virtual world is achieved by dragging. In the case of Classic2D representation, users can slide their finger anywhere on the 2D plane in order to move left, right, up and down. Navigation inside the virtual tunnel is accomplished in a similar way, as the view of the tunnel is changed by dragging the tunnel's floor. The available directions towards which the users may drag the tunnel include forward, backward, left and right along the tunnel's extent. Kinesthetic Interaction. TimeViewer supports kinesthetic interaction through body movement and gestures which are tracked using Microsoft's Kinect. The general idea regarding TimeViewer's kinesthetic interaction involves the distinction of navigation in the virtual world and interaction with the timeline's elements: users use their feet to move and their hands to reach out virtual elements, corresponding to a person's actions in the real world. As TimeViewer is currently a single-user system, when multiple persons are within Kinect's field of view, only the nearest person's actions are taken into account, as follows.

User Position. The user's position in comparison to the display is used as a tool through which the user is able to manipulate TimeViewer in respect to the virtual environment shown. Firstly, the user may stand off-center, towards either side of the display, in order to scroll the projected view towards the respective direction in Clas-sic2D mode. Furthermore, the system interprets the distance of the user from the display as a zoom mechanism, enlivening the experience of his or her movement in space. Finally, the Tunnel mode of TimeViewer aims to provide an immersive experience of the users' actions; thus, the virtual tunnel augments the real space and the system maps the actual position to the place from which the user looks in the tunnel, creating the illusion of "being there".

Virtual Hand. TimeViewer adopts the concept of using a virtual cursor to allow the selection of items due to the remote control of the system. The cursor, in the form of a three dimensional virtual hand, appears when the user extends his or her hand towards the display, following and mapping the user's real hand. Items' selection is accomplished by keeping the hand cursor over them for a short period of time.

Hand Gestures. The usage of the virtual hand, however, is not the optimal solution in some cases, where gestures may suit better: such examples may be the act of pushing an item away as well as scrolling through elements in a row. Therefore, Time-Viewer offers the interpretation of users' gestures, using either both hands or a single one. Both user hands are tracked and a number of single and dual hand gestures are supported.

The simplest single-hand gesture supported is the movement of one hand left or right in order to scroll multimedia elements lists in both modes and fort navigation purposes in Classic2D mode. Another common task in three dimensional virtual environments is the provision of camera rotation in a straightforward way: the TimeViewer equivalent single hand gesture involves raising the hand left or right while turning the torso in the corresponding direction, which results to the rotation of the projected view around its vertical axis.

Dual-hand gestures include pushing and pulling by moving both hands forward and backward respectively. These gestures can be used with items residing in front of the user, such as the multimedia elements. Additionally, in the Tunnel mode, users may move to the next or previous available event by pulling or pushing respectively. In Classic2D mode, users can push the event back to its original position when finished examining detailed information.

Leg Gestures. Legs are a natural means for supporting navigation-related interactions. Time-Viewer adopts the notion of stepping: while a user steps right, left, forward and backward, navigation is achieved in the corresponding direction. In Clas-sic2D mode, stepping forward or backward results in increasing or decreasing the scale of the time-line respectively, while stepping left and right results in exploring the timeline's display at the respective direction. In Tunnel mode, stepping forward or backward allows navigation along the tunnel, while stepping at the side results in displaying the tunnel as if the user is looking from the equivalent side.

Evaluation

The Evaluation Process

The evaluation session was divided into three separate segments, one for each of the primary interaction modes (touch and kinesthetic interaction) and a complementary one for the assessment of stereoscopic view: 3D user interfaces lately involve the adoption of stereoscopic displays, which can offer a sense of depth in the visualized information, creating richer representations and making the systems more immersive, but, can also be tiresome since the related technologies has not yet matured. Therefore, an assessment of their impact and usefulness should provide insights on their helpfulness in comparison to traditional displays. The first evaluation segment included the interaction using a touch screen, the second one integrated the manipulation of the system remotely, through the use of user tracking, and the third one integrated the usage of NVidia's 3D Vision technology. The aim of the evaluation process was:

1. to evaluate the system design in terms of perceived usability, effectiveness, likeness, and learnability 2. to assess kinesthetic interaction opposed to desktop interaction, as well as provide an insight regarding the factor of fatigue 3. to assess the effects of stereoscopic vision in comparison to two dimensional display

In order to eliminate bias towards either interaction method, users were split into two even groups. The first group started the evaluation process using the touch screen interaction mode first and then the hand gesturing interaction mode, whereas the second group did the opposite. Both groups evaluated the 3D vision at the end.

The evaluation process started with informing the users of the goals and the process of the evaluation. They were then requested to sign their consent to participating in the evaluation and allowing the video taping of the evaluation. After the introduction, the functionality of the TimeViewer system was explained to each user, along with the available gesture sets that TimeViewer supports.

A series of 7 user tasks were assigned to the participants for each one of the evaluation segments. The participants were instructed to express their thoughts aloud (Think Aloud) throughout the completion of each task. In addition, at the end of the evaluation they were requested to fill out a Likert type of questionnaire.

Set-up and Participants

Two different set-ups were used to evaluate the system. The first comprised a touch screen, while the second one a short-throw projector (supporting stereoscopic vision through the use of appropriate glasses) and a Microsoft Kinect allowing fullbody tracking.

A total of 16 volunteers participated in the evaluation, 7 females and 9 males. The age of the participants varied from 20 to 40 years old. Twelve of the users (75%) had intermediate or high computer expertise whereas the other participants had limited expertise. Even though the majority of the users were familiar with computers and touch screen systems, they did not have great familiarity with hand gesturing as a mode of interaction with a system.

Results

Overall System Design. Overall, the users found that the two different content presentation models, the Classic2D and the Tunnel mode, were complimentary to each other. The majority of the users preferred using the Classic2D mode to obtain a chronological overview of the events and the Tunnel mode to explore specific events. This differentiation in the information display was one of the fundamental goals of this system, as far as design is concerned, and the comments from the evaluation participants indicate that this goal is accomplished. Classic2D mode was regarded by all the users as efficient and intuitive in the way it presented the various periods and their events. The users immediately knew what they were looking at. The application of color coding to the chronological periods and their events made the information easy to follow. The immersive design of the virtual tunnel was met with enthusiasm and excitement by the users who in their majority commented on the "fun" factor of the model. The users understood right away the concept of the TimeViewer and how the information was structured in the three dimensional space. One of the design issues exposed during the evaluations was the content categories filtration mechanism. The users could view all content on the screen, or select one of the available content categories from a list. However, the graphic used to note the existence of the categories and its placement in the design did not make it obvious to the users who failed to acknowledge its existence altogether. Another design issue had to do with the icons used to note the various multimedia elements which proved to create confusion to a lot of the users, as they were not representative enough of the multimedia element.

Touch Screen Interaction. The first segment of the evaluation process indicated that the touch interaction proved to be an efficient, effective, and intuitive way of interacting with both the 2D and the 3D models. The vast majority of the users instantly understood which touch gesture they had to apply to each content element to produce the desired action without much instruction. For example, they understood that in the 3D Tunnel they had to apply the dragging motion in order to advance through the tunnel and the respective periods. Or, in the instance of the glass showcases, where a grabbing hand icon would appear over the handle, the users understood that they could drag it across the screen in order to open and reveal the showcase's content. Overall, the users were successful in completing the given tasks and there were only very few instances where they needed assistance or gave up completely on a task.

One of the main comments that came up repeatedly through the evaluations was the lack of multi-touch capabilities, as a lot of users suggested adding it as a feature. Other suggestions made by the users included, controlling the zoom level by dragging the slider instead of just pressing the (+) and (-) buttons, iterating through multimedia content by dragging the slider instead of just the content elements, and using the minimap frame of the 3D model to navigate to the corresponding position in the 2D view. Kinesthetic Interaction. The enthusiasm of the users was apparent during their interaction with TimeViewer with hand gesturing and user tracking, as only one user out of 16 stated his preference to interact with the system using more traditional modes. The users were captivated by the remote handling of the system achieved without the use of any wearable component, especially as passersby and not every-day users. The moves of the hands were regarded as very natural, as indicated by the comments during the evaluation and the completed questionnaires. All the users found the gestures representative of their functions and the only request made by a small percentage of the users (4 out of 16) was that the gestures should become shorter. The rationale of this request was due to the factor of fatigue that could emerge after prolonged interaction with the system. For instance, the gestures for dragging items to the side as well as the gestures of pulling or pushing items using both hands were proposed to be shorter. Finally, gestures were considered easy to learn, as users were generally able to accomplish them on their own after being shown by the evaluator. The manipulation of the virtual hand (cursor) came easily to the users. A common remark involved the stability of the cursor, which the users requested to be more precise in order to be able to select items which are small in size. Furthermore, some of the users would prefer the duration of the hand being over items to be shorter so as to select them more easily. The hand gestures were embraced without difficulty and the users largely preferred them to the use of the virtual hand. Moreover, the system reacted as expected as far as hand gesturing is concerned, avoiding false interpretation of the users' movements.

An issue that was raised and that was common among the vast majority of the users was the fact that single-hand gestures were deliberately ignored when the cursor was visible. This design decision was taken so as to avoid unexpected system behavior, as the users could simply move their hand in one direction to select an element. After explaining this limitation to the users, they felt comfortable with the system interaction and managed successfully to apply gestures with respect to their goal.

Leg gesturing was almost unanimously accepted, as only one user had trouble with navigation using his legs (Kinect failed to successfully recognize the exact placement of his legs due to the material of his trousers). The idea of stepping in any direction in order to travel in space came naturally to the users and interacting with the system elements using their hands served the initial goal of discriminating the navigation from the interaction. This observation is more evident in the non-expert users, who supported the leg gestures even more than the expert users, as they felt more comfortable with handling the system naturally but in a strictly defined manner. The participants did not have any problem understanding the conceptual model of moving in the space as they found it to be efficient, tireless and fascinating, which is clearly illustrated in the table containing the answers from the users' questionnaires.

In conclusion, although kinesthetic interaction was considered as fractionally less responsive and more tiring to use (questions 5 and 6 respectively), both hand and leg gesturing are preferred to touch interaction (question 7). Despite the error rates not being as good as touch in total, the users were almost unanimously (15 out of 16) in favor of kinesthetic interaction as the fun factor was considered as more important than mediocre precision. A task is considered to be completed successfully if the user accomplishes it at the first or second try, partially successful if the user needed some help or guidance and is marked as a failure in the case where the user needed too much assistance, gave up on completing the task or could not complete the task with no less than four attempts. The tasks that users had issues on completing were due to misperception of the system's design rather than inability to understand how to interact with the system. This fact is evident in task 1, where users were asked to find the existing categories of the timeline: the users who partially succeeded or failed to find them did not observe the category filtration mechanism, until the evaluator provided help. Upon assistance on where to focus, the users were able to manipulate the mechanism in both interaction methods. Some users also faced difficulties in task 2, where they were asked to explore a specific event's information and find the types of presented information. In the case of kinesthetic interaction, a larger user percentage needed minor guidance in order to complete a gesture compared to touch interaction (for instance how to pull an item). Finally, task 4 involved finding a video, where the users' failure was due to inability to identify a video (the video element was thought of an image). The presentation of the tasks that were successfully carried out is beyond the scope of this paper, as we focus on the aspects of the evaluation that can be used for the system's improvement or provide interesting outcomes.

3D Vision.

After the evaluation process regarding interaction by user tracking with Kinect was finished, the participants were given NVidia's 3D Vision Glasses to assess the effects of stereoscopic information presentation. The users were overall in favor of the stereoscopic view (table 2, last row), as 13 participants state that 3D Vision adds up value to the system's representation, while only two disagree and would prefer the normal view in a total of 16 users. Another interesting outcome of great importance is the fact that none of the participants expressed that the usage of 3D glasses was tiring or awkward, as such an issue would be a major drawback in the future. Users did not find that stereoscopic view had any direct effect, positive or negative, on their performance (this fact is depicted in the question 5 at the table below); however, it im- Failure Partial Success Success proved the overall user experience and entertainment value. Some of the users stated that they felt intrigued to further explore the timeline when using the glasses. In conclusion, the stereoscopic display was regarded as a feature that added up value to the system, providing a more immersive feeling, especially in the Tunnel mode of Time-Viewer.

Conclusions

In this paper we have presented the design and characteristics of TimeViewer, a system that allows the storing, 3D visualization and multimodal interaction with temporal information. In addition to the typical 2D representation of timelines supported by the majority of existing systems, this paper introduced the metaphor of a timetunnel in a virtual 3D environment, that can represent the same information in a different manner also supporting appropriate interaction methods. TimeViewer was assessed in terms of usability and user experience through an evaluation study with 16 participants. Overall, the results of the evaluation process were very encouraging. As the developed system supports multiple multimodal interaction techniques, even in combination, so as to offer natural interaction in a wide range of set-ups ranging from desktop-based interaction, to non-instrumented fullbody kinesthetic interaction in 3D environments, particular emphasis on the evaluation was given on the comparison among all these supported interaction techniques.

To this end, the evaluation participants were excited by the diversity of the supported interaction techniques and were "open" to any new form of interaction beyond the more traditional mouse, keyboard and touch options, embracing innovative and state of the art natural interaction techniques. Therefore, further work should aim to expand the possibilities of this research area, taking into consideration the factor of fatigue after extensive use. Additionally, as the evaluation results indicate, the perception of, and interaction with, the 3D environment was easy and straightforward for all users, thus encouraging the "expansion" of other widely used 2D visualization approaches to 3D worlds, which provide more creative options for information representation and interaction. Finally, stereoscopic rendering proved to have a positive influence to the users' perception of the visualization.

Our future work includes further assessing the system with a larger set of users with highly diverse profiles after refining and implementing issues that came up during the last evaluation. Apart from improvements on existing aspects of TimeViewer, a number of system extensions are planned, such as the capability of nesting of timelines inside other timelines and supporting the dynamic creation of timelines through information retrieval from online sources, such as Wikipedia.

Fig. 3 .Fig. 4 .

 34 Fig. 3. The nearest event is hidden in the tunnel due to categorization

Fig. 5 .

 5 Fig. 5. The set-up of the second segment of the evaluation (projection + Kinect).

Table 1 .

 1 Qualitative results for each interaction technique, T (Touchscreen), H (Kinect/Hand Gestures) and L (Kinect/Leg Gestures)

Fig. 6 .Fig. 7 .

 67 Fig. 6. Touch interaction success rates

Table 2 .

 2 Qualitative results of stereoscopic vision.

	3D Vision Questions	Strongly	Disagree Neutral Agree Strongly
		Disagree				Agree
	I liked the interaction in 3D Vision.	0%	0%	13%	44%	44%
	It was awkward to use.	75%	25%	0%	0%	0%
	It was tiring to use.	56%	19%	25%	0%	0%
	I prefer to interact with the system to its	6%	56%	31%	0%	6%
	normal dimension (2D).
	3D Vision improves the system usability	6%	25%	19%	25%	25%
	3D Vision display adds up value to user	0%	13%	6%	50%	31%
	interaction experience.

Acknowledgements

The work reported in this paper has been conducted in the context of the AmI Programme of the Institute of Computer Science of the Foundation for Research and Technology-Hellas (FORTH). Special thanks to Anthony Katzourakis for his artistic work and 3D modeling.


Oussama Metatla
email: oussama@eecs.qmul.ac.uk

Nick Bryan-Kinns
email: nickbk@eecs.qmul.ac.uk

Tony Stockman
email: tonys@eecs.qmul.ac.uk

Fiore Martin

Activity Theory as a Tool for Identifying Design Patterns in Cross-modal Collaborative Interaction

This paper examines the question of how to uncover patterns from the process of designing cross-modal collaborative systems. We describe how we use activity patterns as an approach to guide this process and discuss its potential as a practical method for developing design patterns.

Introduction

A key challenge in the design of interactive systems is how to leverage existing design knowledge. Design patterns play a role in addressing this challenge by capturing and communicating reusable design knowledge on how to map requirements to design solutions. In the field of human-computer interaction (HCI), design patterns have received increasing attention and sets of patterns have been published for a variety of application domains (e.g. [START_REF] Borchers | A pattern approach to interaction design[END_REF][START_REF] Tidwell | Common Ground: A patterns language for Human-Computer Interface design[END_REF]). Similarly, a number of studies within the area of computer-supported cooperative work (CSCW) have addressed the question of how to use design patterns to document successful design knowledge, some of which have generated patterns that can be readily employed to develop practical solutions [START_REF] Arvola | Regulating prominence: A design pattern for co-located collaboration[END_REF]. We are interested in the design of interactive systems that support collaboration between individuals who use different sets of modalities to interact with each other. Such differences can arise because of permanent sensory impairments or due to the capabilities of the technology used by each collaborator and the context of their interactions with it; we refer to group interactions in these settings as cross-modal collaboration. In this paper, we focus on exploring the process by which design patterns for cross-modal collaboration can be identified. We describe how we used an approach based on activity patterns [START_REF] Guy | Appropriating patterns for the activity theory toolkit[END_REF] to uncover design patterns from two phases of a typical design process. First, we exemplify the application of activity patterns by reflecting on the evaluation phase of a cross-modal tool that supports collaboration between visually impaired and sighted coworkers. Second, we outline how activity patterns can be used to capture insights from a participatory design workshop that we conducted with visually impaired audio producers and musicians in order to understand their work process and gather requirements about how to design technology that could contribute towards increasing their inclusion in their workplace.

Cross-modal Collaborative Interaction

Despite significant progress in the use of audio and haptics in multimodal interaction design, research into cross-modal collaboration has so far remained sparse. Initial investigations have nonetheless identified a number of issues that impact the efficiency of collaboration in cross-modal settings. For example, an examination of collaboration between sighted and visually impaired individuals on an interactive puzzle game highlighted the importance of providing visually impaired collaborators with a continuous display of the status of the shared game [START_REF] Winberg | Assembling the senses: towards the design of cooperative interfaces for visually impaired users[END_REF]. Providing collaborators with independent views of the shared space, rather than shared cursor control, was also found to improve orientation, engagement and coordination in shared tasks. In another study [START_REF] Mcgookin | An initial investigation into non-visual computer supported collaboration[END_REF], a multimodal system combining haptic devices with speech and nonspeech auditory output was used to examine collaboration between pairs of visually impaired users on graph reading tasks. Results showed that the use of haptic mechanisms for monitoring activities and shared audio output improves communication and promotes collaboration. Although scarce, the literature on cross-modal collaboration has therefore started to generate insights into the knowledge that is needed to come up with effective designs to support interactions involving individuals with differing perceptual abilities across various domains. We propose to use design patterns as a means to capture such knowledge so that it can be effectively leveraged.

Design Patterns

The concept of design patterns originated from the field of urban architecture as a systematic approach to capturing solutions to architectural design problems [START_REF] Alexander | A Pattern Language: Towns, Buildings, Construction (Center for Environmental Structure Series[END_REF]. Each pattern would characterise a given problem in terms of its context, a set of conflicting forces that are at play, a solution to the problem and the consequences of its application. Particular attention is given to the presentation of patterns as a clear and generic set of instructions so that they can be applied in new situations. A critical aspect of design patterns is that they are cross-referenced. Each pattern refers to higher level patterns describing further context and lower level patterns that could be used to refine the solution. This hierarchical organization structures a collection of patterns into a meaningful network and results in what is called a pattern language [START_REF] Dearden | Pattern languages in HCI: A critical review[END_REF]. The notion of pattern language has also been used to emphasise that patterns should support a "lingua franca" amongst all the parties involved in a design process [START_REF] Erickson | Lingua Francas for design: sacred places and pattern languages[END_REF]. Patterns in this case can be used as a common vocabulary to improve communication of design expertise between stakeholders [START_REF] Borchers | A pattern approach to interaction design[END_REF]. Indeed, design patterns have been shown to have a considerable potential as a method to support participatory design activities [START_REF] Dearden | Using pattern languages in participatory design[END_REF].

Uncovering Design Patterns

In both software engineering and HCI, design patterns are often derived from a reference to existing design solutions and hence are the product of observation, trial and error, and experience [START_REF] Coad | Object-oriented patterns[END_REF][START_REF] Dearden | Pattern languages in HCI: A critical review[END_REF]. In CSCW, finding patterns has also been described as a process that relies on extensive observation of how people use tools in context. For instance, [START_REF] Martin | Finding patterns in the fieldwork[END_REF] followed a retrospective approach to examine data from field studies of cooperative interaction in order to uncover patterns that could be generalised across a number of work environments.

The body of work on patterns in CSCW does not always make it clear how patterns come into existence in the first place, however. So the question often remains whether the Alexandrian way of identifying and organising patterns should be used to guide this process, or whether a different approach should be adopted. On a closer look, the hierarchical nature of the scales of patterns described in [START_REF] Alexander | A Pattern Language: Towns, Buildings, Construction (Center for Environmental Structure Series[END_REF] does not match the hierarchy of problems that are typically encountered in CSCW and an appropriation is therefore required to understand the problem-hierarchy that is relevant to CSCW domains. The domain of cross-modal collaborative systems design presents a set of challenges often characterised by the conflicts associated with reconciling interaction that occurs across more than one modality [START_REF] Metatla | Supporting Cross-modal Collaboration in the Workplace[END_REF]. In this case, we are interested in uncovering patterns both from direct observation of use of cross-modal collaborative systems as well as from design activities that seek to improve such systems.

Activity Patterns

We consider activity patterns [START_REF] Guy | Appropriating patterns for the activity theory toolkit[END_REF] as a potential guiding framework for addressing the issues highlighted above. According to this framework, Alexander's patterns could be appropriated to embody the principals of activity theory (AT) and hence could be used to analyse activity in terms of understanding tool-mediated work in its context [START_REF] Bertelsen | Activity theory. HCI models, theories, and frameworks: Toward a multidisciplinary science[END_REF]. AT views human activity in terms of a system of tool-mediated actions carried out by a subject (i.e. and individual or a group of individuals) in order to achieve a desired outcome. Actions are characterised in terms of how they are organised within a community context, and how they are regulated by internal rules and mediated by a division of labour. This unit of analysis is conventionally represented by a triangular model to show how its elements interact with each other [START_REF] Engestrom | Expanding learning at work: Toward an activity theoretical reconceptualization[END_REF] (see Figure 1). According to [START_REF] Guy | Appropriating patterns for the activity theory toolkit[END_REF], there are parallels between the design patterns principles introduced by Alexander and that of human activity as conceptualised by the unit of analysis in AT. These include:

• The definition of a pattern in terms of three related components expressing the relationship between a given context, a problem and its solution, which is consistent with the method of AT. • The characterisation of a problem in a given context as being caused by a system of conflicting forces that arise in that context, which could be captured through AT's conceptual tool of contradictions.

Additionally, the hierarchical levels of activity in AT can also be used in a similar way to the Alexandrian concept of scales to help structure the scope covered by activity patterns. [START_REF] Alexander | A Pattern Language: Towns, Buildings, Construction (Center for Environmental Structure Series[END_REF] used scales to organise their patterns at different levels of architectural design, e.g. city, road, building, room, window, etc. The concept of activity levels in AT are hierarchically structured into three levels, where a given activity is realised through a set of concrete actions, which are in turn accomplished through a series of operations. This gives patterns a sense of scale from high-level activities down to low-level operations. According to [START_REF] Guy | Appropriating patterns for the activity theory toolkit[END_REF], patterns could be written to reflect each element in a given activity system -the design of mediating artefacts; the work of a subject; the rules and procedures; and the roles within the division of labour or community of the work group -as well as organised into a coherent pattern language that preserves the unity of these elements within each of the three levels of activity.

Activity Patterns in the Evaluation Phase

In order to assess the feasibility of this organisational framework in supporting the process of uncovering design patterns, we applied it to data gathered from two phases of a typical process of designing for cross-modal collaboration; evaluation and requirements capture. In the first instance, we used the activity patterns approach to reflect on the design of an existing cross-modal tool, which we evaluated with visually impaired and sighted users. The tool combines a visual diagram editor with auditory and haptic capabilities to allow simultaneous visual and non-visual interaction. That is, two coworkers collaborate on shared diagrams by accessing and editing them through the visual modality (for the sighted user) and the combination of audio and haptic modalities (for the visually impaired user). We deployed this collaborative tool in various workplaces including a local government office and a charity organisation where visually impaired and sighted coworkers access and edit diagrams as part of their daily jobs [START_REF] Metatla | Supporting Cross-modal Collaboration in the Workplace[END_REF].

In the following, we describe two examples of applying the activity pattern approach retrospectively to analyse data that we gathered from the field studies and identify potential patterns for cross-modal design.

Example 1: Consistency of interaction steps. In this example, a visually impaired manager (VI) and their sighted assistant (S) at a local government office edit an organisation chart to reflect recent changes in managerial structures. At one point during the interaction, the pair decides to create a connection between two nodes on the chart diagram to highlight a relationship between an existing and a new position. They do this while discussing how the tool should be used to create this relationship. To create a connection between two nodes using the non-visual audio-haptic editor, the visually impaired user must 1) browse the chart to locate the first node and select it, 2) browse the chart again to locate the second node and select it, 3) select the type of connection they wish to use and 4) issue a command to create the desired connection. To do the same in the graphical editor, the sighted user must 1) select the type of connection they wish to use from the graphical tool bar, 2) select the first node on the chart, 3) drag the connection towards the second node using the computer mouse, and then 4) release the mouse to create the connection. Following the activity patterns approach, the actions of creating a connection between two nodes using the visual and non-visual editors can be represented as the two independent activity systems shown in Figure 3, which highlights a contradiction between the operational rules in the two activity systems; there is a mismatch between the interaction steps that each collaborator has to follow in order to create a connection between two nodes on the chart. Modelling the collaborative action of creating a connection in this manner has therefore uncovered a potential design flaw -which manifests itself as contradictionsthat could hinder collaboration. Addressing this design flaw could lead to a design pattern that can eliminate the issues raised by the contradictions. For instance, the potential design pattern could describe the need to reconcile the two mediating artefacts in this context by ensuring equivalence and consistency of interaction steps between the visual and non-visual modalities.

Activity patterns in the requirement phase. While we used the activity patterns approach to drive retrospective analysis of data gathered from evaluation studies, we used it in a second instance to drive the design process from its initial requirement phase. In a second project, we conducted a participatory design workshop (PD) with visually impaired audio producers, engineers and musicians (Figure 5) in order to understand how they go about their work and what kind of difficulties they come across. Participatory design activities generate a huge amount of data and patterns could help with the process of organising the themes that emerge from this data. Here too, the concept of contradictions can be a useful guide to identify the tensions that exist in activity systems constructed to model the scenarios described by workshop participants, and hence could lead to insights about design solutions that could resolve such contradictions. Scenario 1: Exchanging formats between colleagues. In a first scenario, while discussing his experience of working with sighted colleagues and clients, a visually impaired producer described his frustration with inaccessible plugins on digital audio workstations. The visually impaired producer explained how his work with sighted colleagues often involves exchanging projects back and forth in order to complete sub tasks involving the manipulation of audio captured using inaccessible formats or requiring interaction with inaccessible audio plugins. In some cases, these accessibility issues have led to his exclusion from potential collaborative projects because the standard formats used are not readily accessible or would take too long to work with. Scenario 2: Accessing a visual programming language. In a second scenario, a visually impaired participant who specialises as an accessibility trainer described a similar experience with inaccessible visual tools. In this case, the issue was specific to working on projects that were coded using an audio programming language known as Max/MSP1 , which is a visual programming language that uses diagrammatic representations as its main programming components. The visually impaired participant highlighted how inaccessible such programming languages are even though they are used to code audio, which could be considered a natural working modality for visually impaired individuals.

Both scenarios above could be captured by the activity system shown in Figure 6. Here, there is a clear contradiction between the subjects of the activity, i.e. the visually impaired audio producer and accessibility trainer, and the tools available to them as mediating artefacts in the context of their activities. Capturing these contradictions allows us to think about possible design solutions to eliminate them, which could eventually lead to the development of fully articulated design patterns that embody such solutions. For instance, the diagrammatic nature of the Max/MSP programming language could be made accessible using the potential design pattern described in Example 2 above, in which the spatial properties of a diagram are translated into hierarchical structures that support non-visual inspection and navigation. We chose the above examples and scenarios to demonstrate how the potential design patterns they uncover could be structured around different hierarchical levels of activity. Each design pattern falls within one level of the hierarchy; example 1 was concerned with the operational rules for connecting two nodes on an organisational chart thus modelling activity at the operational level; and both scenarios 1 and 2 relate to a higher level activity involving the organisation of collaboration between sighted and visually impaired stakeholders in sound production activities and the impact of currently available tools on such organisation. Patterns can therefore be written for different levels of activity and structured into a coherent pattern language, which can facilitate both the creation of new patterns and navigation of a patterns' space to help readers find their way through the design knowledge material.

Conclusion and Future Work

The design of cross-modal collaborative systems presents a unique set of challenges because such systems must allow individual users to equally contribute to the shared tasks while accommodating their individual perceptual differences. To date, no research has examined how to capture the knowledge required to design technology that makes cross-modal collaboration easier. Design patterns and pattern languages have a potential to facilitate the capture, presentation and communication of design knowledge. We proposed that designers and application domain experts in crossmodal collaboration could benefit from using design patterns as a uniform representation for expert knowledge. To this end, we explored the question of how potential patterns can be uncovered from an iterative design process and suggested that activity patterns could be used as a structured method to address this question. We also described how the activity patterns approach matches the domain in which individuals with differing perceptual abilities work together to achieve a common desired outcome through the mediation of a cross-modal tool. One of the key benefits of using activity patterns to identify patterns is the conceptual tool of contradiction, which can be a useful guide for designers to identify the tensions that exist in their designs when used in context and modelled as activity systems. We exemplified how this approach was useful for us in the evaluation phase of a cross-modal collaborative tool as well as a technique for managing requirement data from participatory design workshops. We plan to use this approach to both generate and articulate an initial set of patterns to form a pattern language for designing crossmodal collaboration, which we will then validate by applying the patterns in future design iterations and incorporating them in future participatory design activities.

Fig. 1 .

 1 Fig. 1. An activity system as conceptualized by activity theory.

Fig. 2 .

 2 Fig. 2. Sighted and visually impaired colleagues editing a shared diagram.

Fig. 3 .

 3 Fig. 3. Two activity systems for creating a connection between two nodes using the non-visual (1) and the visual (2) editors. Contradictions are highlighted with red circles and wavy arrows.

Fig. 4 .

 4 Fig. 4. Participatory design workshop with visually impaired and audio producers.

Fig. 5 .

 5 Fig. 5. An activity system showing some contradictions captured through the PD workshop. Contradictions are highlighted with red circles and wavy arrows.

See: http://cycling74.com/products/max/