Mirko Fetter

Julian Seifert

Tom Gross

Predicting Selective Availability for Instant Messaging

Keywords: Instant Messaging, Context Inference, Sensors, Privacy

Instant messaging (IM) systems allow users to spontaneously communicate over distance, yet they bear the risk for disruption of the recipient. In order to reduce disruption, novel approaches for detecting and presenting mutual availability are needed. In this paper we show how fine-grained IM availability predictions can be made for nomadic users solely based on sensors installed on a laptop computer. Our approach provides comparable accuracies to previous work, while it eliminates the need for augmenting the offices or the users with further sensors. We performed a user study to collect sensor data. Alongside with labels collected by means of Experience Sampling, the data allow for creating probabilistic models for predicting selective availability. This way, we demonstrate how the required effort involved in proactively managing one's availability selectively towards a variety of recipients can be reduced by automatic adaptation, and give insights in the lessons learned.

Introduction

Instant messaging (IM) systems offer a great possibility for people to spontaneously communicate over distance, yet bear the risk for disruption of the recipient. In order to reduce disruptions novel approaches for detecting and presenting mutual availability are needed. Previous research examined how predictions based on sensors installed in the environment and desktop computer can help a caller to better estimate, if a recipient in an office scenario is currently interruptible [START_REF] Begole | Sensing Unavailability[END_REF][START_REF] Fogarty | Predicting Human Interruptibility with Sensors[END_REF][START_REF] Fogarty | Examining the Robustness of Sensor-Based Statistical Models of Human Interruptibility[END_REF][START_REF] Horvitz | Learning and Reasoning about Interruption[END_REF]. Others have looked at predicting availability for phone calls based on body-worn sensors and sensors installed on mobile phones [START_REF] Ho | Using Context-Aware Computing to Reduce the Perceived Burden of Interruptions from Mobile Devices[END_REF][START_REF] Ter Hofte | Xensible Interruptions from Your Mobile Phone[END_REF]. In this paper we show how fine-grained IM availability predictions can be made for nomadic users solely based on sensors installed on a laptop computer. Our approach provides comparable accuracies to previous work, while it eliminates the need for augmenting the offices or the users with sensors, allowing it to be instantly applied in diverse environments and situations. This way, we demonstrate how the required effort involved in proactively managing one's availability selectively towards a variety of recipients can be reduced by automatic adaptation.

In this paper we first motivate the need for fine-grained predictions from a usercentred perspective since they allow for lightweight selective availability (i.e., easy management of different levels of availability to different groups of social contacts).

We elaborate on the technical and formal setup of the study and outline the underlying concepts for the collection of sensor data and user feedback. Further, we give detailed insight into the exploration, preparation, and analysis of the collected sensor data as well as into the machine learning performance. We identify lessons learned that can inform future research in this field and reflect our findings in comparison to related work. This paper concludes with an outlook to future work.

Accordingly, the contributions of this paper are: A demonstration of how machine learning can help to reduce the users' effort of managing well-differentiated selective availability in various environments, verified by the quality of the classification result; and a set of lessons learned that can inform future work in this field.

Automatic Adaptation of Selective Availability

Laptop computers give users the freedom to work at different places. Besides the office, people use them to work from home, at coffee shops or hotels, to take them to meeting rooms, presentations, or lectures, or to work on a train while commuting to the office or on a plane while flying to a business meeting. Such changing environments often come with different preferences to whom the user is currently available for communication. For communication via IM, presence and awareness information can help users to mutually present their availability for communication. However, with online status and status message most current IM systems only offer rudimentary support for their users to find an appropriate moment for initiating communication. Studies show that this deficit often leads to disruption of a recipient by an incoming message during a specific task or within an inappropriate situation resulting in communication breakdown [START_REF] Voida | When Conventions Collide: The Tensions of Instant Messaging Attributed[END_REF]. Two fundamental challenges can be identified where current IM systems fall short: • First, current IM systems only offer one single global online status. Therefore, users communicate their availability to all their contacts in the same way. A differentiation of their availability towards different categories of social contacts (e.g., family, colleagues, etc.) is only possible with workarounds in current systems. For example, Voida et al. [START_REF] Voida | When Conventions Collide: The Tensions of Instant Messaging Attributed[END_REF] showed with a study that users maintain multiple IM accounts in order to achieve selective availability towards different categories of contacts. Other studies describe similar findings [START_REF] Davis | Using Relationship to Control Disclosure in Awareness Servers[END_REF][START_REF] Patil | Instant Messaging and Privacy[END_REF]. • Second, the users need to adapt the online status manually. This leads to a selfinterruption in their workflow and thus most times is simply forgotten. Hence, the resulting incorrect online status looses it significance for the users' contacts and accordingly is ignored because it is regarded as unreliable. This behaviour was also observed in the study of Voida et al. [START_REF] Voida | When Conventions Collide: The Tensions of Instant Messaging Attributed[END_REF] were users textually manage their availability by enquiring. In their evaluation of a mobile 'Personal Presence' system Milewski and Smith [START_REF] Mileswki | Providing Presence Cues to Telephone Users[END_REF] found that users changed their status on average only 1.4 times a day.

Based on these two premises, we conclude that managing selective availability by adapting multiple online statuses manually is not feasible for users. Therefore, we suggest automatic adaptation of the online status through predicting selective availability, which will help users to (a) maintain an up-to-date online status, (b) tailored towards different audiences, (c) while minimising their configuration effort.

In the following we demonstrate the feasibility of this approach by means of classification results with an average accuracy of 81.35% for the prediction of selective availability on the basis of real user data collected in a study.

Collecting the Data

We performed a user study in order to collect data and to research the predictability of fine-grained selective availability levels. The collected data, on the one hand, needs to describe the user's context. On the other hand, class labels that reflect the users inner states (i.e. availability preferences) are needed for training probabilistic models. Several related studies e.g., [START_REF] Ho | Using Context-Aware Computing to Reduce the Perceived Burden of Interruptions from Mobile Devices[END_REF][START_REF] Horvitz | Experience Sampling for Building Predictive User Models: A Comparative Study[END_REF] show that the Experience Sampling method in combination with sensor data logging is a suitable means for collecting such data.

In the following we illustrate the technical setup of this study and discuss in detail the selection of sensors and the Experience Sampling setup. Further, we report on the execution of the study and the collected data.

Technical Setup

Our study focuses on nomadic users where work and private life often intermix and therefore a need for selectively managing their availability exists. We used the primary tool of the study participants, their laptop computers, as platform for collecting data. On the one hand, recent laptops provide a variety of hardware sensors (e.g., camera, accelerometer), which can be used for detecting the user's context directly. Moreover, the computer usage already reflects a user's context considerably: the current used applications, the number of unread emails or the calendar entries all allow to make estimates about the users current context. We implemented a system that runs as a daemon, drives a number of sensors, and saves lo-fi context data. A sensor is defined as a soft-or hardware component, or its combination that monitors and records the state of an artefact, an entity or the environment in the digital or physical realm. Sensors can be dynamically added as plug-ins, which are loaded at system runtime. The system provides a utility application for the investigator that allows the configuration of sensor parameters such as quantisation level or sampling rate (see Figure 1). The sampling rate for each individual sensor was balanced upfront in respect to expressiveness of the individual sensor and resource consumption and varied between 30 and 150 seconds.

By studying the related work, we came to a collection of sensors for capturing context data that worked in similar approaches [START_REF] Fogarty | Predicting Human Interruptibility with Sensors[END_REF][START_REF] Fogarty | Examining the Robustness of Sensor-Based Statistical Models of Human Interruptibility[END_REF][START_REF] Horvitz | Creating and Fielding Personalized Models of the Cost of Interruption[END_REF]. In contrast to the related work, our approach aims at using only sensors that need no further instrumentation of the environment, in order to be applicable ad-hoc in different locations. Therefore, we discarded those sensors used in related work, which needed a stationary installation (e.g., door sensor, phone sensor) and tried to further exploit the capabilities of the mobile computers to gain more sensor information (e.g., build in accelerometer and photocell). All our sensors were implemented in Java and obtain their information either by making calls to native system libraries, by using command line calls or by running small scripts to access information from the hardware, the operating system, or specific applications. In the following list all implemented sensors and their capabilities are described:

The Experience Sampling Dialog

In addition to this list of sensors an Experience Sampling sensor was installed. Its purpose was to prompt participants for an assessment of their current availability. These self-estimates serve as label for the succeeding training and evaluation of probabilistic models by means of machine learning. This sensor presented a dialog (see Figure 2) in which participants were prompted to give a statement about their current general availability and their selective availability as well as their current location. General availability was defined as the availability towards all of their contacts-as they would set it in their current IM system. Selective availability was measured for three availability categories: Private, Work, and Public/Others. We defined availability category as a group of contacts towards which the user is available in the same way.

In order to allow for comparison between the participants, we pre-specified three availability categories instead of letting the participants choose their own arbitrary number of availability categories. These are based on a pre-study in which we looked at research focussing on privacy and sharing preferences in the context of IM as well as in general awareness support (e.g., [START_REF] Gross | Disclosure Templates for Selective Information Disclosure[END_REF][START_REF] Olson | A Study of Preferences for Sharing and Privacy[END_REF][START_REF] Patil | Uncovering Privacy Attitudes and Practices in Instant Messaging[END_REF][START_REF] Patil | Who Gets to Know What When: Configuring Privacy Permissions in an Awareness Application[END_REF]). Based on these studies we identified seven common categories for grouping contacts, towards which information is disclosed in the same way (e.g., family, team, subordinates, etc.). In order to reduce the effort for the Experience Sampling, we conducted a paper-based questionnaire survey to trim down the number of categories. We asked 30 participants to group the identified categories into optionally three or four clusters according to how they would be available for theses groups of persons. The configurations that occurred most incorporated three clusters, which make up our three availability categories and were labelled Private, Work, and Public/Others. Within these categories the participants could assess their availability in terms of six availability levels: Offline, Do not disturb, Not available, Away, Online, and Text Me!. We chose these six availability levels, imitating the online statuses provided by Skype, as all our participants are frequent Skype users and thus used to these levels. We preferred this over using a five-point Likert-scale or a simple binary assessment of "Available" versus "Unavailable".

During the study, this Experience Sampling dialog was presented every 25 to 35 minutes as the front-most window. When the participants did not start to interact with the dialog for more than 30 seconds, the dialog disappeared and reappeared 5 minutes later. In the moment participants started interacting with the dialog, this countdown was stopped, until the participants completely filled out and submitted the dialog. Afterwards the dialog disappeared and started over to appear in the normal interval. This way, we assured that we don't miss a number of samplings when the participants are often away from screen for a short time.

Study Setup and Execution

While ESM allows collecting rich and in-depth data through repetition the disruption and the resulting effort for the individual subjects during a four-week study is very high. In order to achieve comparable results we used the same number of subjects as in related work [START_REF] Fogarty | Predicting Human Interruptibility with Sensors[END_REF][START_REF] Horvitz | Creating and Fielding Personalized Models of the Cost of Interruption[END_REF] in order to collect a similar amount of data. We therefore recruited four volunteers as participants (P1-P4) for a four-week long study. The participants were all males and aged between 25 and 33 (M=28.0). Three participants have a background in computer science one participant in social science: two participants were research assistants and the other two participants were student assistants. All come with several years of experience with instant messaging (M=9.5; SD=4.5) and use in average 4.5 (SD=2.0) different instant messaging accounts; all participants use Skype. Except one participant, who indicated to use instant messaging only during workdays, the others use instant messaging on a daily basis. All participants indicated that they use their laptop at different locations during a normal workday. The study was framed by a short questionnaire upfront and a semistructured interview afterwards.

Data on the participants were collected over the period of four weeks by means of the sensor daemon installed on their laptop computers. The participants were briefed to run the sensor daemon continually and to turn it off only in exceptional situations (e.g., when holding a presentation or the computer's battery runs low). Therefore, each participant collected a different amount of sensor samples and Experience Sampling self-reports. Figure 3 shows the difference between cast and actually answered self-reports. On average, the Experience Sampling dialog was presented 817.75 times (SD=500.52) and 270,726 (SD=133,612) sensor events were collected by 30 sensors. These numbers (c.f. Figure 3) show that the majority of the dialogs remained unanswered. This is an intentionally introduced bias of the study design. As described in 3.2, as long as a dialog remains unanswered the sampling interval is decreased from 30 to 5 minutes in order to miss less data when the computer is left unattended only for a short period. On the other side this leads to a high number of unanswered dialogs for longer breaks (e.g. a one-hour lunch break would result in 12 instead of 2 unanswered dialogs), which can be neglected. An approach to treat unanswered dialogs as an indicator for unavailability was discarded, as it is unclear whether the subject is unreceptive or really not available [START_REF] Ho | Using Context-Aware Computing to Reduce the Perceived Burden of Interruptions from Mobile Devices[END_REF].

Overall participants gave their estimates concerning their availability in average 355 (SD=167.00) times. The differences between the subjects illustrated in Figure 3 are likely due to the varying periods the sensor daemon was running as well as different computer usage patterns. For instance, P4 had the software installed on a computer usually being used only during common working hours. P3, who answered the smallest amount of samples, had the sensor daemon installed on a personal laptop computer but turned it off frequently because off the performance loss it caused. P1 and P2 ran the daemon in average 336 hours during the four-week study period. The high number of unanswered self-reports of P2 could be caused by the individual usage pattern letting the computer run while being physically away from it. The samples

!"#$ %&"!$ #'($ "#%$ &!)$ &"!$ %*($ '%*$ +$ &++$ (++$ %'++$ %*++$ ,%$ ,'$ ,#$ ,&$ !"#$%#&'()'*+,-'+./'+.,&#$#/'#01#$%#.*#',+213#,' -./0$ 12/34546$
were collected at 4.8 (SD=2.2) locations including the respective users' office, meeting rooms, lecture rooms, other people's offices, the university' cafeteria or at home. A more qualitative interpretation of the Experience Sampling data in respect to the participants' needs for selective availability is discussed in [START_REF] Fetter | Lightweight Selective Availability in Instant Messaging[END_REF]. In the following we will solely focus on the collected data from the perspective of predictability of selective availability through machine learning.

Predicting Selective Availability

In this section we illustrate the pre-processing of the data and to what extend it is possible to predict selective availability levels based on collected sensor data. We describe the process of transforming the sensor data into features, the generation and selection of features, and the selection of the classification algorithms. This proceeds with the discussion of the evaluation of the machine learning performance.

Feature Generation, Selection, and Classification

The data provided by sensors do not allow reasoning about the users' availability right away. It is necessary to transform the collected sensor data logs from each user into meaningful separated features. For instance, the Applications-sensor (1×n) provides a list of all running applications with each sensor reading. Each application name contained in the sensor value refers to one feature, which needs to be extracted. Hence, when creating a probabilistic model all the lists of application names need to be processed to collect the complete list of available features. Besides this simple transformation of sensor data into features, we generated additional features by integrating the feature values over time. This allows to take advantage of a sensor's value history which reveals trends and tendencies. In order to keep the number of generated features low, we decided to generate time integrations only for five, eight, and 14 minutes. In particular, for numeric features we calculated the mean and standard deviation for the respective history time. For Boolean values we calculated to what extend the particular feature was true or false during the respective period. This approach of feature generation increases the dimension of the feature space enormously. The next step therefore was to discard features with a minimal information gain upfront. The resulting number of features after this reduction still was 1437.3 in average.

As most machine-learning algorithms perform better in a low dimensional feature space, it was necessary to further reduce the number of features for the classification. We compared different techniques for selecting a small subset of features. In particular, we focused on Correlation-based Feature Selection (CFS), Information Gain, Gain Ratio, and Chi-square Feature Subset Selection. Since different classifiers perform significantly different when trained with differing feature subsets, it is necessary to explore the selection of the classifier in combination with the feature subset-selection. Therefore, we considered four different classifiers: Naïve Bayes, C4.5 Decision Tree, Random Trees, and Support Vector Machine (SVM). We used the machine-learning workbench WEKA [START_REF] Witten | Data Mining: Practical Machine Learning Tools and Techniques[END_REF] and its experimenter tool for determining which subset of features in combination with which classifier leads to the best classification results. The results show that WEKA's SMO Classifier-a SVM implementation utilising the Sequential Minimal Optimization (SMO) [START_REF] Platt | Fast Training of Support Vector Machines Using Sequential Minimal Optimization[END_REF] algorithm for training-in combination with a feature subset selected by means of Correlationbased Feature Selection (CFS) outperforms all other combinations. Subsequently, the best parameters for the SMO algorithm were determined using WEKA's GridSearch for further optimising the classification performance.

Evaluation of Machine Learning Performance

The evaluation of the predictability of selective availability is based on the data collected in the user study. The evaluation consists of building four probabilistic models for each user using the SMO algorithm based on the (in average 23.4 (SD=10.8)) features selected by the CFS in combination with the (in average 355 (SD=167.00)) samples based on the labels given by the individual participants. For each participant, four models were built-one for General availability and three for the availability categories. Each model is trained to predict the six availability levels, which makes the classification task a multi-class problem. The 10-fold cross-validated classification results for the 4x4 models are summarised in the following Table 1.

Table 1. Results of 10-fold cross-validations for each of the four participants' models, and the average over all models of all participants. Each table presents the accuracy in percent for the ZeroR classifier (to show the base probability), the SMO classifier, and the improvement of the SMO classifier against ZeroR classifier (Diff.) for the different models. In the most left column of each table the name of the respective data set is listed. In the second column, the base probabilities generated by the ZeroR algorithm are listed. In the third column the accuracy based on a stratified 10-fold cross-validation of the models that were built by the support vector machine SMO are listed. The last column contains the difference between base probability, as estimated by the ZeroR classifier, and the percentage of correctly classified instances by the SMO. The average over the four models of each participant reflects the overall experience the users will have, as all four models will work in unison to adapt the users selective availability.

The results show that over all participants the base line is quite high with 67.66%. This means, in average, one online status is selected more often then all other online status together by a user for a specific availability category. Accordingly, a classifier that always guesses this dominant class would make a correct prediction in 67.66% of the cases. Therefore, the results from the evaluation of the probabilistic model need to be regarded relative to their respective base line. The results shows that in average the probabilistic models perform considerably better (13.69%) than the base line. Overall, with an average accuracy of 81.35% our results can compete with the results of related work; as discussed in detail later.

Lessons Learned

In the following section we elaborate on lessons learned during the study and the explorative machine learning procedure, which can inform future research on similar approaches. We elaborate on the performance of the individual sensors for machine learning. We give guidelines on the engineering of robust sensors for this kind of studies. We motivate our argumentation for building personalised models for each individual user. We describe our findings on the usage of the Experience Sampling method. And we show patterns we found in the configuration of selective availability.

Promising Sensors and Features

The results of the feature selection mechanisms allow drawing conclusions on the relevance of single sensors for the prediction result. Such knowledge can inform future research at selecting relevant sensors. Therefore, we analysed the features of all 16 models (four participants, each with a model for General, Private, Work, and Public/Others) that were selected by the correlation-based feature selection algorithm as most relevant. CFS uses a heuristic evaluation method and has proven to be equally good at finding relevant features in machine learning tasks in comparison to wrapperbased approaches. Hence, the selected subset can give a valid estimate on the contribution of individual sensors.

The CFS algorithm reduced the number of features from 1437.3 (SD=303.6) per model to the 23.4 (SD=10.8) most relevant. First, we consolidated the reduced feature sets of all 16 models and analysed what feature generation mechanism delivered the largest number of features to this set. 71.93% of the features were generated by the time-based feature generation. The features that take into account the last 14 minutes made up the majority with 32.89%, followed by 5 minutes (21.39%) and 8 minutes (17.65%). As expected, taking into account the sensor values for a period of time instead of only a point in time can deliver richer features for the classification [START_REF] Fogarty | Toolkit Support for Developing and Deploying Sensor-Based Statistical Models of Human Situations[END_REF]. Further, we analysed the sensors providing one or more relevant features to one or more of the four models for each participant according to the result of the CFS. Table 2 shows that while eleven sensors generated features relevant for at least one model of each of the participants, ten sensors did not. For the category of unused sensors, we found that the users did not influence some measured parameters during the fourweek study period. For example, none of the users connected FireWire devices or mounted a volume. Accordingly, these sensors can be valuable for users that use such devices or services more often. Other unused sensors, like the Mouse Connected Sensor might be substituted by the information of a second sensor, like the Connected USB Devices sensor. Further, our expectation that different combinations of sensors monitoring cables attached to the computer (Ethernet, Headphones, Power, etc.) would give a good estimate about the current location, were not met, as theses sensors practically played no role for the models.

However, most of these unused sensors are not computationally or energetically expensive, as they basically listen to system events, in comparison to more expensive operations like constantly polling for available Wi-Fi networks or performing facedetection on the camera image. As there is a chance that these sensors might be useful for other users, it might be reasonable to not discard them completely. Further, some of these sensors might be valuable for indicating transitions between different contexts. As our Experience Sampling study used a random interval of circa 30 minutes to assess the users' availability, the collected samples cannot reflect the exact moment when users' inner states change e.g., from Available to Do Not Disturb. Some of the sensors that were not used for building the predictive models could be valuable for predicting a good moment for adapting the online status. For example, disconnecting the mouse and Ethernet from a mobile computer could indicate that the user is leaving the current place, and suggest a change in availability.

Carefully Engineered Sensors

The design and implementation of the sensors and the surrounding architecture are crucial for the success of the study. In the following we list several technical requirements, which we found in small informal pre-tests and during the study.

The sensors for data collection need to be robust. During the study the aim was to continuously collect data while the system was in normal use. Accordingly, the implementation should be robust to system restarts, idle-times, crashes or reconfigurations while not limiting the users in their tasks. Crashing sensors should automatically recover without the need for user intervention. We implemented mechanisms that monitor and automatically restart sensors that stopped delivering values. At the same time, sensors should not interfere with user interactions. Sensors that exclusively allocate system resources (in our case camera and microphone) need to be designed in away that these resources are still available for the users' primary tasks (in our case for a video chat). In our setup the sensors obtained data and were released for each sampling, with small pauses between the samplings. If a resource was occupied, the sensors tried again until the resource was obtainable (e.g., after the video chat was finished). In the same way, sensors should not open applications like calendar or email to retrieve sensor values, unless the open them.

In addition to this, collecting sensor data continuously from a mobile computer leads to a reduced battery lifetime and system performance. Continuously accessing hardware like the camera or the embedded acceleration sensors in combination with repeated polling for nearby Wi-Fi or Bluetooth networks has an unavoidable but substantial effect on the battery and CPU. In order to reduce this drawback, we balanced the sampling intervals for the different sensors between maximising the amount of valuable data that is collected and minimising usage constraints.

Further, privacy-protecting mechanisms have a major effect on the users' willingness to collect data. We implemented a hash-based privacy based on the Subtle system [START_REF] Fogarty | Toolkit Support for Developing and Deploying Sensor-Based Statistical Models of Human Situations[END_REF]. Each nominal value that was collected (e.g., the title of the current focussed application, the SSID of nearby Wi-Fi access points) was translated into a unique hashed representation. In this way the data is still meaningful for machine learning algorithms, but reduces its descriptiveness for humans, who prepare the data for machine learning. However, this comes with a decreased interpretability for reflecting on more qualitative aspects of the data in an analysis of the study results. For example, we are unable to make statements about availability preferences while using specific applications like Word or Firefox-which could provide interesting insights for CSCW research-due to the encryption.

Finally, the data collection process needs to be as unobtrusive to the users as possible, yet has to allow the users to stay in control. Whether for reasons of system performance or for privacy reasons, the user has to be able to pause the data collection at any time. In our system, users were able to stop and start the collection with the click on a button. However, in order to reduce the users' need to do so, and to obtain more data from the study, the above guidelines can help.

General Models vs. Individual User Models

One drawback of building an agent-based system is that the system needs to learn from scratch and so is of limited utility to the user at the beginning or in new situations. In order to overcome this issue Collaborative Interface Agents [START_REF] Lashkari | Collaborative Interface Agents[END_REF] were introduced that share their knowledge among user models. In the field of predicting human interruptibility Fogarty et al. [START_REF] Fogarty | Examining the Robustness of Sensor-Based Statistical Models of Human Interruptibility[END_REF] for example built one general model for all subjects and general models for each group of subjects (manager, researcher, intern) in order to overcome the cold-start problem. However, their individual group models performed better than the global general model. From our results we can confirm this findings and even advocate for completely personalised individual models. The first reason is that we found that even among the small number of our subjects, the individual availability expressed through the chosen availability levels strongly varied as can be seen in Figure 4. The Figure shows that while P3 had a clear tendency to being highly available, P4 has a clear tendency to be unavailable, while P1 and P2 have a broader spectrum of availability. This argues for the users' individual understanding of and a personal perspective on availability that a general model can hardly reflect.

Further, such general models need to be built solely on features that all users have in common. This bears three challenges: First, a feature with a high informative value for an individual model, like a wireless network at home, would most likely be discarded from the feature vector of a general model, as it would contain missing values for the other subjects. Second, some of the features may represent contradictory information for different users. Referring to wireless networks as an example, the wireless networks that are accessible from a certain place, may indicate the office for one user and a meeting room for another user, and accordingly different availability preferences. Third, some features' values might correlate very well for different users, but cannot be automatically merged into one feature. For example, the use of the browser or a word processor might lead to similar availability preferences for different users. However, when different users use different applications (e.g., one user uses Firefox, the other user Opera) this sensed information cannot be combined easily. Of course, all three challenges could be overcome by introducing a semantic layer between the sensed data and the generated features. But in many cases (e.g., mapping the Wi-Fi Access Points at home to the semantic representation home) requires further effort by the users for generating individual semantic abstractions.

Accordingly we found for our data: The performance of general models that were built with different classifiers on the intersecting subset of features from all the collected data performed insignificantly better than the base probability of the ZeroR classifier, but most times worse. Satisfactory classification results could not be achieved by using a general model.

Designing the Experience Sampling

The Experience Sampling dialog is the predominant interface to participants and needs special care. During the study the users are interrupted by the dialog several times a day during their daily tasks. In order to keep the study participants motivated to deliver valid self-reports, it is necessary to minimise the effort for the users on the one side and to prevent habitual, repeated responding [START_REF] Scollon | Experience Sampling: Promises and Pitfalls, Strengths and Weaknesses[END_REF] on the other side.

The design of our interface was optimised in iterative steps. It minimises the number of clicks for users, and prevents a bias towards repeated responses. We abandoned a design, where the choices from the previous sampling were pre-selected, as this resulted in a bias towards just confirming the previous choices. A design where we randomised the ordering of the items each time dramatically increased the users' cognitive effort and accordingly increased their frustration with the study-and so was also discarded.

In order to minimise the variance of free-text answers for the "Current location"field, we implemented an auto-complete mechanism. From the subsequent interviews we learned, that this sometimes led to an automatism to accidentally enter and select the location that the users stayed at the most (e.g., their office). This came with a desire of the users for a mechanism to recall and alter or discard the last assessment.

The dialog always appeared as the front-most window in the centre of the screen and could not be discarded by the user. This rather invasive design helped to maximise the number of samples, since the fastest way to get rid of the dialog was to quickly respond to the questions. In the interviews we learned that this was perceived as acceptable in situations where the participants directly interacted with the computer, but was regarded as problematic where the computer was used more indirectly (i.e. to watch a video or to give a presentation). One participant suggested a configuration, which would allow shifting the Experience Sampling to a mobile phone for a certain period of time.

Further, the timing of the Experience Sampling plays a crucial role. We decided to show the dialog at a random interval, with one exception: Every time the users logged in, a sampling was triggered. During the study and afterwards, from the data, we learned that there are a lot of simple cues (that our sensors are able to measure) which can be easily formulated as plain rules to allow sampling at a moment, that typically marks the transition between different availability levels: when the computer is moved (Motion-Sensor), cables are attached or detached (e.g., Power Connected-Sensor, Ethernet Connected-Sensor) or someone starts to speak (Voice Activity-Sensor). These are a few examples for such moments that show how a context-aware Experience Sampling approach [START_REF] Intille | A Context-Aware Experience Sampling Tool[END_REF] could help to improve the quality of the data.

In combination with the sensor logging, the time between the presentation of the dialog and the response have to be taken into account. As the presentation of the dialog (Application Focus-Sensor) and the users interaction with the dialog (Input Idle-Sensor) have an influence on what the sensors measure, they introduce a systematic error. Therefore it is necessary to discard the sensor data from the moment before the dialog is shown until the response was entered. If the time between presentation and response is to long, the sensor data and the Experience Sampling data will loose its correlation. In our design the user had 30 seconds to answer to the Experience Sampling before it disappeared, and was presented again five minutes later. In these 30 seconds no sensor data was recorded.

Finally, users showed great interest in the data collected when we discussed it in the subsequent interviews. To make their data available after the study to each of them individually in a meaningful way (e.g., as charts or diagrams) could make a great incentive to keep them motivated throughout the study.

Patterns for Selective Availability Configurations

The analysis of the collected data revealed that users tended to give similar answers in reoccurring contexts. For example during work-time a user might be available for work related contacts, less for private, and offline for others. This raises the question, how many different patterns or configurations users utilise to state their selective availability in the majority of the times.

We used the expectation-maximisation-algorithm for finding the patterns that fit best to the users' answers. The following table presents the results. For three participants, four patterns were identified (a, b, c, d). For P3 only three patterns (a, b, c) were identified. These patterns respectively make up 84.32% (SD=3.40%) of the used configurations. On average the participants only used 23.25 (SD=14.97) combinations of the 216 possible permutations of availability levels and availability categories.

Since the numbers of the patterns for each participant are smaller than the number of all possible permutations, the complexity of the resulting classification problem is reduced. After transforming the data, again, we created probabilistic models, which were evaluated in terms of their ability to predict the correct pattern when given an example. The results showed, that the individually trained models for each availability category significantly outperform the models for predicting patterns.

Even though the results in this case are discouraging, the insight in such patterns can inform the design of future studies and of instant messaging systems.

Related Work

We briefly discuss related work and outline similarities and differences to our work. To our knowledge, so far no research has focussed on the predictability of selective availability based on sensors on laptop computers. We concentrate on work that focuses on predicting the availability for phone calls based on mobile sensors and two prominent examples that deal with the predictability of general interruptibility based on sensor data in office environments, which inspired our approach.

Ter Hofte [START_REF] Ter Hofte | Xensible Interruptions from Your Mobile Phone[END_REF] used the software SocioXensor for PDAs and mobile phones to collect data from 10 participants over 7 days via ESM. The resulting predictions are not explicitly focused on selective availability. These predictions-based on answers to the ESM (in conversation, location, etc.) and not on real sensor information-only achieve an accuracy of 63,9% for a two-class problem (green light/no green light). Ho and Intille [START_REF] Ho | Using Context-Aware Computing to Reduce the Perceived Burden of Interruptions from Mobile Devices[END_REF] use activity recognition based on wearable accelerometers and a decision tree classifier on a PDA to detect postural and ambulatory transitions. They found that transitions are a good indicator for receptiveness towards interruptions. Their approach burdens the user to wear accelerometer sensors at ankle and thigh.

Fogarty et al. [START_REF] Fogarty | Examining the Robustness of Sensor-Based Statistical Models of Human Interruptibility[END_REF] use a combination of computer (mouse and keyboard activity, used applications, etc.) and room sensors (door, telephone, motion, etc.) to predict the interruptibility of 10 participants. They collected 975 oral interruptibility self-reports with interruptibility on a five-point Likert-scale. They used a naïve Bayes classifier to build three models for the three different types of participants and one general model for all participants. The achieved accuracies lie between 80.1% to 87.7% for the three models and 79.5% percent for the general model. However, they reduced the complexity of the prediction to a two-class problem (1-4 and 5) to obtain these results. In comparison, our average accuracy for predicting a six-class problem based on individual models lies at 81.35%. Their room sensors limit its applicability. Our participants collected data in several different locations in work and private contexts, which also introduces a greater complexity to the prediction task. Also the BusyBody system [START_REF] Horvitz | Creating and Fielding Personalized Models of the Cost of Interruption[END_REF] relied on sensor data collected on computers in an office environment. Four participants collected between 789 and 2365 assessments of their interruptibility ("Busy" and "Not Busy"). The average achieved accuracy with a Bayesian Network is 78.25%. Their prediction is also only for a binary classification problem. There is no information given, in which contexts the data was collected.

Concluding, none of the related work focussed on predicting selective availability for sensors on laptop computer. However, all the works greatly informed our research. Our aim was to combine the advantages of stationary and mobile approaches and to deepen it, in order to show that even more complex problems can be addressed with this approach while minimising the setup effort for deploying additional sensors.

Conclusions and Future Work

We have shown that the prediction of selective availability with machine learning is a feasible approach. Based on our results we conclude that systems that rely on sensor data collected on a laptop computer can be trained by users in order to allow for the automatic adaptation of selective availability in Instant Messaging. Such systems support users in better managing their availability towards different social contacts while at the same time minimising their effort in comparison to manual adaptation.

We showed how this is possible by information that is directly accessible on the users' laptop computer. We showed that we can predict, with an accuracy of 81.35%, in which way users are available for different social contacts in different situations. This was shown for work and private contexts. And it allows users to express availability with fine-grained online statuses.

In order to evaluate how this approach gives a greater flexibility to users while minimising their effort, user tests based on a system that provides such predictions in real-time are needed. To support individual availability categories for each user such systems have to support life-long learning to train an arbitrary number of individual models for each user (i.e. continuously and actively learn from the users' inputs as well as predict to adapt the system in the background). To achieve this, such systems should automatically extract, generate, and select useful features from sensor-data streams. At the same time they should request input from the users in a much more implicit way.

Figure 1 .

 1 Figure 1. Configuration dialog for adjusting the parameters of the sensors. In total we implemented 30 different sensors, each collecting one or more information chunks with each sensing, which where recorded in one of four different xml-data structures as a sensor event: • 1×1 sensors: return a single value per sensor event • 1×n sensors: return several, semantically different values of same or different types per sensor event • n×1 sensors: return a list of values of the same type • n×m sensors: return a matrix of values

Figure 2 .

 2 Figure 2. Experience Sampling interface for collecting user selective availability preferences.

Figure 3 .

 3 Figure 3. Amounts of cast and actually answered self-reports for each participant.

Figure 4 .

 4 Figure 4. Selective Availability distributions for the four subjects in percent.

P1 Average 56.19 76.41 20.22 P3 Average 77.38 90.77 13.39 Model ZeroR SMO Diff. Model ZeroR SMO Diff.

	Model ZeroR SMO Diff.	Model ZeroR SMO Diff.
	P1 General 58.15 75.65 17.51	P3 General 74.40 91.67 17.26
	P1 Private 55.53 75.86 20.32	P3 Private 86.90 92.26	5.36
	P1 Public 59.76 78.27 18.51	P3 Public 69.64 86.90 17.26
	P1 Work 51.31 75.86 24.55	P3 Work 78.57 92.26 13.69
	P2 General 68.62 84.10 15.48	P4 General 62.50 74.54 12.04
	P2 Private 70.29 83.68 13.39	P4 Private 70.37 76.85	6.48
	P2 Public 66.53 82.01 15.48	P4 Public 74.54 76.85	2.31
	P2 Work 71.13 84.94 13.81	P4 Work 64.35 69.91	5.56
	P2

Average 69.14 83.68 14.54 P4 Average 67.94 74.54 6.60 Model ZeroR SMO Diff. All Average 67.66 81.35 13.69

Table 2 .

 2 Sensors that delivered features to the final feature subset: "+" sensor delivered one or more features , "-" sensor delivered no feature for this participant's models.

	Sensor P1 P2 P3 P4 No.	Sensor P1 P2 P3 P4 No.
	Active Access Point + + + + 4	Focus Title + + + + 4
	Active Chats -	-	-	-	0	Headphones Connected -	-	-	-	0
	Active Network Interfaces -	-	-	+ 1	Input Idle -	+ + + 3
	Ambient Light -	+ + + 3	IP Address + + + + 4
	Application Focus + + + + 4	Motion + +	-	+ 3
	Applications + + + + 4	Mounted Volumes -	-	-	-	0
	Battery -	-	-	-	0	Mouse Connected -	-	-	-	0
	Bluetooth Devices + + + + 4	Power Connected +	-	-	-	1
	Calendar + + +	-	3	Screensaver Active -	-	-	-	0
	Connected FireWire Dev. -	-	-	-	0	Second Monitor -	-	-	-	0
	Connected USB Dev. + + + + 4	Skype -	+ + + 3
	CPU -	-	+ + 2	Time + + + + 4
	Email -	-	-	-	0	Voice Activity + + + + 4
	Ethernet Connected -	-	-	-	0	Volume Settings + + + + 4
	Face Detection -	+ + + 3	Wi-Fi + + + + 4

Table 3 .

 3 Patterns (a-d) of availability, by means of clustering the participants ESM data (1=Offline, 2=Do not disturb, 3=Not available, 4=Away, 5=Online, and 6=Text Me!).

Acknowledgments. We thank the members of the Cooperative Media Lab, the participants in this study, and the anonymous reviewers for their insightful comments. Part of the work has been funded by the Federal Ministry of Transport, Building, and Urban Affairs (TransKoop FKZ 03WWTH018).


Nicolai Marquardt
email: nicolai.marquardt@ucalgary.ca

Ricardo Jota

Saul Greenberg
email: saul.greenberg@ucalgary.ca

Joaquim A Jorge

The Continuous Interaction Space: Interaction Techniques Unifying Touch and Gesture On and Above a Digital Surface

Keywords: Touch, gestures, surfaces, interactive tabletops, 3D interaction, tangibles, portable devices, continuous interaction space

The rising popularity of digital table surfaces has spawned considerable interest in new interaction techniques. Most interactions fall into one of two modalities: 1) direct touch and multi-touch (by hand and by tangibles) directly on the surface, and 2) hand gestures above the surface. The limitation is that these two modalities ignore the rich interaction space between them. To move beyond this limitation, we first contribute a unification of these discrete interaction modalities called the continuous interaction space. The idea is that many interaction techniques can be developed that go beyond these two modalities, where they can leverage the space between them. That is, we believe that the underlying system should treat the space on and above the surface as a continuum, where a person can use touch, gestures, and tangibles anywhere in the space and naturally move between them. Our second contribution illustrates this, where we introduce a variety of interaction categories that exploit the space between these modalities. For example, with our Extended Continuous Gestures category, a person can start an interaction with a direct touch and drag, then naturally lift off the surface and continue their drag with a hand gesture over the surface. For each interaction category, we implement an example (or use prior work) that illustrates how that technique can be applied. In summary, our primary contribution is to broaden the design space of interaction techniques for digital surfaces, where we populate the continuous interaction space both with concepts and examples that emerge from considering this space as a continuum.

Introduction

The advent of highly interactive digital surfaces has motivated researchers to develop a rich set of accompanying interaction techniques. While there are now a broad variety of techniques, hand input (sometimes holding tangibles) dominates. Most prior research has focused on two modes of recognizing hand input.

(a) On the surface includes touch interactions directly on the reachable parts of the display, usually using fingers and hands [START_REF] Cao | ShapeTouch: Leveraging Contact Shape on Interactive Surfaces[END_REF] [10] [START_REF] Lee | Gesture-Based Interactions on Multiple Large Displays with a Tabletop Interface[END_REF] [40], or tangible objects [START_REF] Baudisch | Lumino: tangible blocks for tabletop computers based on glass fiber bundles[END_REF] [START_REF] Underkoffler | Urp: a luminous-tangible workbench for urban planning and design[END_REF]. Interactions typically include selecting, grabbing, throwing, rotating, and moving.

(b) Above the surface includes free-hand gesture recognition that occurs in the space above the surface. These interactions typically point to, select and access content not reachable by the user's direct touch [START_REF] Parker | TractorBeam: seamless integration of local and remote pointing for tabletop displays[END_REF] [24] [START_REF] Vogel | Distant freehand pointing and clicking on very large, high resolution displays[END_REF], or map gestures to particular actions [START_REF] Izadi | Going beyond the display: a surface technology with an electronically switchable diffuser[END_REF] [20] [START_REF] Subramanian | Multi-layer interaction for digital tables[END_REF].

The problem is that the vast majority of this prior research has explored hand interactions only within these two distinct modes. The limitation is that these two modalities ignore the rich interaction space between them. Our perspective differs. We propose and contribute a unification of these discrete interaction modalities called the continuous interaction space. The idea is that many interaction techniques can be developed to not only fit into a particular modality, but that they can leverage the space between them. That is, we believe that the underlying system should treat the space on and above the surface as a continuum, where a person can use touch, gestures, and tangibles anywhere in the space, and naturally move between them (illustrated in Figure 1).

In order to illustrate possible forms of interaction in the continuous interaction space, we introduce and contribute a variety of techniques that exploits the space between these modalities. We constructed these interactions in a way that takes full advantage of this unified space between touch on the surface and the space above it. For each interaction category, we implement an example or refer to prior work that illustrates how that technique can be applied. Thus, we also fit in earlier approaches of extending the interaction around digital surfaces; such as Hilliges' [START_REF] Hilliges | Interactions in the air: adding further depth to interactive tabletops[END_REF] and Wilson's [START_REF] Wilson | Combining multiple depth cameras and projectors for interactions on, above and between surfaces[END_REF] techniques for natural interaction with physically simulated digital objects, Baudisch's [START_REF] Baudisch | Lumino: tangible blocks for tabletop computers based on glass fiber bundles[END_REF] stackable tangible blocks, or Subramanian's [START_REF] Subramanian | Multi-layer interaction for digital tables[END_REF] interaction layers above a surface. While the techniques we introduce are specific to a horizontal tabletop display, the underlying concepts partially generalize to other digital surfaces such as electronic whiteboards and large wall displays.

In summary, our primary contribution is to broaden the design space of interaction techniques for digital surfaces, where we populate the continuous interaction space both with concept categories and examples that emerge from considering this space as a continuum. We don't claim that our categories are complete or our examples perfect. Rather, they show what is possible by considering the entire interaction space. The following section briefly introduces our understanding of the continuous interaction space. We then introduce our infrastructure that allowed us to develop and explore our interaction techniques. The main section of the paper introduces various categories of interaction techniques that illustrate how the unified interaction space can be applied in practice. We close with related work and a conclusion. We define the continuous interaction space as being composed of the direct touch surface and the space above, as illustrated and implemented in Figure 2. We argue that these are not two distinct spaces but instead a single interaction space. Specifically, a person can interact fluently in this 3D area, where gestural acts flow from the space above, to touch, and vice versa. As well, we believe that gestures should not be limited to interactions immediately below one's hands. That is, while the possible physical reach of a person's hands in this space is bounded to around ~1 meter above the surface, gestural acts can extend one's reach beyond these physical limits.

Infrastructure for Rapidly Prototyping Interaction Techniques

In order to explore possible interaction techniques in the continuous interaction space, we built an interactive tabletop system infrastructure for sensing touch and gestures. Atop this, we developed an application test bed that contains a variety of objects that people can manipulate. In particular, a person can view collections of digital content (visible on the tabletop surface in Figure 2): text documents, photos, videos, as well as abstract entities such as files and folders. The person can interact with this digital content through a variety of techniques, such as moving digital content, creating stacks of items, turning pages in digital documents (Figure 3), picking up content to reveal documents underneath (Figure 4 left), navigating videos (Figure 4 right), or browsing stacks of documents (Figure 5). These are part of the interaction techniques that we explain in Section 4 1 .

The underlying infrastructure is implemented atop an interactive horizontal touchsensitive SmartBoard surface [www.smarttech.com] 2 , and a Vicon motion tracking system [www.vicon.com]. This tracking system is composed of 8 high-speed infrared (IR) cameras that track reflective markers illuminated by IR spots attached to the cameras. As long as at least three cameras of the system see each individual marker, the system can infer the accurate 3D position of this marker (to around 1mm). We attached markers to gloves to track people's hands on and above the tabletop surface (see right hand of the person in Figure 2). Our raw information includes the tracking of the hand model which returns the hand's and finger's yaw, pitch, and roll angle and its position in the 3D space (where a distance of 0 to the surface = touch), and the position of the surface in the 3D space. From these, we compute the pointing vector of the fingers to the surface (ray casting), and the normal vector of the hand perpendicular to the surface. We calculate the intersection of these vectors to the surface in screen pixel coordinates (so we know exactly what spot and objects on the surface these vectors are referring to). We also compute the hand's position, direction and vertical distance relative to the surface. We recognize postures such as pinching by measuring fingertip distance from one another. At a higher level, particular gestures (introduced shortly) register for updates of these hand movements.

We also track physical objects on and above the surface, where we attached similar IR reflective markers to tangible objects like wooden boxes (Figure 6), or portable tablet computers (Figure 5). Similar to the earlier described tracking of a person's hand, the system provides accurate position information of these tangibles. By comparing the distance of the tangible objects to the surface or a people's hands, we know whether the objects are placed on the surface or held in a person's hand.

Systems such as the Vicon motion tracking are currently too expensive and unwieldy for deployment out of a lab environment. However, we believe our examples of unified interaction techniques will be implementable in the near future on much more affordable emerging technologies, e.g., shadow tracking [START_REF] Echtler | Shadow tracking on multi-touch tables[END_REF], switchable diffusers [START_REF] Hilliges | Interactions in the air: adding further depth to interactive tabletops[END_REF], depth cameras [START_REF] Wilson | Combining multiple depth cameras and projectors for interactions on, above and between surfaces[END_REF], or any other technology that can accurately detect position and movement above the surface. Currently, these other technologies do not yet provide the necessary accurate tracking of 3D positions of hands, fingers, and objects above a tabletop surface. Our use of marker-based motion tracking allows us to rapidly and accurately prototype and explore possible interaction techniques, where we can re-implement them when these other technologies are ready.

Interaction Categories

To illustrate the possibilities for novel interaction techniques in the suggested continuous interaction space, we now introduce a variety of categories of interaction techniques that make use of this extended space. We explain the concept behind each category. We also introduce one or more examples applying this technique: examples are mostly implemented in our own tabletop system, but a few refer to point systems found in the prior art. Our concepts begin with gestural interaction techniques of a person's hand with digital content. As we will see, some categories not only consider the position and movements of people's hands above the surface, but also people's use of tangible objects or digital devices (usually held in one's hands or placed onto the surface) for interacting with digital content displayed on the interactive surface.

Importantly, we stress that the key idea and contributions of the interaction categories explained below is the flow of hand gestures across the interaction space as a continuum, from touch to moving above the surface. Part of doing so leverages input dimensions such as distance above the surface (where 0=touch), and hand orientation with respect to the surface. By designing such gestures, people can interact with digital content in a very rich, fluid and complex way.

Mirrored Gestures for Redundancy

Mirrored gestures are gestural pairs that redundantly encode identical functionality in either space. That is, a person can invoke the same action via a gesture either directly on the surface or the space above it. Mirrored gestures are related to the concept of equal opportunity in interface design, as suggested by Cockburn and Bryant [START_REF] Cockburn | Leogo: An Equal Opportunity User Interface for Programming[END_REF]. Mirrored gestures offer multiple representations for gesture input, adding to the flexibility of the tabletop interface. The redundancy of functionality lets a person select the interaction technique most convenient in a particular situation, and freely choose to use either a gesture on or above the surface to interact with the digital content.

The mirrored gestures may be different or similar. Our first example uses two different gestures to produce an identical resize action: touch-based pinching with fingers touching the surface vs. a two-handed gesture that brackets an object via L-shaped fingers and thumb, and then stretches or shrinks the area to scale the object. Our second example uses two similar gestures to produce an identical action; in this case the two-handed bracketing gesture can be on or above the surface. A third example considers the navigation through pages in a digital book. To flip a page a person either uses touch to flip a single page through dragging (Figure 3), or does a hand waving gesture above the table that mimics page flipping.

Extended Continuous Gestures to Avoid Occlusion

With extended continuous gestures, a gesture that a person starts through direct touch on the interactive surface can continue in the space above the surface to avoid occlusion of the digital content visible on the tabletop display. That is, the person can lift the hand that performs the gesture off the surface, and continue the gesture in an invisible layer parallel to the touch surface.

As an example, this technique can be applied to allow precise movement of small digital content displayed on the tabletop. A person can grab the object through a 'pick' gesture while touching the surface and move it along the tabletop plane through dragging. The novelty is that this action can be continued by lifting the hand into the 3D space above the surface. Now, this reveals the exact position of the digital object on the screen that was previously occluded by the fingers touching the surface. Once moving the hand above the surface, a normal vector of the plane through the pinching fingers defines the moving position of the object on the screen. In this technique, the actual distance of the person's hand to the surface has no impact on the movement action that is performed.

Extended Reach / Raycasting Gestures

The extended reach or raycasting technique allows a person to extend their interaction range -which is physiologically limited by the reach of their hands -to remote locations on the surface. The fingers of the hand or the arm of a person extend as a pointing ray when the hand is lifted above the surface, where the intersection of this pointing ray with the tabletop plane defines the point of action in the interface. This approach is related to techniques such as Grossman's raycasting for volumetric displays [START_REF] Grossman | The design and evaluation of selection techniques for 3D volumetric displays[END_REF] or Parker's TractorBeam pen raycasting technique on tabletop interfaces [START_REF] Parker | TractorBeam: seamless integration of local and remote pointing for tabletop displays[END_REF]. It differs in the way this technique allows fluid transitions from dragging through direct touch to raycasting remote pointing when lifting the hand off the screen, and back to direct touch.

For example, consider an extended reach gesture for moving an object around a surface. Similar to the extended continuous gesture we introduced before, a person starts by grabbing an object with a 'pick' gesture on the surface, and -while still touching -moves the object by dragging. Lifting the hand off the touch surface affects that action's behaviour. Now, the hand extends to a pointing ray, where the person continues to move the object, even to positions out of reach by direct touch. What is important is that these are not two separate actions. Rather, they are done as a continuous flow.

Lifting Gestures to Reveal Objects

The lifting gesture allows one to virtually lift up digital content, primarily to reveal other content lying underneath. Our example has a person interacting with a large collection of digital documents. To reveal content that is currently occluded by overlapping documents, the person can move their fingers together into a 'pick' posture, and then lift their fingers off the surface into a 'pick-up' gesture. As in real life, this lets them pick up the object above its current surface plane.

The picked-up object becomes increasingly transparent to reveal the other objects that are now underneath it -transparency is a function of the vertical distance of the hand from the surface (Figure 4 left).

Lifting to Adjust Scale Precision

Our next interaction technique allows lifting to adjust scale precision to increase precision of lateral actions.

Here, instead of just using the two dimensional touch surface as input to control a drag operation (e.g., moving a slider), a person can move their hand in the space above the surface to gain increased precision, where the lifting of a person's hand in the Z-axis acts as an input parameter that readjusts the scale precision of the lateral movement.

To illustrate, consider a digital video object that can be searched and played. Using direct touch, a person can drag the finger to the left or right on a video navigation slider, which causes the video to advance forwards or backwards. Yet if the object is small (or the video large), navigating is hard because small touch movements translate to large jumps in the video timeline. With lifting to adjust scale precision, a person initiates and begins the search by first touching the video slider, and then gains precision by lifting above the table (Figure 4 right). Hand height continually rescales the slider, where the same lateral movement will result in a smaller move through the video. Thus both lateral and vertical hand position controls the search.

Interaction with Discrete Layers and Stacks of Digital Objects

Subramanian [START_REF] Subramanian | Multi-layer interaction for digital tables[END_REF] previously suggested that the space above the tabletop surface can be divided into discrete interaction layers. In our adoption of this idea, the space above the surface is divided into multiple parallel planes, each corresponding to a layer. Each of these layers can then (for example) correspond to layers of visual content in the tabletop application, or even to different interpretations of gestures within a layer (e.g., an annotation layer, an editing layer, a movement layer, etc.). A person can navigate through these layers by moving in the Z-axis above the table. In our first example, a person uses a 'pick' gesture with their fingers (or use a pen as in [START_REF] Subramanian | Multi-layer interaction for digital tables[END_REF]) to select content in each of these layers, and even move content from one layer to another.

Our second example reconsiders layers as stacks.

Here, a person browses the digital content ordered in a stack of digital objects (such as photos) by holding and moving a tablet computer in the space above the stack. The screen of the tablet computer then shows the particular photo corresponding to its particular Z position inside of the virtual stack of photos (Figure 5). This technique has two practical limits. First, the maximum height of the virtual stack of documents is limited by the reach of a person's arms. If the person is holding a tablet (as in our 2 nd example), this also limits the visibility of the tablet computer display. Second, the height of each individual layer is limited by the ability of people to select (and stay) inside of these layers.

Subramanian's [START_REF] Subramanian | Multi-layer interaction for digital tables[END_REF] evaluation found a layer thickness of 4 cm optimal for tasks where people have to stay in the layer. In our own implementation of browsing through a stack of documents with a tablet computer, we found a layer thickness of 1.0 cm to 1.5 cm as still practical for selecting content (and holding the device in this position).

Magic Lenses and View Ports

Magic lenses (inspired by Bier et al. [START_REF] Bier | Toolglass and magic lenses: the see-through interface[END_REF]) let a person move a plane above the tabletop to see individual, modified views onto the data that is visualized on the table [17] [27]. This moving plane can be, for instance, a sheet of paper that shows information of a projector (e.g., [START_REF] Izadi | Going beyond the display: a surface technology with an electronically switchable diffuser[END_REF]), or a portable tablet-sized device that shows information. While moving these planes through the 3D space above the surface, the content displayed on the magic lens represents the current view corresponding to the 3D position of the lens. For instance, when visualizing 3D volumetric medial scans on the tabletop, the magic lens can visualize cut sections of that dataset [START_REF] Izadi | Going beyond the display: a surface technology with an electronically switchable diffuser[END_REF].

Stacking of Physical Objects

People can stack tangible objects on a tabletop, where the surface application reacts to the stacks built on the table. Baudisch et al. [START_REF] Baudisch | Lumino: tangible blocks for tabletop computers based on glass fiber bundles[END_REF] proposed this approach with the Lumino project, where a Microsoft Surface tabletop recognizes stacks of small cuboid blocks. Bartindale et al. proposed a related approach detecting stacks of items on a surface, by determining the order of fiduciary markers [START_REF] Bartindale | Stacks on the surface: resolving physical order using fiducial markers with structured transparency[END_REF]. Overall, these approaches allow a person to create physical stacks and structures on top of the usually flat tabletop surface.

6-DOF Manipulation

The free movement of people's hands in the space above the surface can be leveraged to allow full 6 degree of freedom (DOF) interaction with digital content. These dimensions are movement along and rotation around the three axes. In previous work, Hancock contributed 2-and 3-touch techniques that allow 6-DOF manipulations of objects directly on the surface (e.g., [START_REF] Hancock | Shallow-depth 3d interaction: design and evaluation of one-, two-and three-touch techniques[END_REF][START_REF] Hancock | Sticky tools: full 6DOF force-based interaction for multi-touch tables[END_REF]). Our technique differs: instead of using only multi-touch techniques on the surface, a person can 'pick up' any digital object, and modify the object's position and orientation by the way they move their hand above the surface. Our example displays a digital object (a 3D cube) that can be picked up; the cube's position and orientation are then directly mapped to the position of the person's hand (3D location: yaw, pitch, roll). Of course, the movement along this 6-DOF is again limited by the person's reach of the hand, and the rotation is limited by the movement around the wrist joint.

Feedback of Hand and Object Actions by Shadows

While it is relatively easy for a person to know what they are manipulating when they are directly touching it, this becomes harder to do when one's hand is lifted away from the surface as the physical connection is broken. To ease this, the system can provide feedback of how it is recognizing a person's gestures via hand and arm shadows (such as in [START_REF] Hilliges | Interactions in the air: adding further depth to interactive tabletops[END_REF] or [START_REF] Shoemaker | Body-centric interaction techniques for very large wall displays[END_REF]), or by visualizing abstracted hand shapes atop the objects it is manipulating. Examples of such shapes are a pointing hand, a pinching gesture, or the flat hand to reflect user actions atop the object(s) being manipulated. To avoid occlusions of user interface elements, transparency can be used to overlay feedback visualizations.

Digital objects can also provide additional continuous feedback about their status. For instance, objects that are picked up from the table (with a grab gesture) and moved around by the user can render shadows onto the table surface, where the shadow size depends on the current distance of the hand to the surface -or maybe even 'merges' with the shadow of a person's hand as proposed in [START_REF] Hilliges | Interactions in the air: adding further depth to interactive tabletops[END_REF]. This is a very natural mapping of the position in the 3D space to the displayed content on a 2D surface. Of course, all other object behaviours should reflect fine-grained actions corresponding to gestural movement: rotation, transparency, and so on. If done well, a person should be able to understand, self-correct, and fine-tune their gestures to control the object in a meaningful way.

Feedback of Possible Actions by Hovering

Related to the technique above is feedback about what actions are possible as a person moves and hovers above objects. For example, by sensing the position of people's hands above the surface interface, widgets on the touch screen (such as buttons, sliders) can give people visual or audible feedback about possible actions (e.g., [START_REF] Echtler | Shadow tracking on multi-touch tables[END_REF]). Similar to the way GUI buttons often change their color when the mouse cursor hovers over the button, the interactive elements of our tabletop application interface could change their visual appearance (e.g., glowing border, different background color) once a person's hand approaches the widget. This technique would extend the common two-state interaction with touchscreens (i.e., touching the surface or not) to the three-state interaction (i.e., away, hovering, touching) familiar from GUI mouse interfaces or pen-based interaction [START_REF] Buxton | A three-state model of graphical input[END_REF], and possibly even to continuous states where the feedback is very specific to the possibilities allowed at a particular Z-coordinate.

As well, the hovering feedback does not necessarily have to be binary (on or off). Instead, widgets and other on-screen elements can adjust the feedback as a function of a person's hand to the surface. For example, a button could begin showing a thin graphical border when the hand is around 20cm away, and continue increasing the thickness of this border the more the hand approaches the button. This kind of increasing amount of feedback might be in particular useful for widget actions with higher impact (e.g., deleting).

Picking and Dropping Gestures

Another interaction technique in the continuous interaction space is picking and dropping of digital objects, where those objects react according to a physical simulation imitating real-world behaviour. This is also called physics-based interaction (e.g., [START_REF] Hilliges | Interactions in the air: adding further depth to interactive tabletops[END_REF]). That is, a person can pick up a virtual object (say a 3D cube) with a 'pick' gesture, move it to another location, and drop it again by releasing the gesture. The virtual objects displayed on the screen behave according to a simulated physical model and gravity; for example, they fall back to the 'ground', push other objects away, or stop when bouncing against 'walls'. The Z distance can affect this behaviour, e.g., longer falls have larger consequences. This approach of simulating the physics behaviour of digital objects has been the basis for various interactive tabletop systems (e.g., [START_REF] Benko | Cross-dimensional gestural interaction techniques for hybrid immersive environments[END_REF][START_REF] Hancock | Supporting sandtray therapy on an interactive tabletop[END_REF][START_REF] Hilliges | Interactions in the air: adding further depth to interactive tabletops[END_REF][START_REF] Wilson | Depth-Sensing Video Cameras for 3D Tangible Tabletop Interaction[END_REF][START_REF] Wilson | Combining multiple depth cameras and projectors for interactions on, above and between surfaces[END_REF]). Some of the proposed interaction techniques are:  Picking up objects through a 'virtual joint' between the hand and the object; placing objects in virtual containers (e.g., cup); piling and stacking virtual objects [START_REF] Hilliges | Interactions in the air: adding further depth to interactive tabletops[END_REF].

 Swiping objects off the side of the screen onto a person's hand; moving the virtual content on a person's hand to another surface [START_REF] Wilson | Combining multiple depth cameras and projectors for interactions on, above and between surfaces[END_REF].

 Picking and dropping virtual objects, augmented with 2D and 3D visualizations [START_REF] Benko | Cross-dimensional gestural interaction techniques for hybrid immersive environments[END_REF].

 Passing a virtual object from one person's hand to another person's hand (e.g., by dropping the object onto the other person's hand) [START_REF] Wilson | Combining multiple depth cameras and projectors for interactions on, above and between surfaces[END_REF].

In these techniques, the distance of a person's hand to the touch surface can have impact on the physical behaviour of the objects, such as when dropping an object and it falls back onto the surface. Depending on the modelling of simulation, the distance can affect the bouncing of an object (and thus the location where the object lands), or even the possible deformation of an object (e.g., when using rendered 3D shapes).

Picking and Dropping Objects Through Filters

As a novel variation of the above physics based interaction, we created another interaction category: picking and dropping virtual objects through physical objects functioning as a filter. In our example, a person picks a virtual object from the tabletop surface and lets the object fall though the filter (a thin acrylic panel). Once the virtual object 'falls' through the plane of the physical filter, the content of the digital object is modified according to the filter settings associated with this tangible filter object.

Pouring Gesture

When exploiting physics-based interaction, physical artefacts can function as containers for virtual objects, and afford gestures such as pouring to empty the container.

For example, in our prototype tabletop application a person can take a wooden box (Figure 6) and add virtual objects to this container by picking them up from the surface and dropping them from above into the container. The person can then move the box on the table or in the 3D space around the surface (the current virtual objects are displayed as shadows underneath the box). To release the content, the person turns over the box, and thus 'pours' the digital information out of the physical container (see Figure 6).

Related Work

Our work relates to previous research of interaction techniques out of the following three areas: multi-touch interaction with digital tabletop surfaces, 3D and physics based interaction, and tangible objects and digital devices on tabletops. In the following, we briefly review this work.

Multi-touch Tabletop Interaction

Touch-related research is thriving. While most initial work was on detecting contact points (e.g., one or two fingertips), current interest has shifted to whole hand interactions. Wu [START_REF] Wu | Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays[END_REF] and Cao et al. [START_REF] Cao | ShapeTouch: Leveraging Contact Shape on Interactive Surfaces[END_REF] present a touch surface that understands such whole hand touches. Epps et al. studied hand shapes use in tabletop gestures [START_REF] Epps | A study of hand shape use in tabletop gesture interaction[END_REF]; their study suggests the need for both a touch screen and computer vision-based gesture tracking and recognition, for applications that require a wide range of commands. Marquardt et al. introduced the fiduciary tagged glove on the Microsoft Surface that facilitates identifying the hand part that touched the surface [START_REF] Marquardt | What Caused That Touch? Expressive Interaction with a Surface through Fiduciary-Tagged Gloves[END_REF]. Others have explored how hand gestures can control a large scale display from a distance [START_REF] Lee | Gesture-Based Interactions on Multiple Large Displays with a Tabletop Interface[END_REF][START_REF] Malik | Interacting with large displays from a distance with vision-tracked multi-finger gestural input[END_REF][START_REF] Vogel | Distant freehand pointing and clicking on very large, high resolution displays[END_REF], and the influence of virtual embodiments to increase awareness when interacting with digital surfaces [START_REF] Pinelle | The effects of co-present embodiments on awareness and collaboration in tabletop groupware[END_REF].

Recently researchers began to use the space above the surface for interaction. Unlike the approach in this paper, most developed 'point systems' or techniques rather than consider the space holistically. Echtler tracked hand shadows to support hovering actions on a tabletop [START_REF] Echtler | Shadow tracking on multi-touch tables[END_REF], mimicking the mouse hover action. Izadi applied a switchable diffuser to the tabletop, which captures hand gestures above the tabletop [START_REF] Izadi | Going beyond the display: a surface technology with an electronically switchable diffuser[END_REF]. Parker's Tractor beam [START_REF] Parker | TractorBeam: seamless integration of local and remote pointing for tabletop displays[END_REF] studied how pointing with a pen, in addition to touching the surface, could improve interaction by enabling users to get out-of-reach objects [START_REF] Parker | Integrating Point and Touch for Interaction with Digital Tabletop Displays[END_REF]. Benko's muscle sensing technique for people's hands also allows inferring information about their movement above the tabletop surface [START_REF] Benko | Enhancing input on and above the interactive surface with muscle sensing[END_REF]. Other techniques include dividing the space above the surface into interaction layers. Lucero [START_REF] Lucero | Augmenting Mood Boards: Flexible and Intuitive Interaction in the Context of the Design Studio[END_REF] defines gestures that allow vertical movement in-between layers to organize piles of pictures. Subramanian et al. present a multi-layer approach to tabletops [START_REF] Subramanian | Multi-layer interaction for digital tables[END_REF], where people can interact with several layers of visual content by moving their pen in the 3D space above. Kattinakere [START_REF] Kattinakere | Modeling steering within abovethe-surface interaction layers[END_REF] further investigated user performance of steering tasks along paths in such above-the-surface layers. We were inspired by (and thus integrated) concepts out of this body of related research into our categorization of techniques.

3D Gestural and Physics Based Interaction with Tabletops

A few researchers introduced physics based and 3D gestural interaction techniques to interactive tabletops. Wilson's Touchlight [START_REF] Wilson | TouchLight: an imaging touch screen and display for gesture-based interaction[END_REF] and depth sensing above tabletops [START_REF] Wilson | Depth-Sensing Video Cameras for 3D Tangible Tabletop Interaction[END_REF] opened up a field for rich 3D gestural interaction with digital content. Benko introduced cross dimensional gesture interaction techniques [START_REF] Benko | Cross-dimensional gestural interaction techniques for hybrid immersive environments[END_REF] that function as a bridge between 2D and 3D interaction. Grossman investigated interaction techniques with content on 3D volumetric displays [START_REF] Grossman | The design and evaluation of selection techniques for 3D volumetric displays[END_REF]. Later, Hilliges introduced physics based interaction techniques enabling natural interaction with digital content [START_REF] Hilliges | Interactions in the air: adding further depth to interactive tabletops[END_REF]. This technique was later extended with depth-sensing capabilities [START_REF] Wilson | Bringing physics to the surface[END_REF], in order to provide an even richer set of possible gestures. In [START_REF] Wilson | Combining multiple depth cameras and projectors for interactions on, above and between surfaces[END_REF], this approach was taken further by including tracking of multiple interactive surfaces as well as the full 3D volume of the room they are located in. This permits interaction techniques spanning across multiple interactive surfaces.

Research of tabletop systems is partially inspired by other domains. These are, for instance, computer vision techniques for reliable tracking of people's hands and arms captured by cameras (surveyed in [START_REF] Yilmaz | Object tracking: A survey[END_REF]), or techniques developed in virtual reality environment research for sensing 3D hand and object motions [START_REF] Sturman | A Survey of Glove-based Input[END_REF]: optical, markerbased, magnetic, and other tracking approaches.

Grossman's taxonomy of 3D on tabletops [START_REF] Grossman | Going Deeper: a Taxonomy of 3D on the Tabletop[END_REF] categorizes properties of the visualization and interaction with such systems. In this context, we focus on tabletops with 3D volumetric direct input; but are not limited regarding display properties (2D, surface constrained, heads-up projection, or volumetric; see [START_REF] Grossman | Going Deeper: a Taxonomy of 3D on the Tabletop[END_REF]). Our contribution is the emphasis on a continuous interaction space above surfaces, and the set of interaction techniques that in particular leverage the continuous input from people using touch, gestures, and tangibles on and above tabletop surfaces.

Tangible objects and digital devices on tabletops

One restriction when interacting with digital tabletops is the limited tangible expressiveness of a flat tabletop surface: even if the display visualizes 3D objects, a person cannot touch, grab, or feel these virtual objects. Consequently, researchers began to introduce tangible objects in tabletop interaction. In early research, Ullmer's Metadesk introduced physical interface elements in tabletop interaction [START_REF] Ullmer | The metaDESK: models and prototypes for tangible user interfaces[END_REF]. Underkoffler's URP [START_REF] Underkoffler | Urp: a luminous-tangible workbench for urban planning and design[END_REF] combined tangible blocks representing buildings with a tabletop interface to enable intuitive interaction with a digital urban design simulation.

Starner's Perceptive Workbench [START_REF] Starner | The perceptive workbench: Computer-vision-based gesture tracking, object tracking, and 3D reconstruction for augmented desks[END_REF] later combined the recognition of people's gestures above the tabletop surface with tangible computing. Later, Weiss introduced Madgets [START_REF] Weiss | Madgets: actuating widgets on interactive tabletops[END_REF] as physical control widgets on a tabletop (such as buttons, sliders) that can automatically move through an emitted field of an array of electro magnets underneath the table. With Baudisch's Lumino [START_REF] Baudisch | Lumino: tangible blocks for tabletop computers based on glass fiber bundles[END_REF], a tabletop system can also sense the geometrical form of physical cubes stacked on a tabletop.

Tangible objects or digital devices can also facilitate the exploration of visualized digital content. Inspired by [START_REF] Bier | Toolglass and magic lenses: the see-through interface[END_REF], magic lenses allow a person to get a personalized view onto a data set, or cut sections of volumetric data visualized on the table [START_REF] Spindler | PaperLens: advanced magic lens interaction above the tabletop[END_REF]. In another approach, Voida's i-Loupe allows a person use a mobile handheld device to view and interact with the content displayed on a large interactive surface [START_REF] Voida | Getting practical with interactive tabletop displays: designing for dense data, "fat fingers," diverse interactions, and face-to-face collaboration[END_REF].

While many techniques for tangible interaction on tabletop surfaces exist, our categorization focuses on those that in particular leverage the continuous distance of objects and their movements above the surface.

Summary

We proposed the concept of a continuous interaction space above a digital surface, where people can fluently move from touch interaction to gestures above the surface. The idea is simple, yet the implications are profound. Most current interaction techniques are still anchored into falling in one of two modes: either based on direct touch or gestures above the surface. The concept of a continuous interaction space is to remove these arbitrary restrictions, where we want people to consider both modalities working together. Our categorization of techniques is suggestive, where we use them to illustrate an (incomplete) range of new interaction techniques that merge these modalities. Through example implementations and interwoven related work, we illustrated various gesture compositions that make use of this extended space.

Of course, more advanced combinations and extensions of such gestures are not only possible but highly probable. We do not claim that our categories or examples are ideal ones or exhaustive. There is much left to do. Overall, we believe that the understanding and designing gestures that exploit this continuous space above the digital surface is beneficial for creating intuitive interactions with the digital content. Our categories and examples illustrate some of the possibilities. As a starting point they suggest future exploration, iterative refinement, and eventual evaluation.

Figure 1 .

 1 Figure 1. Continuous interaction from the space above the tabletop to direct touch.

Figure 3 .

 3 Figure 3. Turning pages in a digital document through direct touch.

Figure 4 .

 4 Figure 4. Lifting up content to reveal documents underneath (left); navigating video with increased precision (right).

Figure 5 .

 5 Figure 5. Browsing stack of photos by holding and moving a tablet computer.

Figure 6 .

 6 Figure 6. Pouring virtual documents out of a physical wooden box.

All except two(4.7 and 4.8) of the interaction techniques we present have been implemented in our test bed application. We included these two additional techniques as they fit into our categorization of the continuous interaction space.

In order to allow displaying multiple legible documents simultaneously, the projection surface of this interactive table prototype supports a high resolution of 2800x2160 pixels. Figure 2. The continuous interaction space above the interactive tabletop surface.

Acknowledgments

This research is partially funded by the iCORE/NSERC/SMART Chair in Interactive Technologies, Alberta Innovates Technology Futures, NSERC, and SMART Technologies Inc. Ricardo Jota was supported by the Portuguese Foundation for Science and Technology (SFRH/BD/ 17574/2004).