Xiangmin Fan
email: xiangmin@cs.pitt.edu

Youming Liu
email: youmingliu@cs.pitt.edu

Nan Cao
email: nancao@us.ibm.com

Jason Hong
email: jasonh@cs.cmu.edu

Jingtao Wang
email: jingtaow@cs.pitt.edu

MindMiner: A Mixed-Initiative Interface for Interactive Distance Metric Learning

Keywords: M ixed-Initiative Interface, Clustering, Visualization, Convex Optimization, Intelligent User Interfaces, M achine Learning

de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Cluster analysis is a common task in exploratory data mining, and involves combining entities with similar properties into groups. Clustering is desirable in th at it is unsupervised and can discover the underlying structure of data without a priori informat ion. Ho wever, most clustering techniques face one key challenge when used in real world applications: clustering algorith ms expect a quantitative, deterministic distance function to quantify the similarity between two entities. In most real world problems, such similarity measurements usually require subjective domain knowledge that can be hard for users to exp lain. For example, a hu man instructor may easily find that the writing styles of two students are very similar to each other by reviewing their writing samp les. Ho wever, such perceived similarit ies may not be reflected accurately in the distance measurement between two corresponding feature vect ors.

Previous efforts have been made by researchers to imp rove the quality of clustering using both algorithmic [START_REF] Cohn | Semi-supervised clustering with user feedback[END_REF], [START_REF] Wagstaff | Constrained K-M eans clustering with background knowledge[END_REF][START_REF] Xing | Distance M etric Learning with Application to Clustering with Side-Information[END_REF] and user interface [START_REF] Cao | DICON: Interactive Visual Analysis of M ultidimensional Clusters[END_REF], [START_REF] Desjardins | Interactive visual clustering[END_REF], [START_REF] Dy | Visualization and interactive feature selection for unsupervised data[END_REF] approaches. For example, various semi-supervised clustering algorith ms have been proposed by researchers in the machine learn ing co mmunity, either by adapting a similarity measure via user specified constraints or by modifying the process of determining intermediate cluster centers. However, most existing work focuses on theoretical feasibility: they assume users can provide sufficient, unamb iguous, and consistent informat ion to facilitate clustering before the algorithms start. Researchers in HCI and Information Visualizat ion have also explored the use of interactive applicat ions for guided clustering [START_REF] Desjardins | Interactive visual clustering[END_REF], [START_REF] Huang | Exploring Hierarchical User Feedback in Email Clustering[END_REF], [START_REF] Peter | Integrating statistics and visualization: case studies of gaining clarity during exploratory data analysis[END_REF], [START_REF] Seo | Interactively exploring hierarchical clustering results [gene identification[END_REF]. So me interfaces rely on real time feedback of clustering results to help users choose proper features, samples, and the number of clusters to use. Other systems, such as IVC [START_REF] Desjardins | Interactive visual clustering[END_REF], attempt to provide mechanis ms to collect users' a priori knowledge, such as which samples should be in the same group, and which should not. However, most existing intera ctive clustering systems focus on conceptual demonstration and do not address important elements for making such systems practical, such as how to browse, how to manage users' collected a priori knowledge, and how to achieve better clustering results with more representative constraint examples.

To address these challenges, we created a mixed-in itiat ive interface, MindMiner (Fig. 1), to capture users' subjective similarity measurements. MindMiner makes contributions in both interaction design and machine learning algorith ms. MindMiner captures prior knowledge fro m users through active polling with uncertainty and example based visual constraint creation. Active polling with uncertainty enables users to specify their subjective opinion on the global importance of a feature (including the value "not sure") which improves the accuracy and speed of the clustering results. Example based visual constraint creation allows to directly express their a priori domain knowledge via six types of constraints on the data samples being visualized. The constraint management interface allows users to browse existing examp les, investigate the impact of each constraint, and discover conflicting conditions.

MindMiner also provides interface level support that uses active learning to provide optional hints as to which examp les might be mo re helpful for cluste ring. We also report how inequalit ies are formu lated based on the collected a priori knowledge and how the inequalities are used in a convex optimization process to extract the "mental model" of entity similarity fro m users in the form of the Mahalanobis d istance metric.

Specifically, this paper makes the following contributions:

 We propose two interaction techniques, active polling with uncertainty and exa mple-based constraints collection, to collect, visualize, and manage implicit, subjective domain knowledge by scaffolding end-users incrementally. These techniques assume that users' domain knowledge may be ambiguous and inconsistent.  We introduce an improved distance metric learning algorith m that takes into a ccount input ambiguity and avoids trivial solutions1 in existing algorithms.  We present effective active learning heuristics and corresponding interface design to collect pairwise constraints at both entity and group levels. We show in a 12subject controlled study that our design can significantly enhance the clustering relevance.  We present an interactive data explorat ion and visualization system, MindMiner, to help end-users externalize domain knowledge and improve data explo ration eff iciency via distance metric learning. To our knowledge, this is the first interactive system that provides both algorith m and interface level support for handling inconsistent, ambiguous domain knowledge via distance metric learning.

MindMiner in Action

We present a scenario giving an overview of M indMiner. MindMiner was orig inally designed for co mputer assisted peer-review and grading scenarios, but can also be used for other interactive clustering tasks.

Alice is an instructor for a philosophy course with 23 students. There are four writing assignments, and the essays submitted by students are graded via three features (accuracy, clarity, and insight). The grading is done by herself, the TA, and "doubleblind" peer-review by students. Alice feels it is tedious and time consuming to get a clear picture of the overall performance of the whole class. Alice also wants to identify students with similar writing problems so that she can provide customized feedback to them. Alice can use MindMiner to achieve a balance between workload and feedback accuracy. After logging into MindMiner, A lice retrieves student performance data fro m a remote server. Alice believes that writing accuracy is the most important factor she cares about and clarity a close second. She is not sure about the importance of insight. Therefore, she uses the Active Polling Panel (Fig. 2.a ) to make a choice for each feature. She chooses "very important" for accuracy, "important" for clarity and "not sure" for insight.

Then Alice teaches MindMiner her subjective judg ments on performance similarity of students by labeling some examp le constraints. Alice reviews detailed information of the students by mousing over the nodes. MindMiner automat ically selects the most potentially informative pairs and highlights the suggestions with dashed lines (Fig. 2.b). She examines two students involved in a constraint suggestion. After judging that they performed similarly, she drags them together, which creates a must-link constraint between the two students , telling MindMiner that these students should be grouped together. A corresponding symbol fo r this constraint then appears in the Constraints Management Sidebar (Fig. MindMiner also checks if there are any conflicts caused by new constraints. If so, it gives a warn ing by highlighting the corresponding constraints in the Constraints Management Sidebar using a red background. Alice checks the conflict ing constraints and finds that one of the previous examp le constraints she created is not correct so she deletes it. Each constraint item in the Constraints Management Sidebar is doublelin ked with corresponding students via mouse hovering , so it is easy for Alice to diagnose the cause when a conflict is reported by MindMiner.

Alice clicks the "group" button located on the top of the Constraints Sidebar to see whether the examp les provided by her are sufficient for grouping students together in a useful manner. M indMiner applies the updated distance metric using a k-means clustering algorith m, and then displays the resulting groups . Alice then checks the results and finds that the groups are not as good as she expected. She adds a few mo re constraints and then she checks "automatic regroup". In this mode, once there is a new constraint, MindMiner's learning algorith m executes and the system automatica lly regroups the students based on the most updated distance metric. Alice continues this iterative process by adding new constraints, deleting existing constraints or adjusting importance levels of the features, until she gets satisfactory clustering results.

Related Work

We have organized related work into three categories: interactive machine learn ing, clustering interfaces, and semi-supervised clustering algorithms.

Interactive Machine Learning

Because of the inherent amb iguities in hu man activit ies and decision -making processes, many researchers believe that machine learn ing algorithms can never be perfect in replacing hu man experts [START_REF] Shneiderman | Direct M anipulation vs[END_REF]. As an alternative, researchers have investigated mixedinitiat ive interfaces [START_REF] Horvitz | Principles of M ixed-Initiative User Interfaces[END_REF] that keep humans in the loop, provid ing proper feedback and domain knowledge to machine learning algorithms [START_REF] Amershi | Effective End-User Interaction with M achine Learning[END_REF], [START_REF] Amershi | Overview Based Example Selection in End User Interactive Concept Learning[END_REF], [START_REF] Chau | Apolo: making sense of large network data by combining rich user interaction and machine learning[END_REF], [START_REF] Fogarty | CueFlik: interactive concept learning in image search[END_REF], [START_REF] Kapoor | Interactive optimization for steering machine classification[END_REF], [START_REF] Rattenbury | CAAD. An Automatic Task Support System[END_REF], [START_REF] Talbot | EnsembleM atrix: Interactive visualization to support machine learning with multiple classifiers[END_REF].

For examp le, CueFlik [START_REF] Fogarty | CueFlik: interactive concept learning in image search[END_REF] allows end-users to locate images on the web through a combination of keywo rd search and iterative examp le-based concept refinement activities. A lthough both CueFlik and MindMiner support distance metric learning and active learning, there are major differences between the two systems. F irst, CueFlik supports one-class information retrieval wh ile MindMiner focuses on mult i-group semi-supervised clustering that matches a user's mental model. Second, in CueFlik, users' feedback was provided in the form of positive and negative image example s and it is not necessary to handle conflict ing examp les due to the diversity of online images. In co mparison, MindMiner co llects both feature uncertainty informat ion and pairwise relationship informat ion fro m users to formu late a convex optimizat ion pro blem. MindMiner also provides a dedicated constraint management interface to collect, browse user-specific knowledge and resolve prospective knowledge conflict ions, which are common in multi-group clustering tasks.

Apolo [START_REF] Chau | Apolo: making sense of large network data by combining rich user interaction and machine learning[END_REF] is an interactive sense making system intended to recommend new nodes in a large network by letting users specify exemplars for intended groups. Apolo is similar to MindMiner in that both systems accept examp les for intended groups from end-users. However, Apolo focuses on suggesting new members for existing groups in a graph topology, whereas MindMiner aims to generalize examp les fro m end -users, using existing groups to create new groups via distance metric learn ing. MindMiner also works on an unstructured, unlabeled sample space rather than a graph.

In addition to providing representative examples [START_REF] Chau | Apolo: making sense of large network data by combining rich user interaction and machine learning[END_REF], [START_REF] Fogarty | CueFlik: interactive concept learning in image search[END_REF], interactive mach ine learning also allows end-users to specify their p references on system output and adapt to these preferences in model parameters [START_REF] Kapoor | Interactive optimization for steering machine classification[END_REF], [START_REF] Rattenbury | CAAD. An Automatic Task Support System[END_REF], [START_REF] Talbot | EnsembleM atrix: Interactive visualization to support machine learning with multiple classifiers[END_REF]. ManiMatrix [START_REF] Kapoor | Interactive optimization for steering machine classification[END_REF] lets users interactively indicate their preferences on classification results by refin ing parameters of a confusion matrix. In CAAD [START_REF] Rattenbury | CAAD. An Automatic Task Support System[END_REF], users correct classification errors caused by the activity detection algorith m by manually moving docu ments to the correct category. The input matrix for the Nonnegative Matrix Factorizat ion (NMF) algorith m in CAAD also gets updated by such changes.

Clustering Interfaces

Many interactive tools have been designed to facilitate cluster analysis and exploration by providing real time feedback to parameter and dataset changes. For examp le, Hierarchical Cluster Exp lorer (HCE) [START_REF] Seo | Interactively exploring hierarchical clustering results [gene identification[END_REF] is an interactive h ierarch ical clustering system that supports comparison and dynamic querying of clusters. DICON [START_REF] Cao | DICON: Interactive Visual Analysis of M ultidimensional Clusters[END_REF] uses a tree-map style, icon-based group visualizat ions and a combination of k-means clustering and manual g rouping to facilitate cluster evaluation and comparison. NodeTrix [START_REF] Henry | NodeTrix: a hybrid visualization of social networks[END_REF] co mb ines a matrix representation for graphs with traditional node-link graph visualizat ion techniques. Users can select and group nodes to generate an adjacency matrix to visualize cluster patterns in a graph. Many researches also focus on finding clusters in mult idimensional data based on parallel coordinate plots (PCPs) [START_REF] Inselberg | Parallel coordinates: a tool for visualizing multi-dimensional geometry[END_REF]. For example, Fua et al. [START_REF] Fua | Hierarchical parallel coordinates for exploration of large datasets[END_REF] used hierarchical clustering in PCP to detect clusters. Novotny [START_REF] Novotny | Visually effective information visualization of large data[END_REF] use polygonal area in PCP to represent clusters. However none of these techniques incorporate end-user feedback to improve clustering. IVC [START_REF] Desjardins | Interactive visual clustering[END_REF] supports incorporating user specified pairwise constraints in clustering. The authors leveraged the PCK-Means algorith m proposed by Basu, Banerjee and Mooney [START_REF] Basu | Active Semi-Supervision for Pairwise Constrained Clustering[END_REF] for clustering, and the pairwise constraints were incorporated into the kmeans algorith m as a penalty in the cluster assignment state. In contrast to MindMiner, IVC was only tested in simulat ions and there was no method to manage constraints.

Basu, Fisher et al. [START_REF] Basu | Assisting Users with Clustering Tasks by Combining M etric Learning and Classification[END_REF] implemented a document clustering system that combines user specified constraints and supervised classification. However, there was no constraint collection and management interface in their system.

Semi-Supervised Clustering Algorithms

Researchers in machine learn ing have exp lored the use of hu man knowledge in uns upervised clustering [START_REF] Cohn | Semi-supervised clustering with user feedback[END_REF], [START_REF] Wagstaff | Constrained K-M eans clustering with background knowledge[END_REF][START_REF] Xing | Distance M etric Learning with Application to Clustering with Side-Information[END_REF], i.e. semi-supervised clustering. The users' prior knowledge was leveraged in semi-supervised clustering algorithms by either adapting the similarity measure or modifying corresponding search-rules. The semi-supervised clustering algorith m in MindMiner was inspired by the one proposed by Xing et al. [START_REF] Xing | Distance M etric Learning with Application to Clustering with Side-Information[END_REF]. Three major revisions in Xing's algorith ms were made in M indMiner to generate higher quality results on real-world data. First, we added the support for user specified feature uncertainty and two additional groups of pairwise constraints. Second, we incorporated an active learning algorith m and corresponding heuristics to improve the quality of constraints collected. Third, we added a regularization step to avoid trivial solutions derived from the convex optimization.

Design of MindMiner

In the following sections, we discuss these parts in more detail, including the visualization design, the knowledge collection interfaces in MindMiner and the underlying mathematical modeling and the convex optimizat ion algorith m for learn ing the distance metric respectively.

Visualization Design

We use interactive stacked bar charts in MindMiner to visualize clusters of data with mu ltivariate features.

Knowledge Collection Interfaces

MindMiner offers two novel knowledge collect ion techniques, active polling with uncertainty and example-based constraints collection, to make it easier for end-users to externalize their imp licit mental models of entity similarity. We also introduce an active learning [START_REF] Cohn | Improving generalization with active learning[END_REF] heuristic to help users provide similarity examp les that are mo re informative to the follow-up learning algorithms.

Acti ve Polling with Uncertainty. MindMiner lets users specify their perceived importance of each feature via Active polling with uncertainty (Fig . 2.a). Availab le choices are -"not important", "important", "very impo rtant" and "not sure". This step is optional and the default choice is "not sure". These choices correspond to different parameter search spaces in the convex optimization stage. As we illustrate later, e xpressing subjective certainty can reduce the number of examp les needed in the next step and improve clustering quality.

Example -based Constrai nts Collection. MindMiner allo ws users to specify their knowledge on entity similarity via examp les. This ap proach is supported by a ps ychology theory [START_REF] Rosch | Family resemblances: Studies in the internal structure of categories[END_REF], which suggests that people represent categories through exa mples or prototypes. Instead of collect ing absolute annotations or labels fro m end -users, which have been proven by many research findings to be unreliable, especially for subjective domain knowledge, we choose to collect pairwise knowledge instead.

Must-belong

Lets user specify that one entity should be included in a specific group. Leads to multiple must-links, and added as multiple entries in equation ( 2)

Cannot-belong

Lets user specify that one entity should not be included in a specific group. Leads to multiple cannot-links, and added as multiple entries in equation ( 3)

Similar groups

Lets user specify that two existing groups should be put together.

Leads to multiple must-links, and added as multiple entries in equation ( 2)

Dissimilar groups

Lets user specify that no items in the two existing groups should be put into the other group. Leads to multiple cannot-links, and added as multiple entries in equation ( 3)

End-users can provide three types of constraints to represent their prior knowledge on entity similarity (Tab le 1): 1) pairwise entity similarity ( must-lin k, cannot-lin k); 2) entity-group similarity (must-belong, cannot-belong); 3) pairwise group similarity (similar-groups, dissimilar-groups). The latter t wo types include new constraints that are never found in existing semi-supervised clustering algorith ms. All six constraints can be specified by users in the primary interface v ia mouse-based direct manipulation operations (Table 1). Constraints created are shown in the Constraint Management Sidebar (Fig 1 .b). Th is sidebar allo ws users to browse, remove, or check the impact of each constraint created. Conflicting constraints are also highlighted in red. In the clustering stage, the top-right corner o f each constraint in the Constraints Management Sidebar shows whether this constraint is currently satisfied (green means satisfied while red means no). Using visual feedback, rather than directly enforcing them via heuristics, allows end-users to inspect the unsatisfied constraints and refine them if necessary.

Two challenges arise when collecting similarity samp les. First, not all constraints are equally useful. A user could provide multip le examp les, but it might not improve the convergence speed of the convex optimization algorithm or reliability of the distance metric learning process. Second, investigating the similarity between two ent ities or groups can be repetitive, tedious, and demanding on short -term memory.

To address these challenges, MindMiner uses an active learning approach to aut omat ically select the entity/group pairs that could be most informative to the fo llo w-up convex optimization algorith m, and then encourages users to specify their similarities. The detailed active learning algorith m and heuristics are described in detail in the next section.Distance Metric Learning Algorithms Once MindMiner co llects a new p iece of informat ion (feature uncertainty or pairwise sample similarity) fro m users, it converts such information into a set of inequalities, formulates a convex optimization problem [START_REF] Boyd | Convex Optimization[END_REF], and learns a distance metric fro m user provided similarity informat ion. This distance metric is then used for clustering entities into groups. There are four major steps in this distance metric learning process: 1) constraint conflict detection; 2) inequalit ies generation; 3) convex optimization; 4) results regularization. We explain details of each step in the rest of this se ction.

Mathematical Background

An entity in MindMiner is denoted by an n-dimensional feature vector. For examp le, entity s i is represented by (s i1 , s i2 ,…, s in ,) in wh ich n is the dimension in the feature space. The similarity measurement d(s i , s j ) between entity s i and entity s j is defined as:

( , ) ( ) ( ) T i j i j i j d s s s s W s s    (1)
Here W is an n*n distance metric matrix. Letting W=I leads to Euclidean distance. In MindMiner, we restrict W to be diagonal for efficiency concerns, the same framework can be used to learn a co mp lete W with sufficient user examp les. Determining each non-zero element in the diagonal W corresponds to learning a metric in wh ich the different measurement features are given different "weights". Therefore, our goal here is to find W (weights vector) which best respects the information co llected v ia the active polling process and interactive constraint creation process .

Constraint Conflict Detection

The information collected with active polling with uncertainty is used to define the lower and upper bound of the associated weight fo r each feature in the fo llo w-up optimization process. The choice "Very impo rtant" corresponds to a weight of 1 (highest), "not important" corresponds to a weight of 0 (lo west), the weights of "important" features are set to be in a range of [0.6, 1] wh ile "not sure" features are set to be within [0, 1]. In the end, we get a set of ranges for the weights of all features:

WeightBounds WB w w w w 

As shown in Tab le 1, depending on the constraint type, each constraint collected will be converted to one or mult iple pairwise relat ionships and a Boolean flag. For must-link and cannot-link, the corresponding list only contains one pair, with a Boolean flag indicat ing the similarity relat ionship (true for similar and false for d issimilar) between the entities involved in the pair. For other types of constraints, they are first converted to multip le pairwise constraints such as must-links or cannot-links. Then these must-links or cannot-lin ks are added to the pairs list of the corresponding co nstraint.

Algorithm 1. Constraint conflict detection. By using this list based constraint representation, Algorithm 1 presents pseudo code to detect prospective conflicts in the constraints provided by end -users. If a constraint conflict is detected, corresponding constraints in the Constraints Management Sidebar (Fig. 1.b) will turn red. Also, hovering over a conflicting constraint will h ighlight the remain ing constraint(s) in conflict, as well as the corresponding entities and groups.

Active Learning Heuristic

As noted earlier, not all user-specified examp les are equally helpful in imp roving the results from convex optimizat ion. So me examp les could be repetitive and would not justify the time spend by users to specify them or the ext ra co mputer-t ime added to the optimizat ion process. To address this dilemma, we adopted concept of active learning, which allows MindMiner to identify and suggest ambiguous entity relationships that are most informative in imp roving the quality of distance metric learn ing . For example, suppose that an entity S 1 is currently in cluster A and S 1 is on the boundary of A and B while student S 2 is the best exemp lar of A; then the constraint sugges-tion < S 1 , S 2 > would be posed to end-users asking for whether they are similar or not. We designed a three-step active learning heuristic listed below to recommend informative constraint samples to end users. This active learning heuristic will be executed every time when a new constraint is provided by users. The informative entity pairs discovered via active learning are marked with dashed lines in th e main interface.

 Within each cluster c, find the entity with minimu m d istance to the center of c as the exemplar for c.  For each entity s, calculate the difference d between the distances from s to the nearest two clusters c 1 and c 2 . If d is less than a threshold, we mark the entity s as ambiguous.  For each entity s ' that was marked as ambiguous, create a constraint suggestion between s ' and the exemplar of cluster it currently belongs to.

Inequality Generation

We also keep two global sets: S, which is a set of pairs of entities to be "similar" and D, wh ich is a set of pairs of entities to be "d issimilar". All the similar pairs are added to S while all the dissimilar pairs are added to D during the interactive constraint creation process.

After the constraint conflict detection step, we convert the user knowledge co llected through active polling with uncertainty and example-based constraints collection to Weight Bounds which are a set of weight ranges for all features, and S and D wh ich are sets of pairs of similar/dissimilar entities.

A straightforward way of defin ing a criterion for the meaningful distance metric is to demand that pairs of entities in S have s mall squared distance between them (eq.2). However, this is triv ially solved with W= 0 and is not informative. Our approach was primarily inspired by the method proposed by Xing et al. [START_REF] Xing | Distance M etric Learning with Application to Clustering with Side-Information[END_REF]. To avoid the above mentioned trivial solution, we add a new inequality constraint (eq.3) to ensure it takes dissimilar entities apart. In this framework, we transform the problem of learning meaningful distance metrics to a convex optimization problem:

2 ( , ) min ( , ) i j w i j s s S d s s   (2) s.t. ( , ) ( , ) 1 i j i j s s D d s s    (3)
For each : 0(1 )

kk w w k n    (4)
Each sum item in eq.2 corresponds to a positive constraint collected, while each sum item in eq.3 corresponds to a negative constraint collected (Table 1).

It can be proven that the optimization problem defined by eq.2eq.4 is convex, and the distance metric W raw can be solved by efficient, local-minima-free optimization algorithms.

Unfortunately, according to our early experiences on real world data, it is not desirable to use W raw as the distance metric for the follow-up clustering tasks. According to our observations, when the number of constraints is very small, especially at the beginning of a task, convex optimization usually lead to a sparse distance metric where most values in the distance metric are close to zeros, i.e. only minimal features, e.g., 1 or 2 features, are taken into account in similarity measurement, imply ing a trivial solution that does not represent the real-wo rld situation. We use an extra result regularizat ion step and leverage the informat ion collected in the active polling with uncertainty step to generate more meaningfu l distance metric that could be a better representation of a user's mental model.

Result Regularization

In order to make distance met rics respect both feature uncertainty information and the constraints collected by MindMiner, we regularize Wraw by using Weight Bounds (WB). Detailed steps are described in Algorithm 2.

After finishing the result regularization step, we get a W that conforms to all the prior knowledge we co llected fro m end-users. We apply W to the distance metric function and get the relevant distance metric. Then the distance metric W is used in kmeans clustering algorithm to generate meaningful clusters. Algorithm 2. Result Regularization.

Implementati on

MindMiner is written in Java 1.6. The convex optimization algorithm is implemented in Matlab. We use Matlab Builder JA to generate a Java wrapper (i.e. a .jar file) around the actual Matlab programs. MindMiner can run as a desktop application or a web application via Java Web Start.

Evaluation

We conducted a 12-subject user study to understand the performance and usability of MindMiner. We had three basic goals for this study. One was to figure out whether or not the ideas behind MindMiner are easy to understand and if the current MindMiner implementation is easy to learn. The second goal was to evaluate the overall perfo rmance o f M indMiner in capturing the similarity measurements in users' mind s. The third goal was to verify the efficacy of each new co mponent designed (i.e. active learning heuristics, example based visual constraint creation, and active polling with uncertainty). The data loaded in MindMiner in this study was anonymized real world data fro m a 23 student philosophy course in a local university with permission from the internal review board (IRB) and the instructor.

Experimental Design.

The study consisted of five parts: Overview. We first gave participants a brief introduction and a live demo of MindMiner. We explained each task to them, and answered their questions. After the introduction, we let the part icipants explore the interface freely until they stated e xplicitly that they were ready to start the follow-up tasks.

Clustering and acti ve learning. We used a within-subjects design in this session. There were two similar tasks: task 1 was clustering the students into four groups based on their performance in the first assignment; task 2 was the same as the previous task except that users were to only consider the "accuracy" features of the assignments. There were two conditions in this section: (A) providing constraint suggestions via active learning; (B) without active learning. Six participants performed task 1 with condition A and task 2 with condition B. The other six performed task 1 with condition B and task 2 with condition A. The order of the two tasks was counterbalanced. Each part icipant could provide up to ten examp le-based pairwise constraints (both positive examp les and negative examples) for each task. The active polling with uncertainty feature was disabled in both conditions. We collected each participant's task complet ion time fo r each condition and the distance met rics derived by the learning algorithm.

Acti ve polling with uncertainty. We used a between-subjects design in this session with two conditions: the constraints & active polling condition and the co nstraints-only condition. The active learning feature was enabled in both conditions. The task required users to find five students with similar performances to one student named "Indrek". We told the participants that the accuracy and clarity features of the first two assignments were very important to consider and asked them to define the importance of other features themselves. We hypothes ized that given meaningful clustering results, one can find similar students easily just by going over each student in the target's group. Otherwise, if the clustering was not good, the participants would have to view groups besides the target's group to find similar students.

Free explorati on. In this session, the participants were asked to group the students into three categories based on their own grouping criteria. Users were encouraged to think aloud and even write down their rules on a piece of paper. They were also encouraged to explore MindMiner as long as they wanted.

Qualitati ve feedback. After participants completed all the tasks, they were asked to complete a questionnaire and describe their general feeling towards our system.

5.2

Participants and Apparatus.

We recruited 12 participants (5 female) between 22 and 51 years of age (mean = 27) fro m a local university. Two were instructors from physics department and psychology department respectively. The other ten were graduate students who have teaching experience. Each study lasted for around 60 minutes (up to 90 minutes maximu m), and each participant was given a $10 gift card for the time.

A Lenovo Thin kPad T530 laptop computer with Intel Core i5 -3210 CPU, 4GB RAM, running Windows 7 was used. An external NEC 23 inch LCD mon itor with a resolution of 1920*1080 was attached to the laptop to run MindMiner.

Evaluati on Results

Clustering and Active Learning. Fig. 4. Average cosine similarities between "gold standard" and distance metrics learned by different numbers of constraints (the higher the better).

The average task completion time in the "with active learn ing" condition is significantly shorter than that of the "without active learning" condition (266.4s vs. 357.4s, F 1, 11 =13.403, p<0.01). We observed that with active learning suggestions enabled, participants tended to compare the students involved, instead of randomly p icking several students to compare. MindMiner suggestions gave them clear "targets" to inspect; otherwise, they would look for "targets" themselves, which usually leads to more time. Furthermore, when there were no suggestions , participants had to compare mu ltip le students, which required having to remember many students' scores. In co mparison when they had system suggestions, they only need to compare two students.

To evaluate the quality of distance met rics learned in the t wo conditions, we defined our "gold standard" to be a weight vector where the weights of predefined important features are 1s, and the weights of other features are 0s. We used cosine similarity between the standard weight vector and the weight vector learned fro m our algorithm to measure the quality of distance metric learned (Fig. 4).

Analysis of variance revealed that there was a significant difference (F 1, 11 =7.42, p<0.05) in the quality of the distance metric learned. We found that there was a significant main effect (F 3, 9 =19.30, p <0.05) in quality among different nu mbers of constraints collected. Pairwise mean comparison showed that more constraints led to significantly better quality d istance metrics. With the same nu mber of constraints, the quality of distance metrics learned with active learning was significantly higher than that without active learning for all four numbers of constraints in Fig. 4.

Active Polling with Uncertainty.

When active poling with uncertainty was enabled, the average co mp letion t ime was 252.7 seconds (σ =19.6). When disabled, the average comp letion time was 304.8 seconds ( σ =43.1). However, the difference was not statistically significant (p=0.297).

The active polling with uncertainty condition also led to significantly more similar students discovered (4.67 vs. 2.50, p<0.001) than the condition without active polling (Fig. 5). This finding showed that active polling with uncertainty could also facilitate users by helping them to learning process to derive more relevant entities .

Free Exploration.

A total of 458 interaction activit ies were recorded in the free explorat ion session (Fig. 6). Examin ing details panels was the most frequent activity (51.1%), fo llo wed by adding constraints (20.1%). Of all the 92 constraints added by participants in this session, 74 (80.4%) were fro m system suggestions. Other frequent activities included using the active polling with uncertainty feature (13.8%), grouping entities (8.3%), checking existing constraints (5.0%), and deleting constraints (1.7%). A mong all the 8 constraint deletions, 6 were unsatisfied inappropriate constraints and 2 were constraint conflicts.

We observed that participants tended to add more positive e xamp les (must-link, must-belong, and similar-groups) than negative examp les (cannot-link, cannotbelong, and dissimilar-groups) (78.6% vs. 21.4%) when the active learn ing feature was disabled. Participants tend to not provide negative examp les even when th ey were confident that two entities were very different; when the active learning feature was enabled, the ratio of negative examples almost doubled (40.8%) and the difference was statistically significant. This observation indicated that the current activ e learning interface and heuristics in MindMiner can increase users' awareness and contribution to negative examples. Although participants were encouraged to take a look at the suggested entity relationships first before searching for their o wn examples in the act ive learning cond ition, some subjects chose not to do so. When asked for why, the reasons were either that they didn't completely trust the computer or that they simply enjoyed the feeling of finding examples fro m scratch. In either case, the active learning suggestions provided hints for reducing their example finding efforts. Subjective Feedback. Fig. 7. Subjective ratings on a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree).

Overall, participants reported positive experiences with MindMiner. Participants felt that the system improved their understanding of students ' performance through peerreview data (Fig. 7).

Conclusion

We presented MindMiner, a mixed-init iative interface to capture domain experts' subjective similarity measurements via a combination of new interaction techniques and machine learning algorith ms. MindMiner collects qualitative, hard to express similarity measurements from users via active polling with uncertainty, example based visual constraint creation and active learning. MindMiner also formu lates human prior knowledge into a set of inequalit ies and learns a quantitative similarity distance metric via convex optimization. In a 12-subject user study, we found that 1) MindMiner can capture the implicit similarity measurement fro m users via examples collection and uncertainty polling; 2) active learning could significantly imp rove the quality of distance metric learn ing when the same numbers o f constraints were collected; 3) the active polling with uncertainty method could improve the task completion speed and result quality.

Fig. 1 .

 1 Fig. 1. The primary UI of M indM iner, showing 23 students in a college-level philosophy class grouped into five clusters based on their performance (accuracy, clarity, and insight) in four writing assignments using six example constraints specified by an instructor. M indM iner consists of three parts: (a) The Active Polling Panel allows users to optionally indicate the importance for each measurement. Each colored square box represents one feature (4 assignments x 3 features). The rectangular bars beneath show real-time updates of the corresponding "weights"; (b) The Constraints Management Sidebar displays example-based constraints collected; (c) The Interactive Visualization Workspace lets a user see detailed information about entities, create example-based constraints, split and combine groups, examine and refine clustering results and examine personalized groups.

Fig. 2 .

 2 Fig. 2. Knowledge collection interfaces of M indM iner. a: Interface for active polling with uncertainty. b: Interface for example-based constraints collection.

  1.b). She later creates a cannot-link between dissimilar students by right clicking and dragging from one to the other. Every time Alice adds a new constraint, the distance metric learning module runs a convex optimization algorith m to derive the optimized solution. The bars in the Active Polling Panel (Fig. 1.a) show the updated weights of corresponding feature dimensions in real-time.

  Fig.3illustrates an examp le of our design in wh ich a student dataset is visualized. Each student, treated as an entity, is characterized by his/her performances in a writing course along different features, i.e. accuracy, clarity, and insight. These features are defined by the user, and are measured based on the peerreview scores of three writing assignments.As shown in Fig. 3.a, we use different background colors to illustrate different assignments, and use different foreground colors to rep resent the different features. A student's feature vector is represented as a bar chart (Fig. 3.b) in wh ich the sizes of the bars represent the corresponding review scores. Similarly, we represent a clustered group of students (Fig. 3.c) by packing all of the students' review scores together into a stacked bar chart, categorized by assignments (Fig. 3.d). We also represent the averaged student feature scores of each assignment as another grouped bar chart attached to the group. The position of the bar chart, i.e. left, right (defau lt location, Fig. 3.d), bottom, and top, can be customized by users. The resulting visualizat ion shows the overall distribution of data while keeping individual details easily visible.

Fig. 3 .

 3 Fig. 3. M indM iner visualization design. a) Feature vector of a student based on three writing assignments and three different features. b) Student barchart icon. c) A group of similar st udents. d) Stacked bar chart icon for a cluster of students.

Table 1 .

 1 Symbols and descriptions of the six pairwise constraints supported by M indM iner. Collected constraints are shown in the Constrains Management Sidebar (Fig. 1.b). that two entities should be grouped together. Leads to a new entry in equation (2) Cannot-link Lets user specify that two entities should not be grouped together. Leads to a new entry in equation (3).

Fig. 5 .

 5 Fig. 5. Average number of similar students discovered by condition (the more the better).

Fig. 6 .

 6 Fig. 6. Activity distribution of participants.

When the number of constraints is small, e.g., less than

[START_REF] Peter | Integrating statistics and visualization: case studies of gaining clarity during exploratory data analysis[END_REF], existing algorithms tend to generate trivial distance metrics that have only one or two non-zero dimensions.


Nan Li
email: nan.li@epfl.ch

Łukasz Kidziński
email: lukasz.kidzinski@epfl.ch

Pierre Dillenbourg
email: pierre.dillenbourg@epfl.ch

Augmenting Collaborative MOOC Video Viewing with Synchronized Textbook

Keywords: M OOC, peripheral display, contextual help, collaborative learning 1

We designed BOOC, an application that synchronizes textbook content with M OOC (M assive Open Online Courses) videos. The application leverages a tablet display split into two views to present lecture videos and textbook content simultaneously . The display of the book serves as peripheral contextual help for video viewing activities. A five-week user study with 6 groups of M OOC students in a blended on-campus course was conducted. Our study in this paper reports how textbooks are used in authentic M OOC study groups and further explores the effects of the book-mapping feature of the BOOC player in enhancing the collaborative M OOC learning experiences.

Introduction

Massive Open Online Courses (MOOCs) are in recent years growing in popularity. Popular platforms such as Coursera and edX typically replicate traditional classroom pedagogy online, featuring with v ideo lectures , quizzes, tutorials, d iscussion forums and wikis. A mong these learning components, lecture videos play a central role in MOOC learning; the foru ms prov ide peer-to-peer support analogous to that in the classroom, and other components mainly serve as supporting resources. Note that textbooks are not essential for MOOCs . Their ro le as references is perhaps partially displaced by the lecture videos. This does not mean textbooks are not useful in MOOC learning. In a M OOC taught with a co mpanion textbook, Belanger et. al [START_REF] Belanger | Bioelectricity: A quantitative approach--Duke University's first M OOC[END_REF] found that the students often spontaneously identified related content in the textbook, then shared and discussed them in the foru m. This finding does not only exhibit that textbooks still function as potentially effective supporting materials for MOOCs, but also indicates the potential needs for providing the students with supervised book-tovideo references.

Apart fro m textbooks, study groups, which is a common collaborative learning setting at schools and universities, are also suppressed by MOOCs, particularly by its massive and distributed nature. Compared to online foru ms and online groups, collocated study groups allow more intimate discussions. In our recent work [START_REF] Li | Watching M OOCs together: investigating co-located M OOC study groups[END_REF], we have shown that students are enthusiastic about studying MOOC in face-to-face groups. We also exp lored how students studied in different group arrangements, i.e. watching videos on individual displays or on a shared display. The key question to be addressed by this paper is, however, about a combinational usage of study group and textbook. We are interested in how the students use textbooks in their groups . Particularly, when referenced textbook content is lin ked to the videos, how does it affect the students' collaborative learning behaviors?

Textbooks are known to offer more co mprehensive learn ing materials. When the books are linked to videos, the support becomes contextual help. Such help has been recognized as effective means for learning graphical interfaces [START_REF] Chilana | LemonAid: selection-based crowdsourced contextual help for web applications[END_REF]. In fact, most of the contextual help literature dealt with step-by-step web tutorials, and only several investigated video-based tutorials. Pause-and-Play [START_REF] Pongnumkul | Pause-and-play: automatically linking screencast video tutorials with applications[END_REF] p roposed a method to detect taskperforming events in the video and link them with user actions in the target applic ation as the user tried to imitate the procedure. Th is method avoids manually switching between the user context and the online tutorial. ToolClips [START_REF] Grossman | ToolClips: an investigation of contextual video assistance for functionality understanding[END_REF] embedded video tutorials as contextual assistance for tool functionality understanding in a software application; FollowUs [START_REF] Lafreniere | Community enhanced tutorials: improving tutorials with multiple demonstrations[END_REF] demonstrated that video tutorials could be enhanced by mu ltip le demonstrations of tutorial videos from other co mmunity members . These projects endeavored to create links between graphical interface to be learned and video tutorials. Practicing the software is the main activity that renders the context whereas the videos are seen as external helps. However, situations are poles apart in the M OOC scenario, where video viewing is the central activ ity. Additionally, video viewing is more passive and engaging, capturing most of the learners' attention.

In this paper, we evaluate BOOC, a MOOC video player that synchronizes lecture videos with relevant textbook pages in an authentic study group setting. The book pages are displayed in the periphery to minimize d istractions. The paper contributes in understanding the roles of textbooks, especially when used as peripheral contextual help in MOOC study groups. We reveal how the students leverage the usage of books and peer discussion to study MOOC in groups . As shown in Fig. 1, the BOOC user interface is separated into two parts, a v ideo controller plays the lecture v ideo and a PDF controller manages page navigations. Each controller has a horizontal status indicator, both of which are in itially red, indicating the contents are synchronized. Double clicking on the PDF controller toggles the synchronization. Desynchronized PDF controller will a have green indicator, which contrasts the other one in red, suggesting asynchronicity.

The two controllers are synchronized by default. If the video content is beyond the text book, the PDF controller is greyed out, indicating that no textbook pages are relevant at the mo ment. In case mult iple pages are related to the same v ideo segment, the most relevant one is presented. Others have only their page nu mbers shown as yellow text in the middle of the screen. Students can navigate to those pages at their own effort, if they wish.

Videos and books are dually mapped. Users can also navigate through the digital book to get the corresponding video explanations, if availab le. In case of mu ltip le mappings, the system pops out a list of other relevant videos for selection.

Fig. 2. CC (left) and DD (right) Group Conditions

User Study

Our study was based on an undergraduate-level MOOC offered by our university at Coursera, Numeric Analysis. The on-campus version of the course was arranged as 7week flipped teaching with MOOC plus 8-week trad itional teaching. In the flipped teaching period, the students were required to watch videos and solve quizzes at home. Classroom sessions were reserved for exercises and advanced tutorials. We recruited 25 volunteered on-campus students (8 females/ 17 males) fro m the course. Each subject was co mpensated 150 SFr. together with a print co mpanion textbook for participating 5 weekly study sessions. Since it is common for students to study with familiar fellow students, we allo w them to spontaneously form 6 groups by themselves.

Each group met weekly around a table to study together. Activities typically included watching videos and solving quizzes. Half of the groups shared a tablet that was connected to a beamer for v ideo viewing, and we call this condition CC (Centra l-ized display Centralized control, see Fig. 2 left). Students from the rest groups watched videos on her own tablet (borrowed fro m us) independently with earphones, and this condition is referred to as DD (Distributed display Distributed control, see Fig. 2 right). All the groups were co mposed of 4 students, except one DD group h as 5 participants. We believe the two group conditions both naturally exist in real life, so we would like to understand how the groups study and use books accordingly. In other words, in spite of the group conditions, the study is more of an exp loration rather than a comparison of experimental designs.

In addition, we d id not instruct the students how to watch videos or collaborate. They were encouraged to behave naturally. However, we manipu lated the forms of the video player. The subjects watched the videos for the first three weeks with a normal video player (without PDF); the BOOC p layer with book mappings was used only for the last two weeks. To our knowledge, there was no reliable automatic way to map book content with videos precisely, so we made it manually with the help of a designated teaching assistant of the course. Each week's videos (less than 10) took around one hour to finish the mapping.

For the data collect ion, we have video interaction logs obtained from the tablet video player. We also videotaped the study sessions so as to analyze the book usage and speech fro m the recordings. Two cameras were employed to capture both the front and rear view o f the group interactions. In addition, there were weekly recurring post-experiment questionnaires with repeated questions regarding group collaboration, quality of d iscussion as well as book usage. Each session was followed by a semi-structured interview so as to obtain deeper insights of their learning experiences .

Results

The event logs recorded the interactions on the BOOC player, but print book interactions as well as collective interactions involving mult iple subjects need to be manually coded fro m the video recordings. One of the authors coded 440 book interactions found from video recordings of 30 study sessions (2-3 hours each), and identified the following interactions modes:  Browse. Turning the print book rapidly to look for relevant pages  Gli mpse. Glancing at the book to follow the video (mostly) or quiz. Such interactions typically lasted shortly, and the students' eyes usually quickly moved back-and-forth between the book and the video / quiz sheets  Read. Resting the eyes on the print book page for longer time to read text  Turn. Turning pages on the PDF book.  Scroll. Scrolling the PDF book to view different parts of the page.  Zoom. Zooming in / out to see details in the PDF book  Talk. Talk ing to other members with the book , sometimes with deictic gestures Among these modes, Browse, Glimpse and Read are individual interactions on the print book; Turn, Scroll and Zoom are associated with the embedded PDF in the BOOC player. An interaction becomes collective if it is performed with concious awareness of mutiple group members , e.g. when they have shared interest in the book or discuss the book. Talk is a collect ive verbal interaction by defin ition. The presented interaction modes can be easily identified fro m the videos with the help of both front view and rear view recordings. In princip le Glimpse and Read also apply to the PDF, but it is difficu lt to tell fro m the v ideo recordings without eye tracking, since the videos and PDF reside in parallel.

Textbook Usage in Pre-BOOC Sessions

In the first three weeks, the students watched lecture videos on normal tablet video players, and their companion textbooks were all p laced on the table. Each week, a new sequence of videos was available, and we found that the students usually watched them in order. As soon as a group finished watching a video, they often had short discussions about the just-watched video or the associated quizzes before starting the next v ideo. This period of time is referred to as video break , as compared to video pause, which refers to the time when a video is paused during its watching session. The DD students tended to synchronously start the videos and also tried to fin ish the videos at the same time so as to start discussions together. More thorough discussions about this phenomenon are presented in [START_REF] Li | Watching M OOCs together: investigating co-located M OOC study groups[END_REF]. The book was not essential for studying the course, so the usage was voluntary. Not every group had used books in every session, and our goal was not to predict the usage. Rather, our interest lies in the interaction modes when the book was used. Fig. 3 illustrates the frequency of each print book interaction mode of the DD and CC groups in the first three weeks. We find that the time period when the book interactions occurred differed significantly between CC and DD ( 2 (200,1)=34.98, p < .0001,  = 0.43). The CC groups had more balanced usage of book between during the video break and video watching, while the DD students mostly used the textbooks in video breaks. This was especially notable for the Glimpse action, which indicated the student was following the videos with textbooks in parallel. The frequency is visibly high in the CC but none in the DD. A possible exp lanation could be the DD students did not want to break their video watching synchronicity for using the book.

Fig. 3. Frequencies of the Print Book Interaction M odes in Pre-BOOC Sessions

Overall there were 20 occasions where the students grabbed the book on the table and looked for relevant content while watching v ideos (Browse). 32.5 seconds on average were spent on each Browse. In 3 out of the 20 Browse occasions, the students failed to find the intended informat ion on the book. Co llective book interactions were only in the form of Talk with print books. Usually we found a student asked questions to others with the book or read aloud the book to the group. Talk interactions account for respectively 10.6% and 12.5% in CC and DD and mostly (82.3% and 80.0%) happened during video break.

During the semi-structured interviews, we t ried to understand why some of the students did not use the book during the interviews. Main reasons include (1) they were afraid of loosing time in looking for informat ion in the book (2) they did not know exactly what is not clear while watching videos (3) the lectures were easy, so the videos are sufficient for learn ing (4) they prefer to ask the groups, which usually solved their problems. Feedbacks (3-4) reflect more of contextual constraints in the collaborative MOOC learning setup, where the students' doubts about video content depend on the course itself. They may prefer to address the problems through discussions in the group rather than the book. Feedbacks (1-2) as well as the observations presented before confirm the potential needs for peripherally displaying book references, so that the students can quick judge the usefulness of book content.

Textbook Usage in BOOC Sessions

We deployed the BOOC player with book mappings in the last two weeks of the study, but students were still asked to take out their print textbooks during the study sessions for potential usage. In fact, only one student had used the print book in the last two weeks, because he favored the tangibility of the print book. Still, h e used the PDF mapping in BOOC to help quickly navigate to the pages.

To analyze the usage of the peripherally displayed synchronized PDF, we first summarize the usage scenarios collected fro m the questionnaire as below: (1) Extended Knowledge: when they saw another explanation of a concept with detailed theorems and examp les (2) Alternati ve Presentation: when the teacher was talking too fast or the videos were not visibly clear (3) Information Confirmation: when the students had doubts about certain concepts and need to confirm their understanding (4) External Hel p: when none of the group members knew the answer or when they were arguing about certain concepts. Among these usage scenarios, the advantage of the peripheral design is especially notable in [START_REF] Belanger | Bioelectricity: A quantitative approach--Duke University's first M OOC[END_REF], where the students had no explicit needs of help -They actually d iscovered helpful informat ion in the peripheral display of the book with serendipity.

Fig. 4. Frequencies of PDF Interaction M odes in BOOC Sessions

Fig. 4 depicts the frequency of book interactions by mode with BOOC interface. It is not surprising to see that in both conditions, the interactions predominantly happened during video watching rather than during video break due to the book-mapping feature. What is mo re interesting is that the DD students used the book frequently during video watching, wh ich seldom happened before. A possible exp lanation could be the synchronized PDF increased the visibility and accessibility of the potentially useful informat ion in the book. As a result, the students were offered better opportunities to address their situational needs without the fear of loosing synchronicity.

For the students in the CC condit ion, the most notable change after the introduction of BOOC is the increased frequency of co llect ive activ ities, especially during video watching. The proportion of co llect ive activit ies doubles to 20.9%. If we count only Talk interaction, the proportion is 15.2% , with 75% happened during video watching, as compared to 17.7% in the pre-BOOC sessions. The reason behind the increments, we believe, is that the shared display of synchronized book content increased shared attention, so that the students could have more chances to collaborate with the book. One may wonder why the increment of book discussion matters. In fact, when the quality of discussion (5-point Likert scale ratings obtained from weekly questionnaires) in the CC groups was predicted, it was found that the frequency of Talk interactions during video watching (=0.24, p < .05), the proportion of speech during video break (=5.58, p < .001) were significant predictors. The statistics were computed fro m a mixed-effect multip le-regression model where the students nested in groups were modeled as random effects. The overall model fit in terms of R-squared was 0.48, and the marginal R-squared contributed by the fixed effects was 0.21. Significance could also be obtained if we fitted in the model with the frequency of all collective interactions or the duration of Talk interaction, instead of Talk frequency. This finding indicates that more discussions with book or collective book usage in general may increase discussion quality.

Remember that when we designed the BOOC interface, we enabled dual mapping between the PDF and the video. During the experiment, lin king fro m book pages to videos were never intended, since watching videos was their main activ ity, not read-ing books. The students were sometimes annoyed due to abrupt video changes when they accidentally swiped the PDF to a page with a d ifferent video mapping. We argue dual mapping might be useful when used at home than in time-bounded group study sessions, where the students' activities are much centered on watching videos.

Conclusion

In this paper, we presented how textbooks were used in t wo co mmon formats of authentic MOOC study groups. Particularly, we summarized the students' feedback on why print textbooks were not widely accepted and when the BOOC interface , which was essentially a peripheral contextual help display, was considered useful in the group study sessions. We also delivered an emp irical categorizat ion of book interaction modes during study group sessions. These qualitative findings increased our understanding of the role of the textbooks in collaborative MOOC learning setup.

In addition, we d iscussed how the book usage was changed since the introduction of BOOC player in a quantitative way. Most importantly, we found the BOOC player increased collective book usage for the CC groups, wh ich in turn enhanced the students' discussion quality significantly. These findings further revealed the effectiveness of the peripheral book display as it increased mutual awareness of contextual informat ion. Even in the DD condition, the d isplay worked in the sense that it provided situational help to the students without losing much synchronicity, which is essential in the DD style study groups [START_REF] Li | Watching M OOCs together: investigating co-located M OOC study groups[END_REF]. Future work includes replicating similar interfaces to the more dynamic online platforms, and contextual helps other than textbook may be employed to facilitate students' learning as well.

Fig. 1 . 2 The

 12 Fig. 1. The BOOC Player


Evan Morgan
email: e.l.morgan@qmul.ac.uk

Hatice Gunes
email: h.gunes@qmul.ac.uk

Nick Bryan-Kinns
email: n.bryan-kinns@qmul.ac.uk

The LuminUs: Providing Musicians with Visual Feedback on the Gaze and Body Motion of their Co-Performers

Keywords: Musical interaction, computer-supported cooperative work, groupware, eye-tracking, social signals, non-verbal communication

This paper describes the LuminUs -a device that we designed in order to explore how new technologies could influence the interpersonal aspects of co-present musical collaborations. The LuminUs uses eye-tracking headsets and small wireless accelerometers to measure the gaze and body motion of each musician. A small light display then provides visual feedback to each musician, based either on the gaze or the body motion of their co-performer. We carried out an experiment with 15 pairs of music students in order to investigate how the LuminUs would influence their musical interactions. Preliminary results suggest that visual feedback provided by the LuminUs led to significantly increased glancing between the two musicians, whilst motion based feedback appeared to lead to a decrease in body motion for both participants.

Introduction

There has been much focus recently on the use of wearable sensors for lifestyle, sports and activity tracking. Our research concerns the use of wearable devices as tools for understanding and enhancing interactions between musicians during co-present performances. In a previous study we assessed the suitability of various sensors for providing continuous information on the affective and behavioural aspects of collaborative music making [START_REF] Morgan | Using affective and behavioural sensors to explore aspects of collaborative music making[END_REF]. Following on from this work, we decided to design and test a device that would enable musicians to receive real-time sensor-based feedback on aspects of their collaborative interactions. We chose to focus specifically on two non-verbal interaction features that have important expressive and communicative roles in musical performance -eye gaze and body motion.

Studies have shown that gaze has a variety of functions in non-verbal communication, including the expression of emotional and interpersonal information such as liking, attentiveness, competence, attraction, and dominance [START_REF] Kendon | Some functions of gaze-direction in social interaction[END_REF][START_REF] Kleinke | Gaze and eye contact: a research review[END_REF]. It may also be used as a means of directing attention towards shared references and surveying activity in the external environment [START_REF] Argyle | The central Europe experiment: Looking at persons and looking at objects[END_REF]. Schutz notes that visual co-presence during collaborative music making allows each performer to share "in vivid present the Other's stream of conciousness in immediacy" [5, pp. 176]. Despite its apparent significance, surprisingly few studies have been conducted on the use of gaze during collaborative music making. Davidson and Good [START_REF] Davidson | Social and Musical Co-Ordination between Members of a String Quartet: An Exploratory Study[END_REF] observed that members of a string quartet used "conversations with the eyes" to convey important information while playing. A study with a string duet found that visual contact had a positive influence on timing synchrony between musicians for certain types of music [START_REF] Vera | A Study of Ensemble Performance Under Restricted Line of Sight[END_REF].

Eye-tracking technology makes it possible to automatically track where someone is looking. Existing studies have predominantly investigated how eye-tracking might be used to support and evaluate the success of computer-supported collaborative work [START_REF] Vertegaal | The GAZE groupware system: mediating joint attention in multiparty communication and collaboration[END_REF][START_REF] Chanel | Assessment of Computer-Supported Collaborative Processes using Interpersonal Physiological and Eye-Movement Coupling[END_REF]. Music related studies have generally focused on the potential for eye-tracking to be used as a control medium (e.g. selecting notes by glancing at specific markers -for a review see [START_REF] Hornof | The Prospects For Eye-Controlled Musical Performance[END_REF]). We are not aware of any studies that have used eye-tracking to investigate or support the interpersonal and affective functions of gaze during collaborative musical interactions.

As an intrinsic feature of non-verbal communication, body motions are closely tied to gestural [START_REF] Goldin-Meadow | Beyond words: the importance of gesture to researchers and learners[END_REF] and emotional expression [START_REF] Kendon | Some functions of gaze-direction in social interaction[END_REF]. Humans are able to successfully identify emotions from simple point-light representations of body motion [START_REF] Walk | Emotion and dance in dynamic light displays[END_REF][START_REF] Clarke | The perception of emotion from body movement in point-light displays of interpersonal dialogue[END_REF]. More specifically, studies have found evidence for a link between measures of body motion activity and the arousal/activation aspects of emotion [START_REF] Castellano | Recognising Human Emotions from Body Movement and Gesture Dynamics[END_REF][START_REF] Kleinsmith | Affective Body Expression Perception and Recognition: A Survey[END_REF]. Body motion analysis has also been applied to the study of affect in musicians [START_REF] Dahl | Visual Perception of Expressiveness in Musicians' Body Movements[END_REF]. Motion measurements can be obtained using accelerometers worn at positions of interest. Many consumer electronic devices -such as smart phones, smart watches, and activity trackers -already contain accelerometers, from which data can be accessed using custom built applications.

In this paper we describe the design of the LuminUs -a device that provides musicians with real-time visual feedback on either the gaze or body motion of their co-performers. We also report preliminary results from a study that investigated the impact that the LuminUs has on dyadic improvised performances.

The LuminUs

We designed the LuminUs with the aim of exploring how technology might be used to provide performing musicians with an increased awareness of each other, especially in situations where their mutual attention might be hindered by complex musical interfaces and physical obstructions. The LuminUs consists of a strip of 16 coloured LEDs that can be controlled individually to provide dynamic visual information (see Fig. 1(a)). The device is mounted on a flexible arm so that it can be positioned in front of the musician. Control of the lights is provided via USB connection to a computer. The design was chosen based upon three criteria for the way in which feedback should be presented to the musician: visual (as opposed to haptic or audio); minimal (so as not to be overly distracting); and dynamic (to represent time varying signals). The LuminUs has two modes of operation -motion feedback and gaze feedback. In the motion feedback mode we measure the body movements of each musician using small wireless accelerometers worn around the waist. These movements are then processed and displayed on the LuminUs. The greater the movement, the more lights are illuminated -with the colours of the lights ranging from green (low) to red (high). In the gaze feedback mode we use eye-tracking glasses to detect when one musician is glancing towards the other. The LuminUs then visually notifies the musician who is being glanced at. In this case, more lights become illuminated as the duration of the glance increases. For gaze feedback the colours of the lights range from blue (short) to purple (long).

For eye-tracking we used the Pupil headset, which is an open-source hardware and software project [START_REF] Kassner | Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction[END_REF]. The headset tracks the movement of the right eye using a single infra red camera, and simultaneously captures the wearer's pointof-view (POV) using a separate forward-facing video camera. Following a short calibration (∼20 sec), the software is able to map the gaze point of the wearer onto the live POV video. We modified the pupil software so that it could recognise 2D markers within the POV images. Consequently, by placing markers on the headsets of each musician (see Fig. 1(b)), we are able to automatically register the moments when one person is glancing at the other.

The Experiment

Our hypotheses concerning the way in which the LuminUs would influence musical interactions were as follows:

H1 Providing gaze feedback would increase the overall amount of gaze during collaborative interaction. H2 Providing motion feedback would stimulate increased awareness of the other participant, as indicated through more glancing at the other. H3 Providing motion feedback would increase the overall amount of motion during collaborative interaction.

To test these hypotheses we designed an experiment in which pairs of pianists and percussionists were asked to create live improvised accompaniments to a silent animation. This task was chosen for a number of reasons. Firstly, the improvisational nature of the task encouraged the musicians to work collaboratively and heightened the need for mutual awareness. Secondly, the animation meant that the musicians had more to attend to than simply their instrument and coperformer -as would often be the case in real-world performances. In addition to this, it provided a visual stimulus to guide and influence the improvisations created by each pair. Four two minute long animations were short-listed for use in the study. We then asked a separate group of 24 musicians to watch and rate the animations based upon their suitability for improvised musical accompaniment. The animation that received the highest average rating was chosen for use in the study.

Method

15 percussionists and 15 pianists were recruited from music colleges in London. The participants comprised 7 females and 23 males aged between 18 and 38 (M = 22.7, SD = 4.7). Their playing experience ranged from 3 to 33 years (M = 11.6, SD = 5.6). The participants were assigned to percussionist-pianist pairs such that the individuals in each pair did not know one another.

There were seven experimental conditions, coded as: G-G, G-X, X-G, M-M, M-X, X-M, and X-X. The first letter represents the feedback that the LuminUs provided to the percussionist, whilst the second letter represents the feedback provided to the pianist -either gaze feedback (G), motion feedback (M), or no feedback (X).

The experiment was held in a performance space, with overhead stage lighting and black curtains surrounding the performance area. The setup consisted of a large screen, which showed the animation and provided instructions to the participants. The two participants were then positioned facing the screen, but angled slightly towards one another. The percussionist was provided with two electronic drum pads (snare and floor tom) and an electronic ride cymbal, which they played standing up. The pianist was provided with a 61 note keyboard and sustain pedal, which they also played standing up. Both instruments were connected to a computer via MIDI, and the audio was output through speakers positioned either side of the screen. Each participant's LuminUs was positioned on a stand in front of them, such that it was just below their line of sight to the screen. The devices were positioned such that each participant could only see their own device, and the brightness of the lights was adjusted so that no reflections were visible. Figure 1(b) shows the experimental setup as seen from below the screen.

Upon arriving for the experiment the participants were given a couple of minutes to play the instruments together. This gave them a chance to make brief introductions and familiarise themselves with the instrumental set up. The participants were then fitted with the accelerometers and eye tracking headsets, which were subsequently calibrated for each participant. All of the eye-tracking and motion data was time-stamped and saved. The experiment began with a warm up session, where participants watched the animation twice without playing their instruments and then twice whilst playing along. Following this, the participants were given roughly one minute to discuss ideas. This was the last opportunity for them to verbally interact during the experiment. The remainder of the experiment consisted of seven improvisation sessions, each randomly assigned to one of the seven conditions discussed above. Prior to each session each participant's LuminUs would show either green-red (motion feedback), blue-purple (gaze feedback), or no lights (no feedback) to indicate which type of feedback they would receive in that session. This was done so that neither participant would know what kind of feedback the other was receiving. In each session the participants had two attempts to play along to the animation, with a 10 second gap between. After each session the participants completed a short questionnaire to rate aspects of their experience and subjective opinions relating to that particular session. The analysis of our questionnaire results is out of the scope of this paper.

Results and Analyses

To test H1 and H2 we decided to look at the average number of glances that each participant made towards their co-performer within each condition. This information was extracted from our eye-tracking data. Figure 2(a) shows the mean and standard error for the number of glances averaged over participants within each condition. We see that the mean values for both the pianist and percussionist are lowest in the conditions where they are not receiving any feedback from the LuminUs. We can also see that, on average, the pianist tended to glance more than the percussionist.

To test H3 we extracted the mean body motion for each participant within each condition. The results are shown in Fig. 2(b). We can see that the percussionist moved more than the pianist, as might be expected. More interestingly, it appears that the body motion values tend to be lower when the participants are receiving motion feedback, compared to gaze feedback.

We used the Wilcoxon signed rank test to perform statistical analyses of some of the relationships in figures 2(a) and 2(b). In particular, we looked at the differences between the mean values obtained in the 6 conditions where the LuminUs was active, relative to the equivalent means for the inactive LuminUs condition (X -X). The results are shown in Table 1. The results for the mean number of glances indicate that there were significant differences between all of the conditions and the X -X condition. This effect is greatest for condition 1, where both participants had gaze feedback (r = 0.7, p = .000). The second strongest effect is for condition 2, where only the percussionist had gaze feedback (r = 0.57, p = .002). The third strongest is for condition 4, where both participants had motion feedback (r = 0.52, p = .005).

For the body motion results we see that none of the differences are significant. However, of potential interest is the fact that a mild negative effect is observed for condition 4 (r = -0.29, p = .106), where both participants had motion feedback. In addition to this, the only other weak negative effects correspond to the other conditions involving motion feedback. For the conditions involving gaze feedback (1-3) the effect sizes are small (< 0.2).

Condition X -X X -M M -X M -M X -G G -X G -G Mean Number of Glances 30 20 10 0 (a) Condition X -X X -M M -X M -M X -G G -X G -G

Mean Body Motion

Discussion

H1 and H2 are supported by our findings, which indicate that when participants were receiving feedback from the LuminUs they glanced towards each other significantly more than when no feedback was provided. The size of this effect was greatest for the condition where both participants had gaze feedback. However, it is possible that the main effects were simply due to the LuminUs serving as a reminder of the other musician's presence. We intend to carry out further analyses to investigate the nature of these effects in more detail. In particular, Table 1: Statistical results for the differences between means in conditions 1-6 (LuminUs in use), relative to those in condition 7 (LuminUs not in use). we will look at how the specific timings of glances relate to the light output of the LuminUs. This will allow us to see whether musicians responded directly to the feedback, or whether it had a more general influence on their behaviour during the sessions. Regarding H3, our results were less conclusive, but appeared to indicate that motion feedback led to decreased body motion. We envisaged that providing feedback on the motion of a co-performer would, if anything, encourage participants to move more. This hypothesis was partly influenced by studies on behavioural mimicry, which suggest that the actions or emotions displayed by one person can cause congruent behaviour in another person [START_REF] Chartrand | The Antecedents and Consequences of Human Behavioral Mimicry[END_REF]. The fact that an opposite trend was observed is a potentially interesting finding. Unfortunately there is an absence of existing research on the behavioural effects of providing real-time motion feedback, so at this stage we can only speculate as to any causality. A possible explanation is that the musicians felt self-concious when they knew their movements were being displayed on the LuminUs. However, if this was the case then we might expect a similar effect for the gaze feedback. Furthermore, none of the musicians indicated feelings of self-conciousness when we informally interviewed them after the experiments.

The results presented in this paper indicate that the LuminUs had an influence upon the objective aspects of the musicians' behaviours. However, at this stage it is not possible for us to say whether these were detrimental or beneficial to the musical interaction as a whole. Our ongoing analyses will investigate whether the use of the LuminUs influenced the subjective experiences of the musicians; the quality of their coordination; and the musical outcomes of their performances. In order to investigate this we will use the self report data collected from the questionnaires alongside the audio and video recordings of the performances. An additional consideration is that the short duration of the study may not have provided the musicians with sufficient opportunity to familiarise themselves with the LuminUs and make effective use of its feedback. In future work it would be interesting to investigate how musicians appropriate the LuminUs over a more sustained period of usage.

Conclusion

This paper provides a brief introduction to the LuminUs, and some preliminary results from experiments, where we put it to the test with trained musicians. Our initial findings suggest that providing real-time co-performer gaze and motion feedback can have a measurable impact on aspects of non-verbal communication and behaviour during collaborative musical performances. Our continued analyses will attempt to ascertain whether or not the LuminUs also influenced the quality of the interactions between musicians and the accompaniments they created. Furthermore, through detailed analysis of our eye-tracking, motion, and video data, we hope to gain a better understanding of how musicians use nonverbal communication during live performances.

Fig. 1 :

 1 Fig. 1: (a) The LuminUs: different colours were used for gaze (1) and motion (2) feedback; (b) Annotated image of the experimental setup.

Fig. 2 :

 2 Fig. 2: Bar graphs showing (a) the mean number of glances; and (b) the mean motion within each of the seven conditions. G = gaze feedback, M = motion feedback, X = no feedback. Error bars: ± 1 SE.

Acknowledgements This research is supported by the Media and Arts Technology Programme, an RCUK Doctoral Training Centre in the Digital Economy.