Kristina Groth

Jonas Moll

Alex Olwal

Oscar Frykholm
email: frykholm@csc.kth.se

Design and Evaluation of Interaction Technology for Medical Team Meetings

Keywords: Medical team meetings, collaboration, single-display groupware, multi-display groupware, multi-touch, pen, mobile

de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Multi-disciplinary teams in modern healthcare are facilitating the discussion of patients, decisions on diagnosis and treatment, operation planning, interventions and surgery [START_REF] Groth | Efficiency in Treatment Discussions: A Field Study of Time Related Aspects in Multi-Disciplinary Team Meetings[END_REF][START_REF] Kane | Multidisciplinary Medical Team Meetings: An Analysis of Collaborative Working with Special Attention to Timing and Teleconferencing[END_REF][START_REF] Kane | Multidisciplinary team meetings and their impact on workflow in radiology and pathology departments[END_REF][START_REF] Måseide | The deep play of medicine: Discursive and collaborative processing of evidence in medical problem solving[END_REF][START_REF] Lee | Consulting with Radiologists outside the Hospital by Using Java[END_REF][START_REF] Ruhstaller | The multidisciplinary meeting: An indispensable aid to communication between different specialities[END_REF][START_REF] Groth | The role of technology in video-mediated consensus meetings[END_REF]. The medical specialists participating in the meetings are not always located in the same facilities, or even city, whereby tele-or videoconferencing is frequently used. During the meetings, specialists present relevant information about patient cases, based on their domain-specific expertise. Current meeting facilities typically only allow image presentation and demonstration by the radiologist and/or pathologist, whereas the other meeting participants have no means for interacting with the material [START_REF] Kane | Multidisciplinary Medical Team Meetings: An Analysis of Collaborative Working with Special Attention to Timing and Teleconferencing[END_REF][START_REF] Kane | Multidisciplinary team meetings and their impact on workflow in radiology and pathology departments[END_REF][START_REF] Li | Supporting Enhanced Collaboration in Distributed Multidisciplinary Care Team Meetings[END_REF][START_REF] Groth | Efficiency in Treatment Discussions: A Field Study of Time Related Aspects in Multi-Disciplinary Team Meetings[END_REF]. To support the changing style of collaboration, new, effective user interfaces are needed.

We have worked closely with a surgical department at a university hospital, which focuses on complicated deceases in the upper abdomen, where multi-disciplinary team meetings (MDTMs) are a central part of patient care. In this context, we are interested in applying and evaluating technology that can improve understanding, simplify group discussion and increase collaboration for safer and faster medical decisions [START_REF] Groth | Efficiency in Treatment Discussions: A Field Study of Time Related Aspects in Multi-Disciplinary Team Meetings[END_REF]. We have therefore developed a set of tools and techniques, focusing on multi-display groupware (MDG), as shown in Figure 1. In the spirit of previous work [START_REF] Bier | MMM: a user interface architecture for shared editors on a single screen[END_REF][START_REF] Myers | Collaboration using multiple PDAs connected to a PC[END_REF][START_REF] Paek | Toward universal mobile interaction for shared displays[END_REF][START_REF] Rekimoto | A multiple device approach for supporting whiteboard-based interactions[END_REF][START_REF] Stewart | Single display groupware: a model for copresent collaboration[END_REF], our goal has been to distribute the interactive capabilities at such meetings among participants. While our work is applied to the medical domain, we also find it generally applicable to other types of MDG and single-display groupware (SDG) scenarios [START_REF] Stewart | Single display groupware: a model for copresent collaboration[END_REF][START_REF] Myers | Collaboration using multiple PDAs connected to a PC[END_REF][START_REF] Wallace | Investigating teamwork and taskwork in single-and multi-display groupware systems[END_REF]. To ensure qualitative feedback without influence from peers and to minimize the effect of time constraints, this study explores the collaborative techniques with one participant at a time in a simulated team meeting. This first evaluation of tools for facilitated collaboration in MDTMs will form the basis for a large-scale user testing in the real setting.

We first review related work, followed by an overview of the medical team meetings that we are studying and a discussion of a number of initial studies that we conducted, which form the basis for this work. We then present our method, a set of design guidelines and the implemented techniques. The main part focuses on our user study and results, followed by a Discussion, Conclusions and Future Work.

Background

Several researchers report on detailed field studies of MDTMs [e.g., [START_REF] Groth | Efficiency in Treatment Discussions: A Field Study of Time Related Aspects in Multi-Disciplinary Team Meetings[END_REF][START_REF] Groth | The role of technology in video-mediated consensus meetings[END_REF][START_REF] Kane | Multidisciplinary Medical Team Meetings: An Analysis of Collaborative Working with Special Attention to Timing and Teleconferencing[END_REF][START_REF] Kane | Achieving Diagnoses by Consensus[END_REF][START_REF] Kane | Multidisciplinary team meetings and their impact on workflow in radiology and pathology departments[END_REF][START_REF] Li | Supporting Enhanced Collaboration in Distributed Multidisciplinary Care Team Meetings[END_REF]. Time is an important aspect when judging the efficiency of the meeting [START_REF] Kane | Multidisciplinary Medical Team Meetings: An Analysis of Collaborative Working with Special Attention to Timing and Teleconferencing[END_REF][START_REF] Groth | Efficiency in Treatment Discussions: A Field Study of Time Related Aspects in Multi-Disciplinary Team Meetings[END_REF] and supporting technology must take this into account, as well as specific participant roles [START_REF] Groth | The role of technology in video-mediated consensus meetings[END_REF]. Although some problems regarding efficiency and effects on the group work process were likely to arise, Kane and Luz [START_REF] Kane | Multidisciplinary Medical Team Meetings: An Analysis of Collaborative Working with Special Attention to Timing and Teleconferencing[END_REF] acknowledge the need for technology for collaborative decision-making. Unsupported needs include the ability to point to areas of interest and to make annotations on shared displays [START_REF] Kane | Achieving Diagnoses by Consensus[END_REF]. Laser pointers were shown to partially alleviate these deficiencies, although it was noted that the laser dot could not be seen at remote locations during teleconferencing meetings [START_REF] Kane | Multidisciplinary Medical Team Meetings: An Analysis of Collaborative Working with Special Attention to Timing and Teleconferencing[END_REF].

Several projects have explored digital tools for multi-user interaction on single or multiple displays. The Pebbles Project [START_REF] Myers | Collaboration using multiple PDAs connected to a PC[END_REF][START_REF] Myers | Using Hand-Held Devices and PCs Together[END_REF] is a collaboration framework for multiple handheld computers that are connected to a shared display, providing remote keyboard and mouse control, and multi-user drawing with individual cursors. M-Pad [START_REF] Rekimoto | A multiple device approach for supporting whiteboard-based interactions[END_REF] uses tracked handhelds for advanced multi-user interaction techniques with whiteboards through toolglasses, palettes, cross-device transfer, and annotations. While these systems were pioneering, they had no wireless connectivity and their graphics performance limited possibilities for manipulating medical images.

Architectures for mobile interaction with shared displays [START_REF] Paek | Toward universal mobile interaction for shared displays[END_REF] and multi-cursor interaction [START_REF] Bier | MMM: a user interface architecture for shared editors on a single screen[END_REF] illustrate the complexity when designing multi-display groupware environments. Tse et al. [START_REF] Tse | Avoiding interference: how people use spatial separation and partitioning in SDG workspaces[END_REF] describe a set of experiments where shared workspaces were partitioned by participants into individual areas to address interference, while one of the motivations for designing cooperative gestures [START_REF] Morris | Cooperative gestures: multi-user gestural interactions for co-located groupware[END_REF] was to clarify users' intentions. Wallace et al. [START_REF] Wallace | Investigating teamwork and taskwork in single-and multi-display groupware systems[END_REF] discuss the results from a study comparing SDGs (multiple input devices, single display) with MDGs (multiple input devices, individual displays). They conclude that SDGs benefit in awareness of collaborators' activities, but may be distracting and can interfere with the group's primary tasks. MDGs, on the other hand, may provide a private, customizable, workspace with fewer distractions, but can instead require more efforts for coordinating team work.

Lee et al. [START_REF] Lee | Consulting with Radiologists outside the Hospital by Using Java[END_REF] and Eng et al. [START_REF] Eng | Collaboration System for Radiology Workstations[END_REF] describe remote consultations where radiologists and physicians use a synchronized image viewer. The systems enable shared reviewing with joint image navigation, synchronized cursors for pointing, and image annotations. Shareable displays [START_REF] Osman Ratib | Use of Personal Digital Assistants for Retrieval of Medical Images and Data on High-Resolution Flat Panel Displays[END_REF] have also been developed using PDAs to access and navigate radiology material on larger displays. Recent advances in mobile technology, allow us to overcome the limitations of previous systems [START_REF] Flanders | Handheld Computers in Radiology[END_REF] to support distributed interaction with radiology imagery.

Our paper builds on a body of work, in which we have studied a large number of MDTMs at a gastro-surgical department over four years. Results from these studies indicate many benefits if also participants in these meetings could navigate the material, point to regions of interest, and annotate important features.

MDTMs at Gastro

The centralization of specialized medical care to regional university hospitals is a recent strategy for quality improvement and effectiveness. The gastro-surgical department (Gastro) that we have studied has the regional responsibility for specialised care in the upper part of the abdomen (i.e., complex diseases in the liver, pancreas and esophagus). The 25 surgeons collaborate closely with several other units within the university hospital organisation, including pathology, radiology, oncology, and hepatology. The weekly MDTMs (See Figure 2) are an important element of the work process at Gastro.

The patient care pathway at Gastro [START_REF] Frykholm | Medicine Meets Engineering in Cooperative Design of Collaborative Decision-supportive System[END_REF] consists of four main steps: coordination of medical material, decision on diagnosis and treatment, surgical treatment, and postoperative treatment. Gastro regularly uses three types of MDTMs during the patient care pathway.

The Decision Meeting is a forum for discussion of patient diagnosis, operability and resectability, to decide on the next steps in the care, such as surgery, radiology, oncology treatment, or the need for additional examinations. There are a number of key roles at the meeting. First, a surgeon summarizes the patient's medical history and formalises the discussion topic. Second, a radiologist, seated at a workstation with three high-resolution displays, presents the radiological analysis, while participants follow the walk-through of the medical imagery on two projection screens. Third, if there are pathology reports, then these are presented by a pathologist. Fourth, a senior surgeon leads the following discussion, which relies on the display and navigation of medical imagery, to reach a consensus decision. A group of medical specialists from different disciplines participate with comments, questions and discussions. Less experienced medical doctors attend as part of their training, but do not actively participate in the discussions. The meetings must be streamlined and efficient due to a high number of patients to discuss by the many attendees under time constraints.

Pre-operative Meetings (Pre-ops) are similar to Decision Meetings, but have fewer participants, are more focused on operation strategy and planning and have no time constraints. They are typically attended by three to six surgeons, one radiologist and occasionally one pathologist, all from the same hospital, and take place the week before surgery. The video link is not used, but specialists from other departments can participate if needed.

Post-operative Meetings (Post-ops) allow pre-op attendees to share their experiences after surgery to, for example, discuss how well the pre-op planning matched the surgical procedures. These are only conducted for pancreas cases.

Initial Studies at Gastro

Our work is based on four years of field studies where data has been collected in an on-going process of ethnographic fieldwork and participatory design projects. Our early MDTM observations have showed a need, but limited possibilities, for interaction with patient information. To better understand the interaction with radiology imagery during MDTMs, we observed and analysed twelve additional Decision Meetings and Pre-ops during 2009. The meetings were video recorded and the sections where surgeons specifically tried to point to and navigate in the images were analysed in detail. Similar to related work [START_REF] Måseide | The deep play of medicine: Discursive and collaborative processing of evidence in medical problem solving[END_REF], it was observed that the radiologist relies heavily on mouse pointing when presenting and explaining the medical imagery. The analysis of the observations show that also the surgeons point to parts in the medical imagery (from their seats or by approaching the screen) to illustrate how operations should be performed or when asking for clarifications. This indicates a need to illustrate a procedure or make a deictic reference to a detail in the imagery. It was also clear that the radiologist, surgeons and other specialists have a formalized language with an established common ground that make it easier to verbally refer to parts in the imagery during their discussions. We also observed occasions where some participants used laser pointers to clarify their intentions.

To further understand the interaction during MDTMs, a follow-up field study of three consecutive Pre-ops (each lasting ~40 min) was conducted in which laser pointers were handed out to all participants, except the radiologist. The objective was to observe how laser pointer usage would affect meeting discussions. Participants were not given specific instructions on how or when to use the laser pointers. In the analysis, we found that the laser pointers were used:

1) To point and "draw" in different parts of the projected imagery 2) For joint navigation and guidance 3) To help coordinate communication by pointing to smaller features while asking for more detailed information 4) For gestures that allowed surgeons to illustrate plans, e.g., for cuts of organs The setup's major drawback was that the radiologist had to direct his attention from the workstation to see the laser dot on the projection screens, preventing simultaneous control of the workstation and maintained awareness of the surgeons' pointing.

The advantage of laser pointers, from a technological standpoint, is low cost, portability and infrastructural independence. Still, despite the benefits and immediate availability, laser pointers have not been widely adopted by participants at the MDTMs. From observations it became clear that laser pointers do not meet the requirements of multi-display meetings, where radiologists, surgeons, and remote participants are viewing three separate, but digitally synchronized displays. A surgeon's laser pointer on the projection will thus not be visible on the radiologist's workstation or through the videoconferencing system, which turns out to be prohibitive for practical use. Also, the laser pointer study showed that the person pointing is using words like "here" instead of verbally explaining a part in the image, which is making the referencing even more ambiguous for participants viewing other displays. While it would be possible to track and distribute laser dots as digital cursors using cameras [START_REF] Olsen | Laser pointer interaction[END_REF], the required infrastructural modifications inspired us to instead develop techniques that would support more advanced and flexible multi-user interaction through digital tools, without the need to install any hardware in the MDTM room.

Prototypes for Collaborative Interaction Techniques

The objective of the study was to explore the functionality that new interaction techniques and devices can enable for MDTMs, and evaluate their potential for improving understanding, group discussion, collaboration and decision making. Data from our initial studies allowed us to make informed design decisions for the candidate interaction technology to be used in the study. We first decided to prioritize the most important subset of medical imaging functionality from the radiologist's workstation, based on the observed needs of the meeting participants in our studies: 1) Shared/collaborative pointing 2) Navigation (zooming/panning/scrolling in image stack) 3) Annotation (sketching/drawing) We interviewed and observed two senior radiologists at their hospital workstations (each ~2 h) to gain insight into workflow, important functionality, frequent operations and desired features. We then conducted an interactive workshop with a senior pancreas surgeon (~3 h) where we discussed the use of digital devices for interacting with the radiology images and related these ideas to typical MDTM discussions. A first version of the prototypes was developed after these sessions. The prototypes were demonstrated and discussed with the surgeon at a second workshop (also ~3 h), which provided additional feedback. Two interactive prototyping sessions (each ~2 h) were then conducted individually with one of the previously observed radiologists and one new radiologist. Similarly to the second workshop with the surgeon, they individually explored the first version of the prototype interfaces, but also answered questions regarding current practices/problems, potential for the proposed techniques and suggested functionality and features (See Figure 3). Audio and video was recorded for the sessions with the surgeons, and informal notes were taken during the radiologist sessions.

Insights from these sessions supported our preliminary ideas, inspired topics for the user study questionnaire and indicated an interesting potential for collaborative interaction techniques in MDTMs.

Design Guidelines

A set of design guidelines for the tools that we were developing for Pre-ops was identified from our initial studies and prototyping sessions: Minimal changes to current meeting situation: The physicians are in general satisfied with routines and configuration of current meetings, and that the use of technology is limited to the radiology workstation. We thus wanted our tools to augment the meetings with minimal modifications to current workflow.

Sporadic interaction with minimal learning time:

The users should be able to quickly access the needed functionality, as the dynamic meetings cannot afford complex and time-demanding user interfaces.

Arbitrary number of participants: The system should be able to support interaction from any user through an inherently scalable infrastructure.

Radiologist delegates control: As the radiologist is an expert of interpreting and navigating the medical imagery, and is in control of these processes today, it seems important to maintain that role. While many surgeons will likely want to interact with the imagery, this is not their area of expertise, and may not necessarily be effective. The interviews indicate that distributed interaction could be adopted as long as the radiologist could delegate control when needed.

Synchronization across displays: Current methods (e.g., finger pointing, walking up to the projection screens and the use of laser pointers) exclude remote participants and the radiologist. Digital tools, on the other hand, make it possible to synchronize interaction across multiple displays, which could especially be used to empower remote participants.

Interaction Techniques and Implementation

The design guidelines informed the choice of the technology and architecture for the system. Thus, our system, for example, supports a radiologist's delegation of control, remote participation, and many other features. It is based on a PC server that projects medical imagery, while connected mobile devices are synchronized using the network, such that multiple users can interact using their own device.

Software architecture: Our framework is implemented using C++ and openFrameworks, which allows us to support most of the implemented functionality across multiple platforms (Linux, Mac OS X, Microsoft Windows XP-7 and Apple iOS). The mobile devices are synchronized with the server using OSC (OpenSoundControl), a UDP-based protocol for efficient, platform-independent data streaming. Pen-based devices: We implemented our techniques on Wacom tablets (no display) and Tablet PCs (Dell XT2) that use electromagnetic pens with high precision/pressure and physical buttons, as shown in Figure 4.

Multi-touch devices: We use Apple iOS devices (3.5" iPod Touch 3 rd Generation and 9.7" iPad) as portable multi-touch displays (See Figure 5), and support larger screens through the multi-touch capabilities in Windows 7 (e.g., 21.5" Dell SX2210T, 22" 3M M2256PW, and 12" Dell XT2).

All device types were explored in the prototyping sessions with the surgeon and radiologists, and based on our observations and their feedback we chose to narrow it down to three devices for the user study: the screen-less tablet, and the small and large multi-touch devices (iPod Touch and iPad). We carefully selected the techniques through our initial prototyping and user observations, based on observed gestures at MDTMs and other studies. The hardware was chosen based on commercial availability, mobility, the possibility to use it outside MDTMs, and the different form factors. The pen-based device was selected as the lack of display avoids problem with private focus.

We developed three general interaction techniques for multi-user interaction.

Pointing: One of the primary motivations of the system is to enable participants to point accurately from any location in the room, or over the video conferencing link. It is achieved by touching and dragging with a single finger on the touch-screen devices, or by hovering the pen over the stylus-driven tablet. A coloured cursor distinguishes different users.

Navigation: A central component of the radiology material is composed of CT (computed tomography) or MRI (magnetic resonance imaging) image stacks, where the 3D volume of the human body and organs are viewed one 2D slice at a time. Here, zooming, panning and scrolling up/down in the stack are the most important means for navigation. Pan is controlled by dragging two fingers in the multi-touch interfaces, and zoom level is adjusted with two-finger pinching. A widget on the right-hand side visualizes the stack, with absolute access to each slice, and two buttons allow stepping up/down one slice at a time. On the pen-based device, the user pans by pressing a side button on the pen and hovering it over the tablet surface. Stack scrolling is controlled by sliding with a finger of the non-dominant hand on the tablet's touch-sensitive strip, while the simultaneous pressing of a device button zooms instead.

Annotation: Annotation with individual colours on each slice was implemented to support the exploration of temporary or permanent freehand annotations during the meetings. The multi-touch interface uses an on-screen button to toggle between pointing and drawing, whereas another button allows the user to undo the last stroke for the current slice. Dragging the pen on the tablet surface creates annotations that the user can undo using a dedicated button.

User Study Setup

We designed a formal qualitative user study to evaluate our interfaces and explore our target group's attitudes towards using digital tools for distributed pointing, navigation and annotation during MDTMs. The respondents explored pen-based interaction (Wacom tablet) and multi-touch interaction on a small and a large portable display (iPod Touch and iPad) in a simulated Pre-op setup where interaction was synchronized with a large display (see Figure 6). The user study was conducted with one physician at a time, with the study leader acting as radiologist, in control of the large display (which represented the projected screens used in the MDTMs). Two authentic patient cases (a liver and a pancreas case) were prepared in advance, including radiology images and text from the patient record (i.e., the radiology documentation from the Pre-op).

We report on the results from individual sessions with eleven surgeons (one female) and one senior radiologist from the hospital. Seven of the surgeons were senior gastro surgeons specialised on a specific organ or intervention technique, four were consultants, four were gastro surgeons, one was a resident and one was a visiting gastro surgeon. The four consultants would typically lead the discussions, and the surgeons or the resident would present the case at the MDTMs. The radiologist was asked to respond to the study as if he was attending an MDTM where another radiologist demonstrated the material. The choice of focusing on surgeons is based on our field data showing that they are usually the ones asking more specific questions about the demonstrated imagery. We will broaden our studies to include more disciplines in the future. While the respondents lack experience in using ICT at these meetings, and access to technology is usually limited to PCs and medical devices, most of them were interested in exploring new technology that could support their work. Most of them had used or were familiar with mobile touch-screen devices, knew about iPads but had not used one or seen one in real life, and were not familiar with Wacom Tablets. The oldest respondent was known, among the surgeons, to be conservative about new technology. The respondents were 35-62 years old (avg. 42 years) and received no compensation. The sessions lasted 18-52 min (avg. 36 min).

The study's objective was to identify and assess the potential of the implemented interfaces and technologies, and the type of functionality that would be useful at MDTMs. Questions from the study leader accompanied a pre-test questionnaire to collect initial attitudes. The respondents then explored the three interfaces (in randomized order) using think-aloud, combined with interview questions during the test, and a post-test questionnaire, where the usefulness of navigation, pointing and annotation in the different meeting situations, were respectively ranked on six-point Likert scales. Questionnaire data was not analysed statistically due to the number of respondents.

The respondents were asked to primarily consider the technology for Pre-op meetings, but the concluding discussion also concerned its applicability in other meetings or situations. The respondents were encouraged to comment and discuss with the study leader throughout the whole session. Sessions were recorded on video, transcribed and analysed in detail by categorising the data based on the current situation, potential risks, situations in which the devices could be used, and feedback on the interaction techniques and the input devices.

While individual sessions cannot fully simulate the multi-user interaction at the MDTMs, they were intended to allow us to gather focused feedback from each respondent without time constraints. The most senior participants at the MDTMs tend to dominate discussions given the formal experience-based hierarchies that dictate meeting roles. We thus balance our aim for a focused session and qualitative feedback from individuals with varying expertise and background, by having the interviewer take the role of the radiologist (controlling imagery from the workstation) with the participant simultaneously interacting using different devices. All respondents are well familiar with the MDTMs, and all senior surgeons and the radiologist attend every week. Two of the respondents also participated in the earlier study with laser pointers.

Results

The main target of the interaction in today's MDTMs is the radiology presentation and images. The radiologist is in control of the presentation and the other participants have to rely on other means of interacting with the images, for example when clarifications or more details are needed. In the pre-test discussion and questionnaire, almost all respondents emphasized the need for interaction with the images, especially pointing: "Some colleagues have laser pointers ... They use them to point ... Several approach and point in the projection" (resident F). However, some respondents stressed that Pre-ops are more suitable for interaction than Decision Meetings due to the large number of cases presented there.

Interaction Techniques: Pointing, Navigation and Annotation

Of the three interaction techniques, the respondents were most excited about pointing, which could help resolve ambiguous references, avoid misunderstanding, and improve communication for local and remote users: "Pointing at a detail … is very important" (senior surgeon U). Four physicians said that they today rely on verbal explanation using common terminology (e.g., the eight liver segments can be referenced by index), point with their finger towards the projected imagery, or walk up to the projection screens to point. Some respondents found this acceptable and not a problem that they had reflected on as they considered verbal references easy to understand for the radiologist: "You just say back a little … scroll some" (senior surgeon B) and "I say that I want to see the blood vessel there" (senior surgeon A). The ineffectiveness of verbal references was pointed out by several respondents who found it time-consuming to explain to the radiologist which area they would like them to navigate to. They mentioned that it was an abstract way of pointing and could cause misunderstandings due to the difficulty of verbally directing the radiologist, and that participants may thus hesitate to ask for clarifications.

Precise and unambiguous pointing would also make the discussion easier to follow for surgeons in training. Senior physicians tend to have developed a more precise language and can therefore refer more accurately to specific parts of the images, while it is typically more difficult for junior surgeons to interpret and follow the discussion: "The experienced colleagues can use a completely different language, they are more precise than I can handle … often they point, sort of sweeping … but it's not really obvious what they point to" (resident F). The possibility for clearer pointing would, according to the resident, make it easier to follow and learn from the discussion. Thus, although experienced surgeons may be able to give accurate descriptions of areas of interest without pointing devices, their use of such devices might still be a great aid to less experienced surgeons.

Feedback on navigation primarily focused on image stack scrolling, while zooming and panning was emphasized for usability on the small display. Many respondents were not convinced about the need for navigation during MDTMs as they considered it the radiologist's job: "The radiologist presents the material very thoroughly and can see much better than I" (surgeon C).

Annotations seemed less important for the Decision Meeting discussions, but numerous comments indicate potential benefit in other situations. As for precise pointing, referencing would be clearer for, e.g., residents and junior surgeons, when lesions and relations to arteries can be highlighted. Basic functionality, which radiologists have access to today, such as marking and measurement tools, were frequently asked for by several respondents. Also, one surgeon commented that drawn annotations could be useful in specific cases that need more detailed discussion.

Some surgeons emphasized the ability to save annotations: "There is no point in marking temporarily -you'd want to save and continue later" (surgeon T) and "You'd mark the area that is unclear after the radiology presentation. The tumour often grows into a blood vessel … you could actually mark it and refer to it here [while using the iPod]" (surgeon S). This would help the surgeons understand patient cases faster if they could not attend the Pre-op, or wanted to refresh their memory in preparation for an MDTM or a surgical procedure. Simple symbols like arrows, labels and numbers could complement the documentation, but also be useful for lectures and educational material. For a better overview of the annotations in radiology examinations (where there are multiple annotations in the image stack), an index of annotations, highlighting of annotated slices, and linked bookmarks to annotations were suggested.

Private Interaction vs. Shared Focus

Several respondents were excited to interact on their own high-quality display with easy-to-use controls. The large multi-touch device was appreciated for its large and clear display. Some respondents disliked the smaller device, while some thought that it was sufficient in size: "I can see well on this one too, the zoom is so powerful" (surgeon J). Some respondents expressed interest in exploring the imagery on their own and requesting control upon finding something they wanted to share: "You could work maybe a little in parallel, not having to disturb everyone" (senior surgeon B) and "The radiologist is presenting,

and you have your iPhone to see what he is presenting … then you see something important, and you can annotate it on your own

[device]" (surgeon T). Still, it was generally considered that the radiologist should be in control of image manipulation on the shared display. If there was something respondents wanted to ask about or bring the group's attention to, they could navigate and annotate on their personal device and ask the radiologist for permission to show it on the shared display.

Several respondents were, however, concerned that users would only focus on their private screen -a disadvantage compared to the screen-less tablet: "The objective of the meeting is to have shared discussions … if everyone is looking at their own [screen], then it isn't a meeting" (surgeon J), and "It would be bad for the group discussion [if you had your own]" (senior surgeon M). Also the radiologist commented on this: "I may loose their attention when I demonstrate details … they might not even hear what I say -they would be into their own analysis … it is better that everybody has the same focus".

Applicability for Different Scenarios

Many respondents pointed out that the evaluated tools would be much more valuable at Pre-ops as the discussions in these meetings are more focused, the groups are smaller, there are fewer patients to discuss and the meetings are not under time pressure, in contrast to Decision Meetings. One surgeon said: "There cannot be anything that disturbs the radiologist [

at the Decision Meetings]. If you loose five minutes per patient, and have 19 patients … That doesn't work" (surgeon T)

. Postops, on the other hand, are both less common (pancreas cases only) and less critical, as no decisions are taken. The respondents also commented on making annotations before the meeting, in order to save time during the meeting: "[It] can be good to mark which structure is which -this is actually truncus hiliacus … and that can be done in advance … it is also how much time it [annotating the images] will take from the meeting" (surgeon J).

The benefits of clarity and accessibility were, in particular, highlighted by the less experienced surgeons. The resident said that less experienced surgeons would be able to follow the discussion better and would not need to disturb the more experienced surgeons if they can not follow what is referred to.

During the user studies, respondents also came to think of situations during the entire patient care pathway where the devices could be used. In team-based care, where multiple physicians share the responsibility for patients, it is important to be able to hand over patient information such that another physician can easily catch up on a case and quickly be able to make decisions. Decision Meeting protocols should accompany the imagery and examinations on to the Pre-op, where the surgical plan is annotated on the images. The plan could then be used when preparing the surgical procedure, especially if the surgeon in charge of the operation was not present at the Pre-op: "The image information is very important … today it [the surgical planning] is based on the Pre-op and that the surgeon is present to see and get a mental image for making a strategy to remember" (surgeon J). The plan could be shown both to the patient and to the team in the operating room, and in the end ideally all the material could be accessible for presentation at a Post-op or follow-up meeting: "It would be elegant if s/he [the surgeon] could display the images: 'This is what we planned', 'This is what we did ... but when we physically got there … this structure was there, and we decided to go here' [simultaneously pointing in the image]" (resident F). Some respondents found other situations more suitable than the MDTMs: "I think this is fantastic [while using the iPod] … then you can save, store and work with it on your own computer before the operation. … The radiologists could annotate important things and I'd look at it on my computer later on. That would be more useful than if I annotate" (surgeon T).

Interaction Technology: Multi-touch, Pens, Display, Size, Precision and Portability

Although respondents only used the three devices for about ten minutes each, most were able to give feedback on usability and ergonomics. On the other hand, the respondents were not always consistent in their preferences, as they might have preferred different devices depending on situation and purpose.

The same user interface and functionality was implemented for the two multi-touch devices. The smaller device was appreciated for its portable size, but at the same time several respondents found the display too small for efficient use. Numerous respondents, when interacting with it, only looked at the large, shared display. One respondent even stated that the small display was not needed, as it was sufficient to control the content on the shared display with multi-touch input. The zoom function of the multi-touch devices was much appreciated, as well as the responsive interface and display quality. The large device inherits most of the perceived positive properties from the smaller, but its size accounted for better precision and overview, and was therefore generally more appreciated.

Initially, the pen-based tablet was perceived as complicated to use since the functionality relied on two buttons, a scroll wheel and the pen: "I can tell you directly that this seems awkward" (senior surgeon U). On the other hand, when the respondents had used it for a few minutes, seven of them appreciated the device primarily for its high precision when pointing and drawing, and due to the fact that the large display must be viewed (which during a meeting would make participants focus on the same display): "Once you get used to it, it seems quite user-friendly, in my opinion ... this is good enough for drawing" (surgeon J).

Two respondents did not prioritize drawing precision: "Precise sketching is not that important because I think sketches and annotations are only important when referring to the operating surgeon that you haven't missed anything, that you haven't missed any point that was discussed" (surgeon S) and "The sketches would be quick, you wouldn't have to be very detailed [while using the Wacom]" (surgeon J). Three respondents were disappointed that they were not able to also use a stylus on the touch-screen devices. Two respondents believed that a high-precision device would probably be more relevant for the radiologist and the chair of the meeting, since other participants would probably not need the same level of precision. Thus, according to those participants, not everyone in the meeting needs to be able to interact with the medical imagery through networked devices.

Seven respondents were impressed by the interface and portability of the smaller multi-touch device while the larger was considered, by four respondents, too heavy (680 g) and large (1.34×19×24.3 cm) to carry around. Several respondents described ways in which they could be more effective if they had access to portable technology: benefits during individual preparation (e.g., at home, the evening before surgery) and for patient consultation. One of the surgeons said "I would look briefly right before surgery" (senior surgeon B), and another one said "… in a hospital round … [to patient] here is the problem, and we will solve it by removing this part" (Surgeon S).

Suggested Improvements, Additional Functionality and Potential Risks

It should be stressed that most respondents believed that a limited set of basic and easy-to-use tools (for pointing, navigation and annotation) was sufficient. However, the respondents did also suggest additional functionality that could support other situations in their work. As all respondents, except one, were surgeons, they did not have access to a radiologist's set of tools, and some of them mentioned features such as copying or overlaying annotations over an image sequence, comparing marked-out lesions over different image slices or examinations, and splitting the screen into two or four viewports.

Text annotations were frequently suggested as support for, and clarification of, discussed findings in the images. One surgeon said "Put the number one here, and write what we plan to do" (resident F). These text annotations could be combined with standardized options, accessible through, e.g., a drop-down menu, for simplicity and consistency. Some respondents even speculated about replacing the Pre-op meeting protocol with a set of text-annotated images as the predefined options could mandate a standard for the required information. Annotations from the meetings should of course be linked to the medical record, and physicians should at any time be able to access them.

Due to the way pointing was implemented on the devices (a circle with a size relative to the different display sizes) several respondents wanted a more precise pointing tool, such as an arrow.

While almost all respondents were enthusiastic about the possibilities of using interaction tools, a few concerns were raised about a multi-user system. Several surgeons commented that the radiologists are currently in control of demonstrating the images and that each speciality should do what they are best at. Most respondents were concerned about "chaos" if multiple people tried to interact simultaneously with the radiology images. Several respondents suggested that this would probably be resolved through social protocols: "You'd have to take turns, just like when we speak" (surgeon J). Some respondents suggested that the radiologist should delegate control when needed. Another suggestion was that maybe not all participants should have devices, but only a few senior persons.

Discussion

Our initial studies, as well as field studies by other researchers (e.g. [START_REF] Kane | Multidisciplinary Medical Team Meetings: An Analysis of Collaborative Working with Special Attention to Timing and Teleconferencing[END_REF][START_REF] Kane | Achieving Diagnoses by Consensus[END_REF]), show that surgeons often point to certain areas in medical images during MDTMs. The results from these earlier studies, and interview results from this study, suggest that using gestures far away from the projected images is neither sufficiently clear nor precise. Technical support for pointing and annotating seem to be able to alleviate such problems. Although not tested for statistical significance, our qualitative data gives strong indications for the potential of such interaction technology.

Of the three interaction techniques that we evaluated, pointing was considered most important. Our qualitative data highlighted the potential for precise pointing with digital tools that are synchronized across multiple displays, as it may save time in the discussion, include remote sites and make references more explicit for less experienced participants. Varying level of pointing precision should be supported on all devices, as a physician might want to refer to both small details and larger areas in an image. This would benefit discussions by minimizing ambiguous referencing. Annotation can, on the other hand, be accomplished with lower precision. It is not important to, for instance, exactly outline a lesion, or draw the exact path for incision during pre-operative planning. It is sufficient to highlight the area in which a lesion is located, or sketch the hypothetical incision, as the exact incision can only be determined during surgery.

Annotations could help clarify the discussion further, but were considered perhaps even more interesting when saved with the imagery for use in other situations. As anticipated, these techniques were confirmed to be most useful for Pre-ops, whereas the system should allow the data to be reused along the patient care pathway, such as for individual preparation before surgery or in Post-ops. Annotations made during the meetings, both in the form of sketches or, as several physicians suggested, text, should be saved for future use. As the physicians' work is team-based and several physicians access the patient records during the patient care pathway, annotations would help information handovers. Radiology images augmented with additional information would make it easier and faster to understand what was discussed in a meeting, overview the most important images and findings, and review proposed surgical strategies.

While most study participants felt comfortable with the radiologist navigating the material, implicit navigation on the devices is obviously necessary in the user interface to support interaction with areas of interest.

Concerns were raised about issues that, for example, could arise with personal displays that distract from the discussion, or if multiple users attempted simultaneous interaction. Thus, social protocols and technical policies that control the use need to be explored and evaluated before implementation.

Kane and Luz also discuss new technology's effect on group dynamics [START_REF] Kane | Achieving Diagnoses by Consensus[END_REF]. Often, participants already have a pre-defined role during the MDTMs, which would be affected by the introduction of tools that delegate control to other participants, an issue mentioned by several of our study participants. There was, e.g., concern that senior surgeons might start dominating MDTMs with less experienced radiologists. Surgeons and radiologists in our study, however, thought that communication protocols for turn taking would develop in the same way as when they are talking.

The three devices had their respective advantages and disadvantages, such as the pen's precision vs. the multi-touch interaction that was considered more intuitive, the ergonomic advantage of a large display size vs. portability, or the existence of a display vs. the potential risk for distraction from the shared screen. Whether, for example, personal navigation should be implemented during MDTMs, or if only a shared view should be used was also an issue identified by Wallace et al. [START_REF] Wallace | Investigating teamwork and taskwork in single-and multi-display groupware systems[END_REF], and our next step is to explore multi-user interaction during real MDTMs to evaluate how this technology affects group dynamics and to assess the importance of personal interaction. We emphasize that it is not the comparison of the devices themselves that is valuable in this context, but the qualities the study participants appreciated or found relevant.

Interestingly, several surgeons spontaneously suggested additional activities and scenarios in which the technology could be used, as there are many activities besides MDTMs in the patient care pathway. Surgery planning and preparation (e.g., the morning before the surgery) and patient consultation, were examples of other scenarios where participants thought the interactive devices could be useful. This is in accordance with previous research [START_REF] Frykholm | Medicine Meets Engineering in Cooperative Design of Collaborative Decision-supportive System[END_REF], which shows that cooperative and participatory design activities made them reflect on their own work and processes.

Physicians are also becoming increasingly mobile and will need access to updated, relevant patient information in different situations. Mobile devices could support a number of such situations, from hallway discussions to post-op reviews of complete cases. Personal navigation in a surgeon's office, for example, could be useful to understand anatomy and lesions for a specific case. Before performing surgery, the surgeon sometimes examines a 3D reconstruction of the volume. It is, in general, important for the surgeons to follow anatomical features (such as blood vessels, organs and lesions), to see how they intersect, for example. This is a functionality they all need, independent of activity (MDTMs, personal navigation or hallway discussions).

Conclusions and Future Work

We have developed and evaluated interaction technology for supporting emerging needs in multi-disciplinary collaboration for specialized medical care. Interviews, observations and a qualitative user study with surgeons and radiologists from different disciplines and varying levels of expertise, provided us with interesting insights for the next generation of digital tools for MDTMs. Unsurprisingly; the reactions were overall positive, as our developed system clearly demonstrated the potential benefits of new interactive capabilities.

Our interaction techniques (pointing, navigation and annotation) for multi-user interaction were implemented on two multi-touch devices (small and large) and a penbased tablet. The use of laser pointers that have been observed by several researchers [START_REF] Kane | Multidisciplinary Medical Team Meetings: An Analysis of Collaborative Working with Special Attention to Timing and Teleconferencing[END_REF][START_REF] Måseide | The deep play of medicine: Discursive and collaborative processing of evidence in medical problem solving[END_REF] does not support the typical multi-site video-mediated MDTMs. Digital tools do, in contrast, address the problems caused by, e.g., laser pointers that are not visible to all collaborating participants, without requiring additional hardware installation (e.g., cameras for laser pointer tracking [START_REF] Olsen | Laser pointer interaction[END_REF]). While the proposed technology for augmenting the meetings is advanced, we emphasize the importance of the perceived simplicity of the interaction techniques. We believe that three classes of devices are relevant for further exploration at MDTMs: Simple pointing tools (e.g., multiple wireless mice or trackpads), portable devices (e.g., mobile devices with large touch screens) and larger wireless devices with high-resolution displays that support both stylus (precision) and touch input (e.g., Tablet PCs).

We build on previous work and field studies, by introducing technology and strategies for interacting with information presented during MDTMs and provide a set of design guidelines, the importance of which were implicitly confirmed in the study, that we hope will serve as inspiration for future MDTM systems. Our results show that more precise referencing could augment the expressiveness of participants, improve the communication with the other participants (including the radiologist), help less experienced participants follow the discussion, and bridge the gap to remote experts.

By introducing distributed interaction along with proper social protocols and system policies, its potential for improved collaboration, understanding and discussion could lead to safer and faster medical decisions, with a significant impact for modern healthcare.

This study gathered insights during a controlled simulated team meeting in preparation for follow-up studies during real Pre-operative meetings. It allowed us to collect valuable unbiased, undisturbed and undistracted feedback through a series of individual sessions. Based on the results from this work, we are now refining the techniques and the setup, and plan to run experiments with the next generation of our prototypes in collaborative multi-user meetings.

Figure 1 :

 1 Figure 1: Synchronized interaction with projected radiology material, using multi-touch gestures and widgets to zoom and scroll through the image stack.

Figure 2 :

 2 Figure 2: Left) Radiologist controls the interaction with the imagery from his workstation by navigating (scrolling through image stacks) and pointing out findings with the mouse. Right) Medical specialists from different disciplines (e.g., pathology, oncology and hepatology) view and discuss the imagery from a distance, and can only refer to areas of interest through verbal means or finger pointing.

Figure 3 :

 3 Figure 3: Surgeons and radiologists participated in interactive prototyping sessions. Left) A radiologist manipulates content on a shared display using a mobile device. Right) Interaction on a Tablet PC (Dell XT2) that supports both pen and multi-touch input.

Figure 4 :

 4 Figure 4: The tablet uses buttons and a touch strip with a tracked pen for panning, zooming and drawing.

Figure 5 :

 5 Figure 5: The multi-touch displays use on-screen buttons and widgets in combination with multi-touch gestures.

Figure 6 :

 6 Figure 6: Photographs from our user study, where participants explored the penbased device (left), a small (middle) and a large (right) multi-touch display. The monitor represents the shared projection screen at the MDTMs.


Tony Bergstrom

Andrew Harris
email: harris78@illinois.edu

Karrie Karahalios

Encouraging Initiative in the Classroom with Anonymous Feedback

Keywords: Social Mirrors, Classroom, Feedback, Anonymous

Inspiring and maintaining student participation in large classes can be a difficult task. Students benefit from an active experience as it helps them better understand the course material. However, it's easy to stay silent. Opportunities to participate in conversation allow students to question and learn. The Fragmented Social Mirror (FSM) provides students with the ability to anonymously initiate classroom dialog with the lecturer. The system encourages participation by enabling expressive anonymous feedback to reduce evaluation anxiety. The FSM further catalyzes participation by allowing for many simultaneous participants. In this paper, we introduce the FSM as a classroom device, discuss its design, and describe a pilot test of the interface. Initial results indicate a promising direction for future feedback systems.

Introduction

Students learn more when they actively engage in the classroom [START_REF] Weaver | Classroom Organization and Participation: College Students' Perceptions[END_REF]. However the structure of many classes ensures that the lecturer speaks for at least 80% of the time. Though some students participate, it's expected that five students out of 40 will come to dominate any classroom discussion.

The lecturer's awareness of class comprehension is skewed both by the students' many social pressures and the few speaking opportunities. Students try to present a positive image of themselves to their peers. Thus, they often avoid volunteering information due to evaluation anxiety, a fear of being judged by others for making a mistake or being the focus of attention [START_REF] Weaver | Classroom Organization and Participation: College Students' Perceptions[END_REF]. It's easy to remain silent. Those students who do speak are generally self-confident or understand the material. However, there is a reluctance to appear too engaged in the classroom. Students who raise the expectations on a group may be ostracized by their peers [START_REF] Parks | The desire to expel unselfish members from the group[END_REF].

In this paper, we present the design of an interface prototype to encourage student engagement and improve the lecturer's awareness in the classroom. The prototype, entitled Fragmented Social Mirror (FSM), aims to create a new communication channel of anonymous dialog between the instructor and the class. Unlike many previous Audience Response Systems [START_REF] Fitch | Student feedback in the college classroom: A technology solution[END_REF][START_REF] Stowell | Benefits of Electronic Audience Response Systems on Student Participation, Learning, and Emotion[END_REF], FSM allows for expressive text-based feedback and may be used throughout the lecture. In our short pilot observation, students in a large class began to initiate interaction with the professor, whereas previously they had only mumbled answers in response to posed questions. In this paper, we describe the design of the FSM in the context of other Audience Response Systems. We also discuss promising initial observations from a classroom pilot study of the FSM.

Feedback in Conversation

FSM is designed to extend the benefit of backchannel communication. Familiar face-to-face backchannels include "yeahs," "uh huhs," and head nods that show attention to the speaker. Similarly, facial expressions reveal feelings while gestures provide emphasis [START_REF] Krauss | The role of audible and visible back-channel responses in interpersonal communication[END_REF]. From the listener's perspective, these signals show the speaker that the audience is listening and is interested. In a large audience, these visual signals can be lost in the crowd, which necessitates speaking up, murmuring, and applause. Large lectures and speeches require a strong and focused mediation to centralize attention [START_REF] Dunbar | Size and structure of freelyforming conversational groups[END_REF]. Without that centralization, small groups of 4-5 individuals form based on proximity of others.

The classroom environment has to balance the need for a strongly focused discussion with the need for student feedback. Lack of participation makes it more difficult to assess the current understanding of students and by not encouraging participation students are less apt to risk being wrong [START_REF] Jones | Silent Sixth-Grade Students: Characteristics, Achievement, and Teacher Expectations[END_REF]. However, from a constructivist view of learning theory, students need to be actively engaged in their learning. Students learn by actively building their own understanding of new information [START_REF] Anderson | Strategic teaching and learning: Cognitive instruction in the content Areas[END_REF][START_REF] Von | [END_REF]. Instruction alone does not directly allow for the construction of knowledge, however effective knowledge construction often adopts a social process between the student and the teacher [START_REF] Brown | Situated cognition and the culture of learning[END_REF] Classrooms have addressed this issue by using Audience Response Systems for multiple choice and true/false questions. The interfaces in [START_REF] Fitch | Student feedback in the college classroom: A technology solution[END_REF][START_REF] Stowell | Benefits of Electronic Audience Response Systems on Student Participation, Learning, and Emotion[END_REF] provide a small number of preselected responses of A/B/C/D, and a true/false response. These interfaces are most often used when the lecturer explicitly asks a multiple-choice question of the audience. To be effective, the lecturer has the difficult task of anticipating key moments to query the audience and must specifically structure a lecture to accommodate this new question-answer format. Though each system varies, many include specialized hardware, which are either purchased by or provided for each student [START_REF] Kay | Examining the benefits and challenges of using audience response systems: A review of the literature[END_REF][START_REF] Stowell | Benefits of Electronic Audience Response Systems on Student Participation, Learning, and Emotion[END_REF]. In the worst cases, when a lecturer does not incorporate interesting interaction into the lecture, the Audience Response Systems can become automated attendance and quiz systems, which students grow to resent [START_REF] Kay | Examining the benefits and challenges of using audience response systems: A review of the literature[END_REF].

Other feedback modalities such as text-based systems provide opportunities for students to engage with each other [START_REF] Mccarthy | Digital backchannels in shared physical spaces: experiences at an academic conference CHI '05[END_REF][START_REF] Yardi | Whispers in the Classroom[END_REF]. Studies of dedicated course chatrooms show students will chat about the lecture's content to help explain concepts to confused classmates [START_REF] Yardi | Whispers in the Classroom[END_REF]. However, chat rooms also encourage unrelated discussions, and potentially draw students away from the lecture. Some have gone further and tied the in-class chat discussion to a video recording of the lecture for archival and review [START_REF] Baecker | Webcasting made interactive: persistent chat for text dialogue during and and about learning events[END_REF]. For a practice talk or presentation, these systems allow people to access the initial thoughts of the audience in a asynchronous manner. In addition to making help available, this style of active learning helps students communicate concepts to peers for a deeper understanding of the material. Others have brought affective computing to the classroom by using a custom handheld device [START_REF] Balaam | Exploring affective technologies for the classroom with the subtle stone Proceedings of the 28th international conference on Human factors in computing systems[END_REF]. This handheld ball can be used to indicate the emotional state of the student to the instructor. Outside of the classroom, text based systems similarly open public dialog into shared events through IRC, instant messaging, Twitter, and Facebook [START_REF] Ebner | Social networking in scientific conferences -Twitter as tool for strengthen a scientific community[END_REF][START_REF] Mccarthy | Digital backchannels in shared physical spaces: experiences at an academic conference CHI '05[END_REF][START_REF] Shamma | Tweet the debates: understanding community annotation of uncollected sources WSM '09[END_REF]. Though all of these side-channels can contribute to audience discussion, they often leave the speaker out of the loop during the event.

Writing systems, such as Classroom Presenter, benefit from writing as input while still including the instructor in the interaction [START_REF] Anderson | Classroom Presenter: Enhancing Interactive Education with Digital Ink[END_REF]. A tablet PC system, it allows students to mark directly on the current slide with a stylus, which can then be viewed and shared by students and the instructor. Instructors can set up slides that encourage students to answer questions that can be discussed and reviewed as a group. It also allows for more expressive diagrams, images, and nonverbal communication mechanisms. This method enables a broad sampling of student understanding and encourages active participation with the material.

Related work such as backchan.nl and Conversation Votes create a new feedback channel that integrates approval feedback into group dialog [START_REF] Bergstrom | Vote and Be Heard: Adding Back-Channel Cues to Social Mirrors[END_REF][START_REF] Harry | nl: integrating backchannels in physical space[END_REF]. With backchan.nl audience members organize their collective questions for the speaker in a conference or after a talk. A moderator filters the most appropriate questions from the top rated questions. With Conversation Votes, participants annotated an abstract visualization of conversation with positive and negative votes to highlight agreement during conversation. In small groups, this anonymous feedback increased the level of participation from those less satisfied with previous conversations.

Viewed on an axis of expressivity, distinct categories of low expressivity and high expressivity emerge. Low expressivity systems as in [START_REF] Bergstrom | Vote and Be Heard: Adding Back-Channel Cues to Social Mirrors[END_REF][START_REF] Fitch | Student feedback in the college classroom: A technology solution[END_REF][START_REF] Stowell | Benefits of Electronic Audience Response Systems on Student Participation, Learning, and Emotion[END_REF] limit what a student can communicate, but ensure the feedback can be quickly interpreted. High expressivity systems like [START_REF] Anderson | Classroom Presenter: Enhancing Interactive Education with Digital Ink[END_REF][START_REF] Harry | nl: integrating backchannels in physical space[END_REF] and chatrooms allow students vast communication capabilities, but can require more focused attention for both the lecturer and students.

Our work takes a middle path. FSM provides a meaningful, but constrained, set of signals to be observed alongside the lecture like low expressivity systems, but it allows expressive text to convey personal ideas as in a high expressivity system. As an always-available interface, the FSM captures the fleeting moments of confusion and conveys this information to the lecturer while it can be addressed in context.

The Fragmented Social Mirror

The Fragmented Social Mirror (FSM) provides feedback based on principles borrowed from our previous work in social mirrors [START_REF] Karahalios | Social Mirrors as Social Signals: Transforming Audio into Graphics[END_REF]; however, the classroom setting necessitates a break from the standard social mirror design. A social mirror is a realtime depiction of interaction meant to augment natural face-to-face environment. It captures ephemeral moments in conversation and brings them into the public view through visualization. In our previous work, social mirrors displayed abstract visual-izations to depict participation in conversation. The resulting display of conversational dominance, non-participation, and turn taking encouraged more balanced conversation [START_REF] Bergstrom | Vote and Be Heard: Adding Back-Channel Cues to Social Mirrors[END_REF]. In these social mirrors, one shared visualization of conversation was projected centrally for all participants to see. On a classroom or large lecture scale, this form does not function as well. There are many more participants involved, and the architecture of the space is different from the spatial layout of small group interaction around one shared table. Furthermore, there is a natural asymmetry in participation due to the lecturer-audience dynamic. This results in less interaction between the lecturer and the audience not suited to the traditional social mirror visualizations.

The term "fragmented" in FSM refers to the use of individual interfaces for each participant as opposed to one shared visualization and the shortened time component as opposed to the full history present in previous social mirrors. In our setup, each participant accesses a Java applet from his or her computer or mobile device, while a large public display is presented to everyone. Furthermore, while a traditional social mirror maintains a persistent history of interaction, FSM highlights questions and comments that are pressing at the specific moment.

FSM Design Choices

The FSM design focuses on capturing and reflecting the unheard and unvoiced dialog in the classroom. The current design is the result of a long chain of prototypes that sought to balance the need for attention to the interface with attention to the classroom. The final design in this paper served as a culmination of this prototyping, though our pilot demonstrated areas for further refinement such as moderation.

The process of designing the FSM began by observing an active and engaged classroom of 100+ students to see what students say when engaged in an active class. The lecturers of these classes were generally rated as among the best in the department. They were engaging during lecture and good at encouraging student participation. To facilitate more participation the lecturers posed a question and waited for responses -thus guaranteeing an answer or a question for clarification. We noted all the types of student responses to better understand what a student wants to say during class. The responses were narrowed down to the following list of categorical responses:

• Questions: Students provide new questions based on what has just been taught.

• Information: Students add their own connection to outside subjects.

• Agreement/Disagreement: Answering a Lecturer's question.

• Slow Down/Redo: Students did not understand the lecturer.

• Cannot Hear/Repeat: Students did not hear the lecturer.

Our list was very similar to feedback available in other work to mark up a presentation slide [START_REF] Vandegrift | Promoting Interaction in Large Classes with a Computer-Mediated Feedback System[END_REF]. We began to investigate this set of six messages for our prototypes. These six messages would serve as categories with the ability for students to include a short text message for explanation. The message categories serve as a means to identify and group similar responses and highlight important categories like questions. In parallel with our interface design, we investigated imagery for each of these six categories of messages (described in the next section). Due to this process, "Slow Down/Redo" and "Cannot Hear/Repeat" were eliminated. Suitable icons could not be found and they easily be replaced by an "Information" message with appropriate text.

Many of our initial interface prototypes borrowed design components from the Conversation Clock and Conversation Votes [START_REF] Bergstrom | Seeing More: Visualizing Audio Cues[END_REF][START_REF] Bergstrom | Vote and Be Heard: Adding Back-Channel Cues to Social Mirrors[END_REF], they incorporated the feedback into a timeline that structured the activity throughout the session. In some cases, we included indications of speaker. Much like a standard instant messenger, the full history of messages could be read through at any time. These interfaces showed potential for the review of archival classroom data, but did not serve our purpose of encouraging classroom interaction. These prototypes, tested amongst our own group, required too much attention to adequately understand.

After refining the initial prototypes, we settled on a simple interface students could use without pulling their attention too far from the lecturer. Thus the input of the FSM was used only for capturing one comment. The history of feedback was only seen on the public display and limited that history to the most recent comments. Additionally, the needs of the lecturer necessitated this type of design. The lecturer needed to be able to read feedback from the hundreds in the audience while still being able to teach effectively. In past studies, a social mirror was primarily viewed by the listeners (and not the speaker) in conversation because they had more free attention [START_REF] Bergstrom | Seeing More: Visualizing Audio Cues[END_REF]. In this design, the captured feedback of conversation is significantly pared down, so that the lecturer can receive the benefits from the social mirror with minimal attention. Therefore, current comments/questions are displayed so as not to overwhelm the viewers with a long history.

Iconographic Messages

The FSM interface passes information through icons. These graphics serve to simplify the message so that the lecturer might easily understand the classroom without reading too much content. Based on informal observation of classroom sessions and prior work [START_REF] Vandegrift | Promoting Interaction in Large Classes with a Computer-Mediated Feedback System[END_REF], we designed icons based on the messages earlier: "I have a question," "I have information/an answer," "Yes/agree," "No/disagree," "Speak Up," "Slow Down." Three researchers independently drew any graphic that they felt reasonably captured these messages. We combined them into sets for each category, with a total of 5-15 images for each message.

We conducted a survey of Computer Science undergraduates to test our icon designs. A total of 54 Computer Science undergraduates completed our survey. Their feedback identified 17 icons that convey the intended message. Figure 1 shows all 17 icons. None of the icons for "Slow Down" conveyed an adequate message to the student. We eliminated this message, as well as the "Speak Up" messages in favor of a simpler 4-icon interface. Students can use the Information and Question messages with additional text to signal "Slow Down" and "Speak Up."

The icon represents confusion or the desire to ask a question.

disagree agree disagree agree disagree agree disagree agree

The icon represents an audience member has new information.

disagree agree disagree agree

The icon represents disagreement or a negative response.

disagree agree disagree agree disagree agree

The icon represents agreement or a positive response.

disagree agree disagree agree disagree agree disagree agree

The icon represents that the lecturer needs to increase their volume.

disagree agree disagree agree disagree agree disagree agree Fig. 1. The results of 54 individuals rating our sets of icons produced these icons as the most indicative of their intended messages. The image survey informed our final selection of icons seen in the interface (Figures 1 and3). One other message, to indicate the speaker was moving too fast, produced no positively rated icons.

FSM Interfaces

There are two FSM interfaces -the student's client interface for a computer or handheld device (Figure 2) and a larger public screen for the lecturer and audience (Figure 3). The public display is situated in the front of the room, though the lecturer sees the public display on a personal screen. The four different preselected icons categorize student responses in the student interface. The icons represent: Information, Questions, yes/agree, no/disagree. Of the four categories or signals, the Information and Question signals can be augmented by a 40-character message. The short messages allow students to clarify their questions or possible answers when there is no opportunity to speak while the yes/no buttons allows students to answer simple questions quickly.

Students use the client interface in Figure 2 to send their message to the public display shown in Figure 3. All messages on the public display are grouped by their associated icon to increase legibility for the speaker. The speaker can look up and see many questions that need to be addressed or they can glance over answers that students provided via the display. The icon group with the most messages moves to the top of the screen with a larger icon. The most recent message of this icon appears at the top of that icon in white text set against the black background. As a message ages, it fades to grey before finally disappearing after a pre-configured time. For icons with multiple messages, a count is displayed to the left of the icon.

Messages on the public display are limited to recent messages. Only the most recent minute of activity is visible; each message fades in brightness over the minute before disappearing from view. The rationale for this design was two fold: (1) we did not want the lecturers to be confused or overwhelmed by reading old questions from a prior part of the lecture and (2) if a question goes unanswered and disappears, this removal may encourage a student to verbalize the question in class or to repost it. One of our main goals is to encourage more class interaction. If a student can "see" that they are not alone in their confusion, they may be less apprehensive to speak out and ask a question.

Once a student sends a signal via posting an icon, they are blocked from sending additional signals for a brief period (10 seconds in our pilot) to discourage excessive social chatter and monopolization of the channel. While there is some room for abuse as with the backchan.nl system, where some users voted up questions for humor [START_REF] Harry | nl: integrating backchannels in physical space[END_REF], the public availability of the channel is ultimately at the discretion of the lecturer. We conducted a pilot study to investigate the FSM in the classroom. We began by observing the participation levels before the introduction of the FSM and again with the FSM in place. For this, we observed a required second year course with roughly 180 registered students at the beginning of the semester. The instructor was not affiliated with our research team. We observed a total of six course sessions: three initially without any augmentation, and three with the addition of the FSM. During observation, an average of 100.0 students were in attendance, though there were fewer students in the final sessions (attributed to an intervening midterm and final day to drop the course). Given the large class, not many students had the opportunity to speak, and most did not. A summary of the attendance is visible in Figure 4. Prior to testing the FSM in class, we sent a pre-survey and described the use of the FSM. The survey inquired about the student's comfort level while participating in class versus their smaller discussion sections. Feedback from the survey confirmed that students are not comfortable asking questions or asking for clarification during class, though they are more comfortable asking in their smaller recitation sections. Similarly, they recognize that they do not participate or ask questions during class (Figure 5).

Pilot Study

Preliminary Survey

Our initial observations showed little interaction between audience and lecturer over the course of three 50-minute sessions. The only activity from the audience was in response to questions posed by the lecturer. For example, students were asked "n is divisible by what?" and "What is the cardinality of set Q?" in reference to a proof. The class averaged about four responses per class. The students initiated zero interactions themselves, five of the twelve responses were general indefinite murmurs from the class, and two responses involved raising hands. Various sets of 1-3 unidentified students spoke up to answer the remaining six questions. Fig. 6. We tested the Fragmented Social Mirror in a large lecture hall with three projection screens. Typically, the instructor repeated the same material on all three screens. For the study, the Fragmented Social Mirror replaced one screen during class time. The screen by the lecturer also displayed the public interface.

We tested the FSM in three class sessions and found the students were proactive in using the system. In the classroom, the lecturer used a central projection screen to work through problems by hand while a smaller screen displayed the public display to the right of the larger screen (Figure 6). At the lecture podium, the lecturer also had a copy of the public display available during the class activity.

With the system in place, Students initiated dialog with the lecturer by asking questions 11 times, compared to zero without the system. When on topic, students used the Fragmented Social Mirror to ask questions of the professor, keep the professor from moving on too quickly, and to answer any questions the professor posed. Figure 7 summarizes the participation in each of the 6 classes. Most of the on-topic dialogs either began with or contained a question for the instructor. They lead to discussions with the instructor and information to enrich the class. However, there were also many off-topic messages. These messages were irrelevant to the class topic and were used to draw the attention of other classmates away from the lecture material for their own entertainment.

Questions Posed by Dialog Relevance

Session Instructor Student On-topic Off-topic

1 No FSM 3 0 3 0 2 No FSM 3 0 3 0 3 No FSM 4 0 4 0 4 FSM 2 2 6 (22) 2 (2) 5 FSM 2 7 10 (30) 4 (37) 6 FSM 1 2 6 (8) 3 (7)
Fig. 7. The Fragmented Social Mirror encouraged students to initiate questions and dialog with the instructor. Many dialogs began with or contained questions, though some were responses to indicate comprehension of the material. The anonymity of the interface also encourages unrelated dialog in class. A dialog in the Fragmented Social Mirror could include multiple messages -thus, # dialogs (# messages) includes both the count of interaction instances and total messages.

Example FSM Dialogs

Excerpts from the FSM sessions appear below. In this first example, students requested information that the professor was not trying to teach but established an interesting aside on history related to the lesson:

Instructor [discussing the Karatsuba Algorithm]
<What is Karatsuba?>

Instructor

Karatsuba is the guy who invented it, Anatolii Karatsuba. [Instructor continues with a bit more history.]

As another example, the student's lack of understanding prompts him or her to ask for clarification on calculating tree height.

Instructor [Providing an explanation of tree depth] wow What is the height again?

? The maximum depth of the tree. You can count the Levels by generation ? Not 5? It's max level and not count?

Instructor Yeah, it's 4 not 5 ..

. [continues on 0 based counting]]

In cs you start counting at zero :) However, with the addition of initiating comments, there was also an increase in comments solely intended to draw attention away from lecture. These messages often had nothing to do with the lecture or a question tended to come in bursts in order to overwhelm the public display for a short time. As an example of such a burst:

? HATE HATE HATE HATE HATE HAT ? I DONT CARE WHAT THESE CHICKS SA ? I DONT EVEN LOOK THAT WAY ? EVERY TIME I WALK IN THE CLU ? THEY HATIN ON ME CUSE THEY KNO I LK GOO This type of interaction was most prevalent in the second session. The lecturer was inclined to read them, see that they were not relevant and either laugh, if it were funny, or state "I don't know what this means." However, the increase of messages also meant that the lecturer was more likely to miss relevant exchanges where a student was asking for help:

? can you draw the picture for the tree afte ? after applying rule 3 ? :( ? +1

After the sessions

We had only planned to gather initial observations to refine the system in these first sessions; however, the instructor was excited to see the students participating and invited us to return with the system for further studies. After the lectures, she indicated that it's always been hard to get this many students to say anything, even with encouragement. The simplicity of the display was also deemed useful, as she could read the questions with a glance. Additionally, the asynchronous nature allowed students to ask their questions while she was still explainingthus allowing her to work the question into that explanation or come back to it later. Student feedback indicated the device was useful as they "didn't have to try to get the professors attention" by raising a hand from the back of the lecture.

Students also saw the benefit of the interface, and felt it was easier to participate in the classroom (Figure 8). However, they recognized the difficulty of maintaining order in the anonymous display and provided suggestions to keep the interface on topic. One such suggestion was to make the display semi-anonymous; implement a publicly anonymous interface that retains the identity on the lecturer's display. In this way the instructor could call out any abuse of the display, while protecting the identity of any others who were uncomfortable commenting in front of the class. A similar suggestion would simply log the identities for review after class.

Discussion

The Fragmented Social mirror touched the surface of integrated feedback in large discussions by allowing a controlled set of feedback that could express a student's question or response. As we mentioned in related work, the field can be divided into the high expressivity interfaces that require more attention and the low expressivity interfaces that do not allow students to indicate their questions.

In the sessions with the FSM, the classroom dialog was more involved. The lecturer felt like she was talking to people rather than at people while the students took a more proactive role in directing conversation to points that were not understandable. With 100 students, evaluation anxiety limits the individuals willing to speakhowever we have shown that anonymous feedback can break the barrier and include more students.

Response Questions disagree agree

The system encourages participation.

disagree agree

The system encourages my own participation.

disagree agree

The system encouraged me to ask questions.

disagree agree

The system encouraged my peers to ask questions.

disagree agree

The system made me feel connected to the lecture.

disagree agree

The system made me feel connected to the presenter.

disagree agree

The system made me feel connected to the other audience members.

disagree agree

The icons were clear in their meaning.

disagree agree

The use of text entry was clear in it's meaning.

disagree agree

The system was easy to use.

disagree agree

The feedback from others helped me to understand the lecture material.

disagree agree

The feedback from others made the lecture more enjoyable.

disagree agree

The system is a worthwhile addition to the lecture and discussions.

Fig. 8. Students reported the Fragmented Social Mirror encouraged participation and connected them to the lecture. Students reported the lectures being more enjoyable and saw the feedback as a worthwhile addition to the lectures.

Our interface was not perfect; the classroom sessions revealed that while anonymity opens opportunities for discussion, it must be tempered in some manner. In both the second and third sessions, some individuals engaged in the public conversation by adding potentially disruptive comments that lead to legitimate questions not being discussed. As a prototype, we did not fully flesh out any mechanisms to prevent this type of interaction. Perhaps the lecturer should be able to flag such comments during conversation to lock out individuals, or the lecturer should be able to identify individuals after the classroom session and deduct points in some manner. While it is tempting to simply allow the class to moderate itself, the design must be careful not to become more of a distraction as it requires interaction outside the scope of a learning task and draws them away from the lecture.

The classroom sessions also made it apparent that the positive and negative responses should redesigned to provide more flexibility. Though students did use them as feedback for the lecturer, they also use the agreement checkmark to indicate a "me too" when other students raised a question. Others adopted a convention of adding a "+1" as seen in the example conversations. A revamped system might allow a student to indicate "me too" and ensure the question stays visible long enough for the lecturer to see the question.

Our surveys underscore the need for large classrooms to tap into technological backchannels. Students know that they do not participate in large classroom settings even though they have questions. They are not comfortable asking questions in such a large group. The survey after the use of our system shows that students felt encouraged themselves and the class to ask questions, they found it to make the lectures more enjoyable, and it was a worthwhile addition to the lecture.

The FSM interface received a positive response from both students and lecturer. Our initial study highlights the use of anonymous signals in large classroom has potential to draw in more active participation of the students and audience.

Conclusion and Future Work

A large audience automatically reduces the possibilities for participation in a lecture setting. Rather than accept this and move on, technology has provided new channels to engage students throughout the lecture.

Our own work shows that expressive feedback can be gathered from lightweight systems. The use of these channels was shown to increase engagement, to benefit the students, and to spark initiative where there was none before.

While our system was successful in engaging students and encouraging participation, we acknowledge it is not a definitive study and has limitations. Our study was a small study to test our conception of anonymous feedback. The system, with refinements, should be further tested over a longer term and in multiple classrooms. Many untested facets of the interface can be further explored. We advocate anonymous feedback based on the premise of evaluation anxiety, though we have not yet tested the effects of allowing or enforcing identity in the FSM.

Fig. 2 .

 2 Fig. 2. The input device is small and simple for classroom use. The two left icons for information and questions allow for typing phrases to send along with the icon message.

Fig. 3 .

 3 Fig. 3. The public display groups messages by icon and highlights the most recent feedback from the class. To the left of each icon, a counter indicates multiple messages of the same type, particularly useful when using the yes/no feedback buttons.

  This table shows the number of people in the classroom for each session as well as the number of computer visible. Though we did request students to bring their computers to the sessions with the Fragmented Social Mirror. The number of computers remained essentially unchanged.

	Session	Students	Computers	On FSM
	1	108	17
	2	125	25
	3	112	19
	4	102	23	14
	5	80	19	15
	6	73	14	5
	Fig. 4.

  Students reported they were uncomfortable asking questions in class, though it was less the case in smaller recitation sections. They are aware that they do not participate in class and are less likely to ask questions even though they have them. There is reluctance amongst the students to speak up and participate. Survey with 23 respondents. All graphs have the same scale.

			Responses and Questions
	in Class		in Discussion	Question
					I participate.**
	disagree	agree	disagree	agree
					I participate more than
	disagree	agree	disagree	agree	once.**
					I ask questions.*
	disagree	agree	disagree	agree
					I ask multiple questions.**
	disagree	agree	disagree	agree
					I have more questions than
	disagree	agree	disagree	agree	time allows.
					I feel my questions are re-
	disagree	agree	disagree	agree	spected.
					I am comfortable asking
	disagree	agree	disagree	agree	questions. ***
					I am comfortable asking for
	disagree	agree	disagree	agree	clarifications.***
	Satisfaction	Dissatisfaction
					I am comfortable expressing
	disagree	agree	disagree	agree	satisfaction/dissatisfaction.
	Ambivalence	Enthusiasm
					I am comfortable expressing
	disagree	agree	disagree	agree	ambivalence/enthusiasm.*
	Opinions		Questions
					My peers are comfortable
					expressing opin-
	disagree	agree	disagree	agree	ion/questions.
			By paired Wilcoxn Rank Sum Test: * p < .05, ** p < .01
	Fig. 5.

The Fragmented Social Mirror indicates that the use of text based anonymous feedback has potential for promoting engagement in the classroom. A long term study could investigate the effects on learning outcomes: does the FSM encourage students who are already engaged in class to further surpass their peers, or does it genuinely help students who just need a small boost to get involved?

We hope to explore these further questions on the benefits of classroom feedback in future works with the FSM and other interfaces.