Min Zhang

Guoping Qiu
email: qiu@cs.nott.ac.uk

Natasha Alechina

Sarah Atkinson
email: sarah.atkinson@nottingham.ac.uk

A Comparison of Five HSV Color Selection Interfaces for Mobile Painting Search

Keywords: Color palette interface, user study, painting search, user interface design, drawing-to-search, mobile application

Color selection is a common task in a plethora of mobile applications. Although a variety of color palettes emerge, there are no design guidelines or studies of the use of color palette interfaces for drawing on touch-screen phone. We are particularly interested in drawing queries to search for paintings on mobile phone. In this paper, we classified the color palette interfaces into several categories. We report results of a systematic experiment with 41 participants using five different types of HSV color palettes for the task of drawing a painting to search on a mobile phone. We investigate which color palette(s) enable users to complete task faster, how good these resulting drawings were for searching, and what were user experiences. Users' drawing behavior is also discussed.

Introduction

Color is a salient visual attribute for both paintings [START_REF] Davidoff | Cognition through color[END_REF] and our daily life. When searching paintings on the keyword-based image search engine (e.g. Google), we found that most of queries describe the expected paintings b y use of color names such as 'golden lady in green', 'yellowish hat' etc. However, the huge 'semantic gap' [START_REF] Hare | M ind the Gap: Another look at the problem of the semantic gap in image retrieval[END_REF] makes search results unsatisfactory. Moreover, it is difficult for users to describe some paintings and pictures in several keywords, such as abstract paintings.

With the proliferation of the touch-screen mobile devices, people are able to express their intention more freely. Zhang et al. [START_REF] Zhang | Can People Finger-draw Color-sketches from M emory for Painting Search on M obile Phone?[END_REF] proposed that most users could draw out rough color-sketches of previously-seen paintings from their memory. As one of the image retrieval paradig ms, Banfi [START_REF] Banfi | Content-based image retrieval using hand-drawn sketches and local featuresa study on visual dissimilarities[END_REF] concluded that Query-by-Drawing (QbD) was the most flexib le way for users to present their search ideas. There are many studies on QbD algorith ms [START_REF] Kato | A sketch retrieval method for full color image database-query by visual example[END_REF][START_REF] Springmann | Towards Query by Sketch[END_REF] which focused on the search accuracy and efficiency, however, little is known about the design guidelines of the natural and intuitive user interfaces that facilitate user's drawing.

For the s mall-screen mobile phone, drawing color patches [START_REF] Giangreco | A User Interface for Query-by -Sketch Based Image Retrieval with Color Sketches[END_REF] roughly is mo re reasonable [START_REF] Ivanova | Analysis of the Distributions of Color Characteristics in Art Painting Images[END_REF]. One of the most important tasks for drawing is color selection. There are a variety of ways to select colors, and the most common method is to directly man ipulate a color palette [START_REF] Douglas | Do color models really make a difference?[END_REF]. The color palette usually includes a color model and an interface to represent colors to users. Several studies were conducted to explo re the effect s of different colo r models on color matching task. Schwarz et al. [START_REF] Schwarz | An experimental comparison of RGB, YIQ, LAB, HSV and opponent color models[END_REF] co mpared five color models: RGB, HSV, LA B, YIQ and Opponent, and they found that RGB color model was the fastest to use yet least accurate, while HSV color model was the most accurate to use. Ho wever, these arguments were only concluded fro m conducting the color matching experiment, wh ich ignored the color selection interface for drawing purpose.

This work aims to focus on the user interface of color palette, by focusing on a specific color model and a current Query-by-Drawing algorithm, to exp lore the influence of different color palette interfaces on color selection for drawing query on mobile phone. Specifically, an experiment was conducted to collect task performance and users' attitudes on five different color palette interfaces.

Related Color Palettes Study

The color palette has been widely exp lored by the scientific v isualization researchers [START_REF] Rogowitz | Why should engineers and scientists be worried about color? Lloydia Cincinnati[END_REF], with aims of displaying information efficiently by comb ining proper colors.

Looking through current mobile application (referred to 'App' in the remainder of this paper) market and computer software, including mobile d rawing Apps, coloring games for kids, note-taking Apps, and photo editing software and Apps, we found that a plethora of different color palettes were adopted by different developers according to their individual preference. To the best of our knowledge, there is no design guid eline about making use of color palettes for drawing on touch -screen mobile devices, especially for image search by drawing mobile App. As mentioned earlier, some previous work has investigated the effects of color model on the speed and accuracy of color selection in co lor matching experiment [START_REF] Schwarz | An experimental comparison of RGB, YIQ, LAB, HSV and opponent color models[END_REF][START_REF] Henry | An Intuitive Color-selection Tool[END_REF][START_REF] Everly | Color selection methods using the color picker[END_REF][START_REF] Douglas | M odel and Representation: The Effect of Visual Feedback on Human Performance in a Color Picker Interface[END_REF], in which part icipants choose a closest color to a given color shown on the computer screen by use of the color palettes with d ifferent color models. Besides Schwarz et al.'s study [START_REF] Schwarz | An experimental comparison of RGB, YIQ, LAB, HSV and opponent color models[END_REF], Everly and Mason [START_REF] Everly | Color selection methods using the color picker[END_REF] also assessed the usability and accuracy of four color selectors via exp loring participants' performance on color matching. Dogulas and Kirkpatrick [START_REF] Douglas | M odel and Representation: The Effect of Visual Feedback on Human Performance in a Color Picker Interface[END_REF] found that the color space had little impact on color selection. However, the color matching task is out of context of the realworld drawing scenario in which choosing one color might be influenced by the su rrounding or background colors.

There are a few studies on the color palettes used in Content-based Image Retrieval (CBIR). For examp le, Broek et al. [START_REF] Broek | Content-Based Image Retrieval: Color-selection exploited[END_REF][START_REF] Broek | Design Guidelines for a Content-Based Image Retrieval Color-Selection Interface[END_REF] rev iewed ten color selection interfaces for CBIR and categorized them into three groups: slider-bars, discrete color matrix, and continuous color square. However, their work was conducted on the computer-based CBIR systems almost 20 years ago. Their limitations include: 1) non -intuit ive interaction of color selection by using mouse; 2) out-of-date color palette interfaces; 3) not representative of the current mobile device capability. Zhang et al. [START_REF] Zhang | User Study on Color Palettes for Drawing Query on Touchscreen Phone[END_REF] co mpared t wo discrete color palettes with t wo continuous color palettes (HSV square and HSV color wheel) on mobile phone and concluded that participants preferred the color palette with full-choice of co lors during drawing search query. Inspired by Zhang et al.'s work [START_REF] Zhang | User Study on Color Palettes for Drawing Query on Touchscreen Phone[END_REF], our purpose is to explore which kind(s) of continuous color palette inte rface work best for a Drawing-to-Search mobile App. We also adopt HSV co lor model in this work, because it is perceptually intuitive and is used by lots of current mobile applications.

Current color research all define a color in a triplet of three parameters, hence selecting a color means finding a location in a three-dimensional (3D) space. We div ide the continuous color palette interfaces into '1D-1D-1D' and '1 D-2D'. In '1D-1D-1D' interface, each one of three parameters is controlled by one motion (such as a trio of 1D sliders); while in '1 D-2D' interface, one motion simu ltaneously change two parameters and one for another parameter, e.g. co lor palettes shown in Fig. 1. In this paper, we focus on comparing the '1D-2 D' color palette interfaces based on the human-o riented HSV (Hue-Saturation-Value) color model, because this pair of interface and color model is frequently used in both current color selection tools and the graphics literature.

3

Query-by-Drawing Application

We When users complete one drawing, they tap the 'Search' button to submit their drawings to our server, wh ich will extract drawing's color features and match them to that of database images by using Fast Multi-resolution Image Querying algorithm [START_REF] Jacobs | Fast M ultiresolution Image Querying[END_REF]. Top fifty-one matched images, three image thumbnails on a line and 17 lines in total, will be returned fro m server to user's mobile phone and displayed in a descending order of visual similarity. That is, the most similar painting is at the top-left (e.g. The experiment adopted a within-subjects design. The independent variable is 'Interface Type' (five Apps), and the dependent variables include performance measures and self-rated satisfaction and interface usability obtained from questionnaires. Partici pants. Forty-one college students (mean age 19 years, s.d. 1.88) have volunteered to take part in the experiment. All participants reported having experience of using touchscreen mobile phone and none of them self-indicated as drawing-expert or artist. All participants were novice users of our Apps and right-handed.

Material and Sti muli. Two Google Nexus 4 (133.9 mm × 68.7 mm) mob ile phones with Android 4.4 operating system were used: One mobile phone was used to present the reference painting to participants; and five Apps and SCR Screen Recorder Pro were installed on the other phone on which participants would draw. Both phones were adjusted to the same level of brightness to reduce the device difference effects. The choice of painting as stimulus followed the fo llo wing rationales: (1) The painting is not well-known; [START_REF] Jacobs | Fast M ultiresolution Image Querying[END_REF] The painting has at least six rough color blocks, wh ich contains the color widely used in literature and agreed as the basic colors [START_REF] Douglas | M odel and Representation: The Effect of Visual Feedback on Human Performance in a Color Picker Interface[END_REF], e.g., blue, red, and green etc. We believe the painting Old man (shown in Fig. 3) is sufficient to evaluate user's color selection performance with at least seven gist color blocks.

Procedure

Only one participant involved in study each time. Part icipant was briefly informed of the aim and procedure of our study, then he/she was instructed to complete the pretask questionnaire before a tutorial on how to use our mobile Apps. After a pract ice session (for eliminating indiv idual d ifference effects), each participant was asked to complete the copying task (copying paint ing Old man shown in Fig. 4) by use of five Apps in a rando m order to minimize the practice effects . One d rawing was required on each App. We encouraged participants to draw the rough color sketches with accurately tracing colo r and proportion of the original painting. At the end of each drawing session, participant was asked to rate on the usability with 7-point Likert scale in terms of the drawing speed, the level of ease-of-use, the similarity of his/her drawing compared to the reference painting, and the overall preference on the color selection interface.

After complet ing all tasks participants were interviewed to specify which interface they believed to be their favorite one among five Apps, and also the preferred squarebased color palette and wheel-based color palette. The time spent on drawing was captured by SCR Screen Recorder Pro. And the position of Old man in search results for each query was recorded by the experimenter. All p rocedures were v ideorecorded.

5

Results and Discussions 206 drawn-queries were produced with relevant retrieval results and drawing-videos recorded. The time taken to comp lete a d rawing, the retrieval result, and the self-rated satisfaction with both drawing process and color selection interfaces were analyzed.

Participants spent the shortest time on V-HSwheel (mean=4m31s) to co mp lete one drawing, followed by Vring-HSwheel (mean=4m47s), and the longest time taken on V-HSsquare (mean=5m46s). A paired samp les t-test revealed that the difference be-tween V-HSwheel and V-HSsquare was highly significant (t(40)=-2.786, p=.008), whereas no significant difference was found between V-HSw heel and Vring-HSwheel (t(40)=-1.089, p =.283). A repeated measures ANOVA with a Greenhouse-Geisser correction indicated that there were no significant differences (F(1.620, 64.816)=0.197, p =.775) among the time taken to co mplete one drawing on H-SVsquare, S-HVsquare and V-HSsquare. We also found that the time spent on drawing a query gradually decreased as participants' experience gained.

Fig. 4 illustrates the proportion of drawings on each App that got the Old man in top 51 results. The V-HSsquare worked best (71.4% drawings succeeded) among five Apps, and V-HSwheel (64.5%) was better than Vring-HSwheel (58.8%). Regarding to the usability of colo r palette interfaces and the user preference, user's self-rated Likert-type scales were co llected fro m questionnaire, were statistically analyzed by Fried man tests and Wilco xon tests with a significant level of 0.01. The Fried man tests were carried out for the comparison C1. There were no statistically significant differences in the interface usability (first seven feature rows illustrated in Table 1) and the preference among three HSV axes comb ination interfaces (H-SVsquare vs. S-HVsquare vs. V-HSsquare), as shown by column C1 in Table 1. Fo r participants' attitudes to drawing process and preference, the Wilcoxon's singed rank tests indicated that V-HSwheel was highly significantly better than Vring-HSwheel in terms of participants' satisfaction rankings on Relaxing-to-draw (Z=-3.123, p=.002), Quick -to-draw (Z=-2.701, p=.007), Easy-to-draw (Z=-3.257, p=.001), similarity of drawing (Z=-2.853, p=.004), and easy-to-pick wanted color (Z=-2.755, p=.006). While co mparing V-HSwheel and V-HSsquare, a highly significant difference (Z=-2.644, p=.008) was only found in the ratings of enough colorchoices, that is, participants thought that V-HSw heel provided mo re co lor choices than V-HSsquare. But in fact, both interfaces supply the same nu mber o f colo r choices.

There were no statistically significant differences in participants' ratings on inte rface preference. However, fro m answers to question about which interface was preferred in each group of bar-square and wheel-based color palettes, most people chose H-SVsquare as their favorite bar-square color p icker, and Vring-HSwheel as their favorite wheel-based color palette. And they preferred H-SVsquare color interface overall.

It is found that each drawing contains 6~8 colors on average. A paired samp les ttest revealed that the difference between the number of colors used for drawing on V_HSwheel (mean 6.87) and V_HSsquare (mean 7.46) was high ly significant (t(40)= -3.438, p=.001).

It is observed that how participants draw varied significantly, however, several common issues of drawing behavior emerged. First, participants with high-level drawing skills, e.g. sketching or o il painting, focused on small details of painting du ring the first drawing session. For examp le, Participant #16 and #20 drew both outline of the coat and face details e.g. eyes, eyeglasses, and mouth etc., as shown in Fig. 5. Second, most of users followed the same drawing style as that of using pencil-andpaper, that is, a) sketching an outline firstly and f illing colors afterwards, e.g. the screenshot of the first sketching process of Participant #32 as shown in Fig. 5, right; b) Rubbing out the mistakes and then drawing the correct one rather than drawing over the mistakes directly.

Conclusions

In this paper, we p roposed a method of categorizing the continuous color palette interfaces into two classes: '1D-1D-1D' interface and '1D-2D'. By focusing on '1D-2D' interface with HSV color model, an experiment (with 41 part icipants) was designed and carried out to explo re the influence of both comb inations of the three axes of HSV co lor model and shape representing these combinations on users' query-drawing performance. Specifically, we co mpared three groups of color selection interfaces: three bar-square color palettes, bar-wheel vs. ring-wheel, and bar-wheel vs. barsquare. The experimental results suggest that shape has more effects on users' pe rformance than color model axes co mbinat ions. In general, wheel-based interface leads to faster drawing than the square-based one; and the usability rating on ring-wheel is higher than that on bar-wheel. Most of participants preferred the H-SV square-bar colour selection interface.

One limitat ion of our study is conducting the copying-task only. However, the fact that our reference painting contains 7 different color blocks makes this preliminary study reliable. We will do more studies on other types of tasks , e.g. landscape drawing.

  built five Client/Server-based drawing Apps on Android phone, and each App provided the same basic drawing tools except the color p icker. The basic drawing tools include brushes with five different sizes (10, 20, 30, 40 and 60 pixels), 'Brush color picker', erasers with five different sizes (10, 30, 40, 50 and 70 pixels), 'Canvas color picker', 'Undo', 'Redo', and 'Trash' on the top of canvas (see Fig. 1, left ). We separated brush color and canvas color to facilitate user to create and refine drawings easily, and the same color selection interface is used for 'Brush color picker' and 'Canvas color picker' of each App. Each color palette interface could keep 20 recently used colors.

Fig. 1 .

 1 Fig. 1. The screenshots of drawing and result-viewing interface of our Query-by-Drawing App.

Fig. 1 ,Fig. 2 .

 12 Fig. 2. Five color palette interfaces: (a) H-SVsquare, (b) S-HVsquare, (c) V-HSsquare, (d) V-HSwheel, (e) Vring-HSwheel4.1 Experimental DesignBased on the results of reviewing 140 drawing Apps, 22 co loring Apps, and 18 photoediting Apps available on Google Play store, we adopted four co mmonly used shapes of color palette interface, includ ing square, wheel, ring and bar. The square and wheel were used for displaying the '2D' co mbinations, and ring and bar were for the '1D'.The bar-square interface is explo red to answer the first research question. For '1D-2D' interfaces, there are three co mbinations of HSV color model: H-SV, S-HV, and V-HS. The '2D' is displayed as a square, and '1D' is controlled by a vert ical slider bar. These three co mbinations were named as H-SVsquare, S-HVsquare, and V-HSsquare, as shown in Fig.2, and the comparison of these three interfaces was comparison 'C1'.As for the second research question, firstly, we built a slider bar and a ring of V (Value) with HS color wheel, which were formed into V-HSwheel and Vring-HSwheel as shown in Fig.2. We defined comparison 'C2' as V-HSw heel vs . Vring-HSw heel. Secondly, we were also interested in looking at two different representations of HS with slider bar: square vs. wheel, that is, comparison 'C3': V-HSsquare vs . V-HSwheel.The experiment adopted a within-subjects design. The independent variable is 'Interface Type' (five Apps), and the dependent variables include performance measures and self-rated satisfaction and interface usability obtained from questionnaires. Partici pants. Forty-one college students (mean age 19 years, s.d. 1.88) have volunteered to take part in the experiment. All participants reported having experience of using touchscreen mobile phone and none of them self-indicated as drawing-expert or artist. All participants were novice users of our Apps and right-handed.

Fig. 3 .

 3 Fig. 3. The reference painting Old man used for copying task.

Fig. 4 .

 4 Fig. 4. The Percentage of drawings that get the Old man on top 51 result lists.

Fig. 5 .

 5 Fig. 5. First drawings from Participant #16, #20, #32.

Table 1 .

 1 The statistics analysis of self-rated usability, user preference on color palette interface.

	Features\Comparison	C1	C2	C3
	Relaxing-to-draw	χ 2 (2)=2.048,	Z=-3.123,	Z=-2.263,
		p=.359	p=.002	p=.024
	Quick-to-draw	χ 2 (2)=0.290,	Z=-2.701,	Z=-1.995,
	Easy-to-draw	p=.865 χ 2 (2)=0.587,	p=.007 Z=-3.257,	p=.046 Z=-1.498,
		p=.746	p=.001	p=.134
	Level of similarity of query to the origin	χ 2 (2)=1.355,	Z=-2.853,	Z=-0.190,
	painting	p=.508	p=.004	p=.849
	Enough color choices	χ 2 (2)=5.291, p=.086	Z=-1.034, p=.301	Z=-2.644, p=.008
	Easy-to-pick the wanted color	χ 2 (2)=2.947,	Z=-2.755,	Z=-2.274,
		p=.229	p=.006	p=.023
	Quick-to-get the wanted color	χ 2 (2)=0.938,	Z=-1.523,	Z=-2.543,
		p=.625	p=.128	p=.011
	Color palette preference	χ 2 (2)=5.291, p=.071	Z=-0.485, p=.628	Z=-1.812, p=.070

http://www.bbc.co.uk/arts/yourpaintings/


Mohammad Alshammari
email: m.t.m.alshammari@cs.bham.ac.uk

Rachid Anane
email: r.anane@coventry.ac.uk

Robert J Hendley
email: r.j.hendley@cs.bham.ac.uk

Design and Usability Evaluation of Adaptive E-learning Systems based on Learner Knowledge and Learning Style

Keywords: usability, adaptivity, learning style, e-learning, experimentation

Designing effective adap tive e-learning systems, from a usability perspective, represents a challenge because of the complexity of adaptivity in order to meet the diverse requirements of learners. Furthermore, there is a lack of well-designed experimental evaluation of adaptive e-learning systems in general, and of their usability in particular. The aim of this paper is the presentation of an adaptive e-learning system based on learner knowledge and learning style, and of the results of an initial experimental evaluation of the usability of its two modes of operation. This involves comparing the usability of an adaptive version of the system with the usability of a non-adaptive version, in a learning environment with 75 participants. The experiment produced significant results; they indicate that an adaptive e-learning system based on learner knowledge and learning style has a higher level of perceived usability than a non-adaptive e-learning system. This may also increase the level of satisfaction, engagement and motivation of learners and therefore enhance their learning.

Introduction

Designing effective adaptive systems, from a usability perspective, is seen as a challenging task [START_REF] Gena | Usability engineering for the adaptive web[END_REF]. Adaptive e-learning systems tailor instructional material to the learner's needs by, for instance, providing personalized learning paths, changing the interface layout or hiding some material links [START_REF] Brusilovsky | User models for adaptive hypermedia and adaptive educational systems[END_REF][START_REF] Oppermann | Adaptability and adaptivity in learning systems[END_REF][START_REF] Akbulut | Adaptive educational hypermedia accommodating learning styles: A content analysis of publications from 2000 to 2011[END_REF]. Meeting the learner's requirements, providing relevant instructional material and supporting the learner-system interaction goals are increasingly important concerns in e-learning systems [START_REF] Brusilovsky | User models for adaptive hypermedia and adaptive educational systems[END_REF][START_REF] Oppermann | Adaptability and adaptivity in learning systems[END_REF]. Adaptive systems may, however, violate standard usability principles such as consistency, privacy and learner controllability [START_REF] Gena | Usability engineering for the adaptive web[END_REF][START_REF] Höök | Steps to take before intelligent user interfaces become real[END_REF]. Eliminating the negative effects on usability is an essential part of the iterative design process of adaptive systems [START_REF] Jameson | Adaptive interfaces and agents[END_REF]. It is argued that if an e-learning system is not sufficiently usable, learners become frustrated and focus on the system rather than on the learning content [START_REF] Ardito | An approach to usability evaluation of e-learning applications[END_REF]. Furthermore, an e-learning system may be usable in terms of its usage but not in terms of the pedagogical perspective. This issue, therefore, may lead to less effective and less efficient learning with these systems. Usability represents a challenge that should be taken into account when designing and evaluating adaptive e-learning systems [START_REF] Höök | Steps to take before intelligent user interfaces become real[END_REF][START_REF] Benyon | Adaptive systems: a solution to usability problems[END_REF]. There is a requirement for a better understanding of where adaptation in e-learning systems is useful, and where it is harmful [START_REF] Ardito | An approach to usability evaluation of e-learning applications[END_REF].

Although there are many adaptive e-learning systems that have been designed and implemented, they suffer from a lack of experimental evaluation in general [START_REF] Akbulut | Adaptive educational hypermedia accommodating learning styles: A content analysis of publications from 2000 to 2011[END_REF]. More particularly, usability evaluation is not usually considered as one of the main criteria in the iterative design process of these systems or in determining their ease of use. According to Zaharias "very little has been done to critically examine the usability of e-learning applications" [START_REF] Zaharias | Developing a usability evaluation method for e-learning applications: Beyond functional usability[END_REF]. It is not always clear how easy and pleasant an adaptive e-learning system is to use.

This paper aims to fill a research gap in understanding the usability of adaptive elearning systems. An adaptive e-learning system, based on knowledge level and learning style, was designed and implemented. It provides personalised learning paths by organising material links according to their relevance to a particular learner; it also provides adaptive guidance and feedback to support learner-system interaction goals. Using a standard usability instrument an experimental evaluation, concerning learners' perception of usability, was conducted to compare the adaptive e-learning system with a non-adaptive version. The main aim is to determine whether adaptivity influences perceived usability given the fact that both systems have the same interface layout. It is expected that a highly usable e-learning system will increase the satisfaction, engagement and motivation of learners, and therefore, enhance their learning [START_REF] Ardito | An approach to usability evaluation of e-learning applications[END_REF][START_REF] Zaharias | Developing a usability evaluation method for e-learning applications: Beyond functional usability[END_REF]. This study is focused on usability. The evaluation of the learning outcome and learner satisfaction in using the system has been reported elsewhere [START_REF] Alshammari | An E-Learning Investigation into Learning Style Adaptivity[END_REF].

The paper is organised as follows. Section 2 presents related work. Section 3 describes the implemented adaptive e-learning system. Section 4 provides the usability evaluation method. Section 5 discusses the usability results, the main usability issues, identifies its limitations and points to future work. Section 6 concludes the paper.

Related Work

Adaptivity is an approach designed to meet the different needs of different learners when it is incorporated in e-learning systems [START_REF] Brusilovsky | User models for adaptive hypermedia and adaptive educational systems[END_REF][START_REF] Oppermann | Adaptability and adaptivity in learning systems[END_REF][START_REF] Alshammari | Adaptivity in E-Learning Systems[END_REF]. Many adaptive e-learning systems have been designed and built [START_REF] Brusilovsky | User models for adaptive hypermedia and adaptive educational systems[END_REF][START_REF] Akbulut | Adaptive educational hypermedia accommodating learning styles: A content analysis of publications from 2000 to 2011[END_REF]. For example, the SQL-Tutor is an intelligent e-learning system that customizes the sequence of SQL lessons based on the knowledge level of learners [START_REF]An intelligent SQL tutor on the web[END_REF]. An approach that takes into account the learning style of learners to provide instructional recommendations to learners is represented by the eTeacher system [START_REF] Schiaffino | eTeacher: Providing personalized assistance to elearning students[END_REF]. The Protus system combines knowledge level and learning style to personalize learning material for teaching Java programming [START_REF] Klasnja-M Ilicevic | E-Learning personalization based on hybrid recommendation strategy and learning style identification[END_REF]. Despite some successful systems, the literature also reports some failures [START_REF] Akbulut | Adaptive educational hypermedia accommodating learning styles: A content analysis of publications from 2000 to 2011[END_REF][START_REF] Brown | Evaluating learning style personalization in adaptive systems: Quantitative methods and approaches[END_REF]. Adaptive e-learning systems provide different designs and presentations to meet the diverse requirements of learners; therefore, usability evaluation plays an important role in the early stages of design, in order to help produce better systems. According to Bangor, Kortum, and Miller "it has become clear that a generalized assessment of the usability and customer satisfaction for different delivery types of interfaces is valuable information to have when trying to determine which technology would be best suited for different deployments" [START_REF] Bangor | An empirical evaluation of the system usability scale[END_REF]. This highlights the importance of usabil-ity evaluations of adaptive e-learning systems in particular, since learners encounter different designs and presentations of content adapted to their characteristics (e.g., learning style). In addition, usability investigations should be taken into account to achieve a harmony between the learner, the learning task, the learning context and the e-learning system [START_REF] Benyon | Adaptive systems: a solution to usability problems[END_REF]. An adaptive e-learning system may be usable in terms of its usage but not in terms of the pedagogical perspective [START_REF] Ardito | An approach to usability evaluation of e-learning applications[END_REF].

Usability has a clear connection with learning and also with adaptivity. However, usability evaluation is not usually taken into account; neither when forming the iterative design process of these systems nor in determining how they are easy and pleasant to use [START_REF] Zaharias | Developing a usability evaluation method for e-learning applications: Beyond functional usability[END_REF]. This situation warrants a study of the design and development of an adaptive e-learning systems and of its usability evaluation.

AdaptLearn: An Adaptive E-Learning System

AdaptLearn 1 is an acronym for 'Adaptive Learning' which is an adaptive e-learning system. It takes into account learning style and knowledge level as important learner characteristics in order to construct personalized learning paths and adaptive feedback and guidance. A screenshot of AdaptLearn is presented in Fig. 1. The adaptive elearning framework used as a basis for AdaptLearn, and relevant technical details are presented in our previous work that can be found in [START_REF] Alshammari | An E-Learning Investigation into Learning Style Adaptivity[END_REF]. The main components of AdaptLearn are the domain model, the learner model and the adaptation model. They are briefly described below.

Fig. 1. A screenshot of the AdaptLearn interface

1 AdaptLearn is implemented using NetBeans environment, PHP, JavaScript and M ySQL.

In AdaptLearn, the domain model is structured as a hierarchal network (i.e., a treelike structure) storing knowledge elements related to the application domain. This structure is widely used as a method to representing domain models in related work [START_REF] Papanikolaou | Personalizing the Interaction in a Web-based Educational Hypermedia System: the case of INSPIRE[END_REF]. The learner model incorporates knowledge level and learning style as learner characteristics in order to provide adaptivity. The knowledge level is an important characteristic that should be taken into account in online learning [START_REF] Brusilovsky | User models for adaptive hypermedia and adaptive educational systems[END_REF]; it is initialized by using a pre-test, and maintained based on learner-system interaction, mainly test items (i.e., associated with each knowledge element) as a main source of interaction data. The learning style is also assumed to enhance learning when it is integrated in the learner model [START_REF] Felder | Learning and teaching styles in engineering education[END_REF]. It is initialized by using a learning style questionnaire following the Felder-Silverman model which is considered as a valid and reliable learning style identification tool [START_REF] Felder | Applications, reliability and validity of the index of learning styles[END_REF].

The adaptation model aims at recommending relevant instructional material to learners to support their interaction goals. It uses the information stored in both the learner model and the domain model in order to provide adaptivity. The adaptation model provides two main adaptive methods including: personalized learning paths and adaptive guidance. The output of the adaptation model is transferred to the interaction model (i.e., the interface).

Personalized learning paths are constructed and continually updated for individual learners based upon their knowledge level and learning style. These paths prioritize links to knowledge elements, hide or remove links to the elements which are not yet ready to be studied (according to the current learning state of the learner), and/or generate links to more relevant knowledge elements , as needed. Examples of personalized learning paths (as provided by AdaptLearn) are presented in Fig. 2. The provision, removal and ordering of elements in the recommended learning paths are e xpected to meet the needs of learners by taking into account their learning style and their knowledge level. This helps eliminate the effect of cognitive overload, support their interaction goals and optimize the learning process [START_REF] Oppermann | Adaptability and adaptivity in learning systems[END_REF]. Adaptive guidance is another form of adaptivity. It is integrated to guide learners as they progress through the learning process to successfully accomplish their learning tasks. It is mainly activated when the learner answers some test items that are related to a specific knowledge element. The system may help the learner review and understand a particularly difficult knowledge element by fetching related supplementary material from the domain model and recommending it to the learner. The system may also highlight some important points for the learner to consider. It may advice the learner to revise a specific knowledge element, and provide general feedback about the learning progress.

List of Lessons

Usability Evaluation

This work contributes to the need to compare different types of interface or system technologies in critical application domains such as e-learning [START_REF] Bangor | An empirical evaluation of the system usability scale[END_REF]. The learners' perception of usability, using a standard usability instrument, is measured by conducting a controlled experiment in a computer laboratory. A number of experimental sessions were conducted, and each session lasted for up to 2 hours. Two experimental conditions were proposed: adaptive condition and non-adaptive condition. In the adaptive condition, a group of subjects interact with an adaptive version of the system. In the non-adaptive condition, another group of subjects interact with the same system but without adaptivity, and a fixed learning path is provided. The main hypothesis that is put forward for this study is that: an adaptive e-learning system based on learner knowledge and learning style has a higher level of perceived usability than a non-adapti ve e-learning system.

In both conditions, the same interface and learning material were used, and the experiment was completed within an equivalent timeframe. Learning material is related to computer security (i.e., the application domain) covering the topics of private-key encryption, public-key encryption and key exchange protocols. Each topic has a number of interrelated knowledge elements. Participants were asked to complete all the lessons provided by the system and to precisely follow its recommendations. The main distinction between the two conditions is the provision of adaptivity.

With regards to the experimental procedure, the participants introduced to the main objectives of the experiment, the system features and informed of the procedure. Second, the participants were asked to access the system via an Internet browser. Then, the participants registered with the system, completed some personal information (e.g., username, age and gender), the Index of Leaning Style (ILS)2 questionnaire based on the Felder-Silverman model [START_REF] Felder | Applications, reliability and validity of the index of learning styles[END_REF] and completed a pre-test containing 22 multiple-choice questions to initialize the learner model. Once completed, the system randomly assigned the participant to a specific study condition: the adaptive condition or the non-adaptive condition. The system then directed the learner to the main page to study the learning material. At the end of the learning process, the participant completed the System Usability Scale (SUS) questionnaire [START_REF] Brooke | SUS-A quick and dirty usability scale[END_REF]. This tool is a quick, reliable and widely used test of system usability [START_REF] Tullis | A comparison of questionnaires for assessing website usability[END_REF]. SUS is a 10 item questionnaire with 5-point Likert scale with anchors ranging from "strongly disagree" to "strongly agree".

Results and Discussion

The experiment was successfully completed by 75 (57% male, 42% female) participants. They were undergraduate students in a computer science degree program. The mean age was 22.21 (SD=3.13). The experimental sample involved 39 participants in the adaptive condition and 36 participants in the non-adaptive condition.

The usability of the adaptive e-learning system and the non-adaptive system was measured and compared. The two systems (i.e., the adaptive and non -adaptive systems) in the experiment are based on the same interface layout and learning material with an important distinction of the provision of adaptivity. The usability scores for the adaptive system (M=79.46) and the non-adaptive system (M=71) were and acceptable in general. This implies that both systems are useful and valuable in learning, and the learners found them easy to use. However, the two systems were co mpared to get deeper insight into their usability, and whether the provision of adaptivity has any impact. Usability may lead to satisfaction, engagement and motivation of learners, and therefore, highly usable systems are expected to improve learning [START_REF] Ardito | An approach to usability evaluation of e-learning applications[END_REF][START_REF] Zaharias | Developing a usability evaluation method for e-learning applications: Beyond functional usability[END_REF].

As there was homogeneity of variance between the study groups as assessed by Levene's Test for Equality of Variances, F=.07, P=.79 and data were normally distributed, an independent sample t-test was conducted in order to compare the two systems using an alpha level (α) of 0.01. It was found that there is a statistically significant difference between the general usability score of the adaptive system (M=79.46, SD=13.14) and the non-adaptive system (M=71, SD=13.67), t(73)=2.73, P= 0.008. The effect size, which provides an objective measure of how important the effect is, was between medium and large (d= 0.63). The hypothesis is therefore confirmed, and it can be inferred that the adaptive-learning system based on learner knowledge and learning style has a higher level of perceived usability than the non-adaptive elearning system.

The experiment, however, was based on two extreme conditions, adaptive and nonadaptive systems; different levels of adaptivity including controllability should be compared with more experimental conditions to get deeper understanding of their effect. However, designing such experiments to evaluate their usability and learning effectiveness represents a significant challenge [START_REF] Brown | Evaluating learning style personalization in adaptive systems: Quantitative methods and approaches[END_REF].

In an investigation of the SUS tool items, the most significant difference is that participants would use the adaptive e-learning system more frequently than the nonadaptive system in response to the item "I think I would like to use this website frequently". This is a very important point in usability of systems, which may provide a justification for the adaptivity and recommendation mechanisms provided by the adaptive system. Adaptivity may influence the learners to perceive that the system would help them when needed, provide them with dynamic support according to their characteristics. The recommendations of the system may enhance their intellectual curiosity, satisfaction, and engagement. In contrast, learners may find the nonadaptive system rigid and unresponsive to their needs. They may be less encouraged to use the non-adaptive system as a tool for learning.

Another significant point in relation to the item "I thought this website was easy to use" is that, participants find the adaptive system, in the long run, easier to use than the non-adaptive system. This is a surprising finding because of the inherited co mplexity of adaptivity. It was expected that the usage of adaptive systems would not be easier than the usage of traditional or non-adaptive systems. Learners may find it more helpful and easier to use a system that provides personalized feedback and recommendations based on their system interaction. For example, the adaptive system suggests to learners what to do if things go wrong, what to do next or what the current situation of the learning process. It may be the case that once learners gain an appreciation of the adaptive system, they may find it easier to use and more useful.

However, the non-adaptive system may seem to be better than the adaptive system in relation to system learnability. This finding was based on the response the item "I think I would need the support of a technical person to be able to use this system". It may imply that the non-adaptive system has less problematic issues when it comes to learning how to use the system very quickly. Another point is that, learners who use the adaptive system are more likely to need technical support to be able to initially use the system. Although this finding is not significant, it should be taken into account when designing adaptive e-learning systems.

Although the adaptive and the non-adaptive e-learning systems had the same interface layout, significant results related to the usability of the adaptiv e system were generated. It is observed that high level of adaptation in e-learning systems enhances usability. The high level of perceived usability may lead learners to be more satisfied, engaged and motivated to use the adaptive e-learning system [START_REF] Ardito | An approach to usability evaluation of e-learning applications[END_REF][START_REF] Zaharias | Developing a usability evaluation method for e-learning applications: Beyond functional usability[END_REF]. It is expected that highly usable adaptive e-learning systems improve learning.

The findings were based on a short-term study with a relatively small number of participants and with few learning resources . With regards to the SUS tool, it was used to test general usability; there is a need for a more specific tool that examines the usability of adaptive e-learning systems in particular. Such a tool would highlight specific usability issues related to adaptivity which will help in designing better systems. In order to improve the adaptive e-learning system, other important factors such as cognitive load, metacognitive skills and affective state may be integrated. Controllability and transparency between the learner and the system can also be investigated.

Conclusion

This paper has presented an adaptive e-learning system that was designed and built based on knowledge and learning style to support learner-system interaction goals. An initial experimental evaluation of the system usability was conducted with 75 participants in a learning environment; it produced significant results. They indicate that the adaptive-learning system based on learner knowledge and learning style has a higher level of perceived usability than a non-adaptive e-learning system. As usability has an influence on the learner's satisfaction, engagement and motivation when using elearning systems, learning enhancement is expected when the system is highly usable. The experiment was based on a short-term study with relatively small number of participants and with few learning resources incorporated in the system. A long -term evaluation with more participants is desirable in future experiments. An experimental evaluation is also being undertaken that builds empirically on the finding of this experiment to investigate learning effectiveness , efficiency and usability when controllability and different levels of adaptivity are provided.

Fig. 2 .

 2 Fig. 2. Examples of personalized learning paths for different learners.

List of Lessons List of Lessons List of Lessons Symmetric cipher Symmetric cipher Key exchange Key exchange

	Example	Math. Notation	Example	Concept
	Practical tool	Example	Concept	Math. Notation
	Concept	Practical tool	Math. Notation	Example
	Math. Notation

https://www.engr.ncsu.edu/learningstyles/ilsweb.html


Xiaojun Yuan
email: xyuan@albany.edu

Ning Sa

Grace Begany
email: gbegany@albany.edu

Huahai Yang
email: hyang@juji-inc.com

What Users Prefer and Why: A User Study on Effective Presentation Styles of Opinion Summarization

Keywords: Text summarization, consumer decision making, user studies, user interface design

Opinion Summarization research addresses how to help people in making appropriate decisions in an effective way. This paper aims to help users in their decision-making by providing them effective opinion presentation styles. We carried out two phases of experiments to systematically compare usefulness of different types of opinion summarization techniques. In the first crowd-sourced study, we recruited 46 turkers to generate high quality summary information. This first phase generated four styles of summaries: Tag Clouds, Aspect Oriented Sentiments, Paragraph Summary and Group Sample. In the follow-up second phase, 34 participants tested the four styles in a card sorting experiment. Each participant was given 32 cards with 8 per presentation styles and completed the task of grouping the cards into five categories in terms of the usefulness of the cards. Results indicated that participants preferred Aspect Oriented Sentiments the most and Tag cloud the least. Implications and hypotheses are discussed.

Introduction

The widespread use of the Internet in many aspects of human activities has resulted in an abundance of publicly-accessible opinions. People can find opin ions on a variety of topics in venues such as Twitter, Weibo, foru ms, e-co mmerce sites and specialized opinion-hosting sites such as Yelp. While most of these opinions are intended to be helpful for others, the sheer quantity of them often makes most of the opinions underutilized, as the information overload overwhelms many potential consumers. For example, A mazon.com has more than 18,000 reviews for Kindle Fire, a single pro duct alone. Summarizing these reviews in some concise form could bring enormous benefits to consumers and business alike. Not surprisingly, research on opinion su mmarizat ion is gaining increased attention [14, 18, 19, 22, 24,]. Ho wever, most of the research emphasizes technical advances in underlying algorithms, wh ile paying less attention to the presentation of the results, which is the focus of this work. Correspondingly, evaluation of opin ion summarization research is normally based on ce rtain notions of precision and recall calculat ion co mmonly used in information retrieval [START_REF] Salton | Introduction to modern information retrieval[END_REF] and data mining [START_REF] Witten | Data M ining: Practical machine learning tools and techniques[END_REF]. Studies have only begun to investigate the effectiveness of opinion summarization in term o f usability (e.g. [START_REF] Yatani | Review spotlight: a user interface for summarizing user-generated reviews using adjective-noun word pairs[END_REF]). Such studies focus on tes ting the newly-proposed techniques. A systematic comparison of the usefulness of different types of opinion summarizat ion is still lacking. Th is paper reports our effort in addressing this deficiency.

One major difficulty with studying the effectiveness of opinion summarizat ion is a confounding effect between content effectiveness and presentation effectiveness. It is often not clear whether a technique's empirical superiority can be attributed to its superior text analytics quality or its effective informat ion presentation style. We plan to isolate the two factors and focus on studying the effect of presentation styles. This goal is achieved by using human-generated summarization as the content, so as to ensure the content has consistent high quality regardless of the presentation styles. We can then vary the presentation styles of the summaries to investigate their effect on the usefulness ratings of the summaries. Any differences found between the usefu lness ratings of the summaries can be safely attributed to the differences in present ation styles. We identified four types of presentation styles of opinion summarization through a crowd-sourcing study on Amazon Mechanical Turk, and then conducted a lab user-centered experiment to compare the effectiveness of the four styles.

Previous work

Although not abundant, studies investigating the effectiven ess of opinion summarization fro m a perspective of both usability and user preference are emerg ing. Several recent studies explore feedback fro m users regarding their preferences for certain opinion summarization styles and approaches. Most recently, Qazi et al. [START_REF] Qazi | A preliminary investigation of user perception and behavioral intention for different review types: Customers and designers perspective[END_REF] addressed a gap in existing studies examining the determination of useful opinion review types from customers and designers perspectives. Specifically, the researchers used the Technology Acceptance Model (TAM) as a lens to analyze users' perce ptions toward different opinion review types and online review systems. The study, a pilot study, focused on three review types which are related to perceived usefulness, perceived ease of use, and behavioral intention: A (regular), B (co mparative), and C (suggestive). Suggestive reviews, the speech acts which are used to direct someone to do something in the form of a suggestion, were newly identified by the researchers as a third innovative review type. To examine user perspectives, researchers used a closed card sorting approach to analyze reviews from A mazon, blogs, and a selfdeployed website. The results of their work indicated that the review types play a significant ro le in developing user perception regarding a new product or system, with suggestive reviews more significant for both customers and designers to find mo re usefulness that ultimately improves their satisfaction level.

Further, in another work [START_REF] Yatani | Review spotlight: a user interface for summarizing user-generated reviews using adjective-noun word pairs[END_REF], researchers conducted a user study of a review summarization interface they created called "Review Spotlight." Review Spotlight is based on a tag cloud and uses adjective-noun word pairs to provide an overview of online restaurant reviews. Findings indicated that study participants could form d etailed imp ressions about restaurants and make faster decisions between two options with Review Spotlight versus traditional review webpages. In a large-scale, co mprehensive human evaluation of three opinion-based summarization models -Sentiment Match (SM), Sentiment Match + Aspect Coverage (SMAC), and Sen timent-Aspect Match (SAM) -Lerman, Blair-Go ldensohn and McDonald [START_REF] Lerman | Sentiment summarization: Evaluating and learning user preferences[END_REF] found that users have a strong preference for sentiment-info rmed summaries over simp le, non-sentiment baselines. This finding reinforces the usefulness of modeling sentiments and aspects in opinion summarization. In another study, Lerman and McDonald [16] investigated contrastive versus single-product summarization of consumer electronics and found a significant improvement in the usefulness of contrastive summaries versus summaries generated by single-product opinion summarizers. To find out which visual properties influence people viewing tag clouds, Bateman, Gutwin and Nacenta [START_REF] Bateman | Seeing things in the clouds: the effect of visual features on tag cloud selections[END_REF] conducted an exploratory study that asked participants to select tags from clouds that manipulated nine visual propert ies (font size, tag area, nu mber of characters, tag width, font weight, color, intensity, and number of pixels). Participants were asked to choose tags they felt were "v isually important" and results were used to determine wh ich visual properties most captured people's attention. Study results indicated that font size and font weight have stronger effects than intensity, number of characters or tag area. Ho wever, when several visual properties were manipulated at one time, no one visual property stood out among the others. Carenin i, Ng and Pauls [START_REF] Carenini | M ulti-document summarization of evaluative text[END_REF] also employed a user study as part of their wider co mparison of a sentence extract ion -based versus a language generation-based summarizer for summarizing evaluative text . In their quantitative data analysis, the researchers found that both approaches performed equally well. Qualitative data analysis also indicated that both approaches performed well, however, for d ifferent, co mplementary reasons. In a related work, Carenin i, Ng and Pauls [START_REF] Carenini | Interactive multimedia summaries of evaluative text[END_REF] examined the use of an interactive mult imedia interface, called "Treemaps," for summarizing evaluative text o f online reviews of consumer electro nics. Treemaps presents the opinion summarizations as an interactive visualization along with a natural language summary. Results of their user study showed that pa rticipants were generally satisfied with the interface and found the Treemap summar ization approach intuitive and informative.

In more recent work, [START_REF] Hu | OpinionBlocks: A crowdpowered, self-improving interactive visual analytic system for understanding opinion text[END_REF] researchers presented a novel interactive visual text analytic system called, "Opin ionBlocks." Opinion Blocks had two key design goals: (1) automated creation of an aspect-based visual summary to support users' real-world opinion and analysis tasks, and (2) support of user corrections of system text analytic errors to imp rove system quality over time. To demonstrate OpinionBlock's s uccess in addressing the design goals, researchers employed two crowd -sourced studies on Amazon Mechanical Turk, According to their results, over 70% of users successfully accomplished non-trivial opin ion analysis tasks using OpinionBlocks. Additionally, the study revealed that users are not only willing to use Opinion Blocks to correct text classification mistakes, but that their corrections also produce high quality results. For example, study participants successfully identified numerous errors and their aggregated corrections achieved 89% accuracy.

Additionally, Duan et al. [START_REF] Duan | VISA: A Visual sentiment analysis sy stem[END_REF], introduced the opinion mining system, "VISA" (VIsual Sentiment Analysis), (derived fro m an earlier system called TIA RA). The VISA system employs a novel sentiment data model to support finer-grained sentiment analysis, at the core of which is the "sentiment tuple," composed of four elements: feature, aspect, opinion, and polarity. Researchers conducted a user experiment to explore how efficiently people could learn to use VISA an d demonstrate its effectiveness. Study results indicated that VISA performed significantly better than the two comparison tools (TripAdvisor and a text edit tool) due to its key features, namely mash-up visualizations and rich interaction features.

In the current study, investigation of user perspectives on opinion summarization styles is taken further with the evaluation and comparison of four distinct, popular summarization styles focused on textual opinions; namely, Tag Clouds, Aspect -Oriented Sentiments, Paragraph Summaries, and Group Samples .

Following we introduce the four presentation styles used in our study, the metho dology, results of the experiment, discussion and conclusions.

Opinion summarization presentation styles

Some opinion hosting sites allow opinion writers to give nu merical rat ings in addition to the textual opinions. Since the visualization of numerical values is a well -studied problem, we focus instead on the summarizat ion of textual opin ions. Similarly, we do not compare visualizat ion systems that emphasize statistics rather than the textual content of the text co llect ions (e.g. [START_REF] Chen | Visual analysis of conflicting opinions[END_REF]). Opin ion summarizations studied here are of the kind that could potentially be used in place of the full documents.

Based on our survey of the literature, we have categorized the presentation styles of such opinion summarization into four major types.

Tag clouds (TAG)

Tag clouds are perhaps the most popular form of summarization on the Internet today [START_REF] Brooks | Improved annotation of the blogosphere via autotagging and hierarchical clustering[END_REF]. This type of text presentation has also been used extensively in research (e.g. [START_REF] Potthast | Opinion summarization of web comments[END_REF][START_REF] Salton | Introduction to modern information retrieval[END_REF]). They consist of enumerations of the most common words or phrases in a colle ction of opinions, laid out on a 2D canvas. The size of the words or phrases often indicates how frequently they were mentioned. The larger the word or phrase, the more frequently it received mentions. The effect of various visual features of the tag clouds on their effectiveness have been investigated [START_REF] Bateman | Seeing things in the clouds: the effect of visual features on tag cloud selections[END_REF], but the comparison with other styles of summarization has yet to be done. See Figure 1.

Aspect oriented sentiments (ASP)

Aspect oriented sentiment summarization is an active area of research in text mining [START_REF] Hu | M ining and summarizing customer reviews[END_REF][START_REF] Jo | Aspect and sentiment unification model for online review analysis[END_REF][START_REF] Ku | Opinion extraction, summarization and tracking in news and blog corpora[END_REF][START_REF] Lu | Rated aspect summarization of short comments[END_REF][START_REF] Liu | aspect extraction through semi-supervised modeling[END_REF]. In this approach, some important aspects or topics (also known as features) of opinions are ext racted fro m an opin ion collection. Sentiment orientation of the text snippets containing the aspects are then estimated and summary statistics reported. A typical summarizat ion for one aspect might look like this: for a collection of reviews on Kindle Fire, "screen" is identified as an aspect, and 100 text snippets in the collection are found to be about this aspect, 60 of them have positive sentiment, 30 of them are negative, and the rest are neutral. Representative text snippets for each sentiment orientation may also be provided. See Figure 2.

Fig. 2. Aspect Oriented Sentiments

Paragraph summaries (PRG)

Automatic text summarization systems traditionally produce short passages of text as summary [START_REF] Hovy | Automated text summarization and the summarist system[END_REF][START_REF] Radev | Generating natural language summaries from multiple online sources[END_REF]. The summarizat ion is called ext ractive when the sentences are selected from original documents; abstractive when the sentences are generated by the system [START_REF] Erkan | Graph-based lexical centrality as salience in text summarization[END_REF]. Regard less of the approach, the output could be a readable abstract that resembles what hu mans would write for generic purposes in order to emphasize intrinsic properties such as fluency and coverage [START_REF] House | The tipster summac text summarization evaluation[END_REF]. See Figure 3.

Group samples (GRP)

Clustering algorith ms are typically used in post-query user interfaces as a way of organizing retrieved documents [START_REF] Hearst | Reexamining the cluster hypothesis: scatter/gather on retrieval results[END_REF]. It has also been used in summarization, where similar documents are grouped together and the representatives of the groups are displayed [START_REF] Ando | M ulti-document summarization by visualizing topical content[END_REF][START_REF] A˜na-L´opez | M ultidocument summarization: An added value to clustering in interactive retrieval[END_REF]. This approach has shown to be effective in interactive information retrieval [START_REF] Leuski | Evaluating document clustering for interactive information retrieval[END_REF][START_REF] Wu | Using clustering and classification approaches in interactive retrieval[END_REF]. See Figure 4. Group samples are similar to paragraph summaries in form. However, unlike paragraph summaries that are written anew, group samples are directly drawn fro m the original document collection, and retain the most amount of contextual information.

Research questions

We hypothesize that hu mans respond differently to different presentation styles of opinion summaries, and some styles would be more effective in terms of hu man acceptance.

In this two-phase study, we are interested in investigating the following research questions:

1. Will users prefer (or not prefer) a particu lar opin ion summarizat ion style in making judgments about product reviews?

2. What are the reasons that users may p refer (or not to prefer) a part icular opinion summarization style in making judgments about product reviews?

Phase I: crowd sourcing opinion summarization

As mentioned earlier, in order to study the effect of presentation styles alone, we want to ensure the consistently high quality of the summaries. We achieved the goal through leveraging the wisdom of the crowd. Essentially, we elicited four styles of the opinion summaries with the help of Amazon's Mechanical Turk.

Procedure. First, we collected the top 50 reviews for one model of the iRoomba cleaning robot from Amazon. We chose this collection of opinion text because of the relative novelty of the product and the ability for the general public to relate to it. Using a within-subject design, we recruited 46 turkers located in the USA to answer a survey we developed to gather information for generating the opinion summaries from the text collection. In the survey, turkers were first directed to the raw text of the 50 reviews, and asked to read them in full. Then, questions about the reviews were asked. These questions were directly mapped from the information need for the four presentation styles of summaries. All questions were mandatory and were individually validated to ensure the quality of the answers. On average, turkers spent 56.3 minutes on the survey, and each was paid 4 US dollars.

Generating opi nion summaries using Turkers. For Tag clouds, turkers were asked to list five short phrases to characterize the clean ing robot. They were also asked to estimate what percentage of the 50 reviews had opinions consistent with each phrase.

The phrases turkers came up with were remarkably consistent and converged to 38 phrases (phrases with minor variat ions were grouped as one). All 38 phrases were used in the subsequent lab study. The average percentages of turkers' estimations were used in the subsequent lab study to determine the font size of the phrases. A total of 8 Tag clouds were drawn by hand, with each cloud containing 4 or 5 phrases.

Turkers were asked to list three important aspects of the product according to the reviews they read. For each aspect, they were asked to give an estimate of how many reviews mentioned the aspect, as well as the estimated percentage for positive, neutral and negative sentiment towards the aspect. The top 8 most -frequently listed aspects were used in the subsequent study. Again, the averages of the estimations were used in the display of the aspects.

Each turker was also asked to write a summary of all the rev iews so that "consumers who read your summary can make an informed decision on the product, without having to read the reviews themselves". Among the 46 summaries, the top 8 most readable summaries, as agreed by two judges, were used in the lab study.

We asked turkers to identify similar reviews and group them together. They were required to list 3 groups of similar reviews.

When two reviews appeared in the same group once, their similarity measure increase by one. This way, we were able to generate a similarity matrix among the 50 reviews.

Using the matrix as input, we used a hierarchical clustering algorithm to cluster the 50 reviews. Four clusters produced the optimal fit, and the four cluster prototypes were used in the lab study as group samples. In addition, for each cluster, the closest summary to the prototype was also selected, so that there were 8 group samples in total.

Phase II: comparing presentation styles of summary using card sorting technique

The goal of this phase was to co mpare the four presentation styles of opin ion su mmary in a consumer decision making to respond to the two research questions.

Experi ment design. The co mparison of the opinion summaries was conducted as a lab card-sorting task. For each of the four presentation styles (experiment conditions), eight opinion summaries were prepared according to the procedure described in the previous section.

Each opin ion summary was put on a single image of 960 x 720 pixels. Figure 1, Figure 2, Figure 3 and Figure 4 show a sample display for each of the conditions. In total, 4*8=32 image items were placed in the preparation bin in a random order.

The lab experiment was a within-subject design. Each participant's task was to take all 32 image items and place each of them in one of the five category bins. These category bins were defined as "Not at all useful", "So mewhat useful", "Useful", "Very useful" and "Extremely useful". Participants were told to ignore the card order within each category box. Essentially, we asked participants to give a usefulness rating for each opin ion summary. We use such a card-sorting setting in order to record participants' thought process, as they were asked to th ink-aloud wh ile placing the cards.

Partici pants. Thirty-four participants were recruited fro m Un iversity at Albany, half of them were males. All o f the participants stated that they read online reviews reg ularly for making purchase decisions.

Procedure. Each participant was tested individually in a human computer interaction lab in a University campus in the USA. The subjects first filled out a consent form. Next, the subjects completed an entry questionnaire. The participants were then d irected to http://websort.net/ to do the card sorting. After they comp leted the exper iment, they were asked to answer several questions regarding the four presentation styles and their thoughts about the experiment. The whole experimental process was logged by Techsmith Morae 3 software.

Content analysis scheme

To address the earlier mentioned research question(What are the reasons that users may prefer (or not prefer) a part icular opinion su mmarization style in making jud gments about product reviews?) we emp loyed a qualitative content using an open coding approach to analyze the exit interview data of the Phase II experiment.

The content analysis began with a co mprehensive read -through and evaluation of all 34 part icipants' interview transcripts by each of the first three authors , the primary investigator and two doctoral students. In terms of the initial review and several discussions between the authors, a number of themes emerged fro m the interviews that pertained to the reasons of preference towards the presentation styles. Th emes included: Co mprehensiveness (Co mprehensiveness of Information), Time (Time required to read the summary), Organizat ion/categorizat ion (Organizat ion/categorization of the summary's content), Length/Amount (Length/amount of informat ion), Appearance (Appearance of summary content), and Ease of use (Ease of use of summarization style). A coding scheme was designed according to these themes and is shown in the Table 1.

E2UP E2UN E2UU

Ease of use (Ease of use of summarization style) (E2U)

The unit of analysis for the open coding was the individual interview docu ment. Each of the 34 interview documents were independently coded by the three researchers and data collected in an Excel spreadsheet. The average pairwise percent agreement among the 3 coders is 0.81. Along with each code, snippets of supporting text were extracted from the interview data.

Results

User perception of the presentation styles

As initially noted, the goal of this paper is to better understand the relationship between opinion summarization styles and user decision judgment and the underlying reasons. The results are aligned with this goal and are based on a qualitative content analysis of the participant exit interview data.

In the exit interview, the part icipants answered questions relevant to the four presentation styles, including, (1) most helpful/ least helpful, (2) easiest/most difficult, (3) fastest/slowest, (4) most informative/least informative, (5) sacrifice/keep the most, and ( 6) like/dislike. The six questions were the basis for the measurement of user perception of the four presentation styles. The results are displayed in Figure 5.

Participant responses also included their opinions on the usefulness of the summary in general and strategies they used to make the decision, as well as suggestions on system features they best liked, disliked, and thought could be added in the future.

Although "think-aloud" data and additional computer log data was collected using the Morae software, the current paper focuses exclusively on results fro m the exit interview data.

As can be seen from Figure 5, participants felt that paragraph summary was the most helpful, and the most informat ive presentation style while it was also the most difficult and the slowest one. Tag clouds were reported to be the easiest and the fas test to use, but accordingly they were the least informative and the least helpful style and the one disliked by most of the participants. On the other hand, though Aspect Oriented Sentiments didn't get the most votes, they were found to be generally helpful, easy to use, fast, and informat ive and the part icipants liked them the most. Group samples were relatively less helpful, more difficult, and slower. Fig. 5. User perception of the four presentation styles (y axis is the number of participants who voted for the style)

Users' opinion on the presentation styles

We were interested in finding out the major reasons influencing users' preference of the various opinion summarizat ion styles. It appears that "Comprehensiveness", "Organization/Categorization" and "Ease of use" are equally important key reasons affecting users' preference o f the p resentation styles . The "Appearance", "Time", and "Length/Amount" were found to be less important reasons to the participants. Table 2 shows the distribution of reasons coded across participants. To generate the table, each unique code instance was counted for each participant.

We further investigated the distribution of the identified reasons across the four presentation styles. Table 3 shows the top 5 reasons for each style. Tag clouds received positive comments on "Appearance" and "Time" and a nearly equal amount of positive and negative comments on "ease of use." However, most of the participants agreed that they were negative regarding "co mprehensiveness." This finding can e xplain the above-mentioned finding that participants disliked tag clouds the most. As mentioned by participants, "They don't, they don't have details." All o f the top 5 reasons related to aspect oriented sentiments were positive and covered all of the main reason categories except "time." The participants liked them because they "contain negative and neutral and the positive opinions ," were "very convenient or easy to read," and "very clear, brief," among other reasons. This finding correlates with the findings in Figure 5 and also exp lains why the participants liked aspect oriented se ntiments the most. Most participants found the paragraph summary good in terms of "comprehensiveness" and some liked its organization and found the formatting was "…what's most normal for me." But on the other hand, the paragraph summary received negative comments regarding "time" and "length." The participants found it "too long to read" and they needed to "Spend time reading it." Co mpared with the previous three styles, the group sample received far fewer co mments. Some of the participants mentioned that it was good in terms o f "co mprehensiveness" and "organization," but, some d idn't like it because of the "length," "ease of use," and "organization." Typical user comments can be found in Table 3.

Unsurprisingly, in both paragraph summary and the group sample styles, comprehensiveness was the predominant reason in users' preference decision. Specifically, participants claimed that the paragraph summary is "getting a lot of information in a fairly simple package" and the group sample helps them "imagine what if I had that product."

For tag clouds, though the participants liked them because they were "much faster" and "the font size was there for the words ," they agreed that "They don't, they don't give details" and were negative regarding "comprehensiveness." On the contrary, the paragraph summary was long and time-consuming, but most of the part icipants found it provided "a lot of information in a fairly simple package" and was positive in its "comprehensiveness." Overall, aspect oriented sentiments were the best among the four styles. They were brief and, at the same t ime, co mprehensive. The part icipants found them easy to use and liked their appearance and organization.

Discussion

In this paper, we were interested in discovering users' preferences and the reasons affecting their preferences of the representation styles in an opinion summarization card-sorting task.

Results demonstrate that: (1) Aspect oriented sentiments are the most preferred presentation style; (2) co mprehensiveness, time, organization/categorizat ion, length/amount, appearance, and ease of use are the major reasons impacting users' preferences of a presentation style in making decisions in a product review task.

Our results supported the finding reported in [START_REF] Brooks | Improved annotation of the blogosphere via autotagging and hierarchical clustering[END_REF] in that our participants disliked the tag clouds the most. As [START_REF] Brooks | Improved annotation of the blogosphere via autotagging and hierarchical clustering[END_REF] pointed out, "tags are useful for grouping art icles into broad categories, but less effective in indicating the particular content of an article " In our study, the participants acknowledged that tag clouds were the easiest to use because "We can construe what the product was like in very short time,," and the fastest to use because "It's very fast; the tag clouds was much faster." But, in making the decision on the usefulness of the presentation styles, their first priority was the comprehensiveness of the information in the summary presentation. As mentioned by participants, they understood people liked the tag clouds because of the "font size" and "color," but, they disliked them because they don't "give details" and "drive my decision making." This finding raised an important issue here for the design of info rmat ion systems: Ho w can the user interface balance the need for co mprehensiveness of information and the need to provide key features enabling users to quickly grasp the desired information? On the other hand, [START_REF] Brooks | Improved annotation of the blogosphere via autotagging and hierarchical clustering[END_REF] reported that, compared with humanassigned tags, automated tagging produces "more focused, topical clusters ." The tag clouds in our study were generated by the turkers. As a result, the co mprehensiveness of automatically generated tags might be our future research direction .

It was interesting to learn that participants liked the aspect oriented sentiments the mo st. Organization/ categorization is the most critical reason in users' decision ma king relevant to this presentation style. Most importantly, they liked them because they contain "negative and neutral, and the positive opinions," "color," "nu mber" and "percentages."

Our results indicated that there may be a relationship between consumers' informat ion needs and their preference of an opinion summary presentation style. With regards to the opinion summary of a cleaning robot, it is within expectation that consumers may want to look for informat ion about system usability, performance, and reliability. This factor may contribute to the finding that participants prefer aspect oriented sentiments, not tag clouds.

It can also be noticed that there exist biases potentially introduced by manually generated summary presentations. Our summarizat ions were generated by human turkers, but, this generation could have been influenced by the instructions and contents distributed by the researchers.

Results of this study have practical imp licat ions for developers of text summarization. A few design considerations for imp roving usability and user experiences emerged based on participant responses and our observations. First, a deeper understanding of users' information behavior and their in formation needs in using info rmat ion systems supporting consumer decision-making is important. In this study, we made a step towards this understanding in using turkers to generate the summary reviews. Second, after having identified key features in the consumer decision -making system, a good design should well balance the nu mber of the features and the amount of information p rovided in the interface. Th ird, an appropriate and co mprehensive organization/categorizat ion scheme should be selected in terms of targeted user groups and task and design considerations. Many participants expressed their opinions about the importance of organization/categorization. We feel it should be given greater attention in the design process in the future experiments.

Conclusion

This paper reports a study comparing the effect iveness of four major styles of opinion summarization in a consumer purchase decision scenario. The study leverages the power of the crowd to bypass issues of text min ing quality in order to reach mo re meaningful conclusions.

Our next step is to design and implement an experimental system based on the findings of this study. Such an experimental system will provide customers with a better view in the system interface. Additionally, the experimental system will be compared with our baseline system in a user-centered lab experiment to test its effectiveness and efficiency. Our goal is to contribute to improving the user experience and usability of information systems that support consumer decision-making.

As a lab-based user-centered study, limitations exist. In this experiment, the generalizab ility of the findings was restricted by the limited types of tasks, the number of topics, and the sample pool. Additionally, the coding scheme we generated is a simple, init ial one. Deeper, more fine-tuned coding and analysis could be applied to the data in a subsequent analysis . Despite the limitations, the results of this type of research will have imp licat ions for the design of information systems that support consumer decision-making.

Fig. 1 .

 1 Fig. 1. Tag clouds

Fig. 3 .

 3 Fig. 3. Paragraph summary

Fig. 4 .

 4 Fig. 4. Group samples

Table 1 .

 1 Coding scheme

	Code	Reasons
	CM PP CM PNCMPU	Comprehensiveness (Comprehensiveness of Information)
		(CMP)
	TM EP TM EN TMEU	Time (Time required to read the summary) (TME)
	ORGP ORGN ORGU	Organization/categorization (Organization/categorization of the
		summary's content) (ORG)
	LENP LENN LENU	Length/Amount (Length/amount of information) (LEN)
	APPP APPN APPU	Appearance (Appearance of summary content) (APP)

Table 2 .

 2 M ajor reasons affecting user preference of the four presentation styles

	Code	Reasons	No. Responses
	CM PP; CM PN;CM PU	CM P	33
	ORGP; ORGN; ORGU	ORG	32
	E2UP; E2UN; E2UU	E2U	30
	APPP; APPN; APPU	APP	27
	TM EP; TM EN; TM EU	TM E	26
	LENP; LENN; LENU	LEN	26

Table 3 .

 3 Reasons affecting user preference per presentation style

	S tyles Code	Reasons No.	User Comments
	TAG	CM PN	CM P	27 They don't, they don't give details.
		TM EP	TM E	14 It's very fast; the tag clouds was much faster
					I can know how many people like it, because the
		APPP	APP	13	font size was there for the words.
					It was like I couldn't do anything with it. It was
					like it didn't seem to drive the decision, it wouldn't
		E2UN	E2U	12	drive my decision making.
					We can construe what the product was like in very
		E2UP	E2U	11	short time.
	ASP				It contains negative and neutral and the positive
					opinions.
		ORGP	ORG	23
		APPP	APP	15 Because color coding and numbers
		E2UP	E2U	15 They're very convenient or easy to read
					Percentages mean a lot for people to review all
					these products, and there's examples, a lot of exam-
		CM PP	CM P	10	ples on it.
		LENP	LEN	7 It was very clear, brief… concise information
	PRG				That's getting a lot of information in a fairly simple
		CM PP	CM P	24	package
		TM EN	TM E	20 Spend time reading it.
		LENN	LEN	16 Too long to read.
					I enjoy the formatting and that's what's most nor-
		ORGP	ORG	12	mal for me
					… it's a lot of effort for little result sometimes. It's
					a tossup;
		E2UN &			It's because of so much text and even if you read it,
		ORGN	E2U	10	it is not organized sometimes…
	GRP				Yeah the personal story with the rich information
		CM PP	CM P	10	helps me to imagine what if I had that product.
					I'm seeing the whole review versus just one per-
		ORGP	ORG	8	son's interpretation of all of the reviews.
					In those group samples, that I think was too large is
		LENN	LEN	7	the content was too large.
		E2UN	E2U	5 because they tended to repeat the same information
					It was not categorized properly. And if it isn't orga-
		ORGN	ORG	5	nized properly, it's confusing.