Heidi Lam Google

Enrico Bertini
email: enrico.bertini@uni-konstanz.de

Petra Isenberg
email: petra.isenberg@inria.fr

Catherine Plaisant
email: plaisant@cs.umd.edu

Sheelagh Carpendale
email: sheelagh@ucalgary.ca

Heidi Lam
email: heidi.lam@gmail.com

Seven Guiding Scenarios for Information Visualization Evaluation

Keywords:

We take a new, scenario based look at evaluation in information visualization. Our seven scenarios, evaluating visual data analysis and reasoning, evaluating user performance, evaluating user experience, evaluating environments and work practices, evaluating communication through visualization, automated evaluation of visualizations, and evaluating collaborative data analysis were derived through an extensive literature review of over 800 visualization publications. These scenarios are described through their goals, the types of questions they embody and illustrated through example studies. Through this broad survey and the distillation of these scenarios we make two contributions. One, we encapsulate the current practices in the information visualization research community and, two, we provide a different approach to reaching decisions about what might be the most effective evaluation of a given information visualization. For example, if the research goals or evaluative questions are known they can be used to map to specific scenarios, where practical existing examples can be considered for effective evaluation approaches.

INTRODUCTION

Researchers and practitioners in the field of information visualization (infovis) have long identified the need to evaluate visual data representations, interaction techniques, and visualization systems. Yet, the difficulty of conducting these infovis evaluations remains a common topic. For instance, in addition to the general evaluations challenges of choosing evaluation questions, methods, and correctly executing them, the infovis focus on data and its exploratory analysis processes pose still further challenges, since both the analysis process and outputs, such as specific insights and more global growing comprehension, are difficult to capture and quantify (cf. Section 7).

While the need for facility with evaluations that are capable of addressing these challenges is much discussed [START_REF] Barkhuus | From mice to men: 24 years of evaluation in chi[END_REF][START_REF] Chen | Empirical studies of information visualization: A meta-analysis[END_REF], ascertaining an approach that can improve upon our existing practices has remained elusive. For experimenters, part of the problem is the vast amount of evaluation methodologies in use. Our community draws from diverse disciplines such as psychophysics, social sciences, statistics, and computer science, using methodologies as diverse as laboratory based factorial design studies, field evaluations, statistical data analysis, and automatic image evaluation. The vastness and diversity of evaluation methodologies make it difficult for visualization researchers and practitioners to find the most appropriate approaches to achieve their evaluation goals. Another aspect of the difficulty is the lack of literature guidelines-while some guidelines are available to create and analyze visualization systems, and to evaluate visualizations, these two sets of literature are disparate as discussions on evaluation are mostly "structured as an enumeration of methods with focus on how to carry them out, without prescriptive advice for when to choose between them." ([51, p.1 ], author's own emphasis). We extend this by taking a different tack-we offer advice on how to chose between evaluation approaches.

In the paper, we take a broad community based approach, discovering from the infovis research literature common linkages between evaluation goals and evaluation approaches. We discuss these linkages via evaluation scenarios. This is based on an extensive literature analysis of over 800 papers (345 with evaluation), we systematically identified seven most commonly encountered evaluation scenarios:

1) Evaluating environments and work practices 2) Evaluating visual data analysis and reasoning 3) Evaluating communication through visualization 4) Evaluating collaborative data analysis 5) Evaluating user performance 6) Evaluating user experience 7) Automated evaluation of visualizations For each of these scenarios, we list the most common evaluation questions and where possible illustrate them with representative published evaluation examples from the infovis community. In cases where there are gaps in our community's evaluation approaches, we suggest methods from publications from other sources.

Given the vast scope of the evaluation topic, we do not provide a comprehensive list of existing evaluation methods, though we do provide a wide coverage of the methodology space in our scenarios to offer a diverse set of evaluation options. We leave in depth discussions of specific methods to the ample literature that is available with these details.

The major contribution of our work is a scenario-based approach that offers advice on how to make evaluation choices illustrated with published examples drawn from a diverse literature. More specifically, our work aims to:

• encourage the selection of evaluation methods based on specific evaluation goals (by organizing our guide by scenarios rather than by methodologies);

• diversify evaluation methods used in our community (by providing examples from other disciplines in context of evaluation goals commonly found in our community);

• act as a first step to develop a repository of examples and scenarios as a reference.

THE SCOPE OF EVALUATION

In this guide, we cover evaluation as part of different stages of visualization development, including: 1) Pre-design e. g., to understand potential users' work environment and work flow 2) Design e. g., to scope a visual encoding and interaction design space based on human perception and cognition 3) Prototype e. g., to see if a visualization has achieved its design goals, to see how a prototype compares with the current state-of-the-art systems or techniques 4) Deployment e. g., to see how a visualization influences workflow and work processes, to assess the visualization's effectiveness and uses in the field 5) Re-design e. g., to improve a current design by identifying usability problems Note that with this broad view of evaluation, it is not restricted to the analysis of specific visual representations-it can focus on visualizations' roles on processes such as data analysis, or on specific environments to which visualizations might be applied. Our scenarios are therefore grouped based on their evaluation foci: process or visualization (Section 6). The outputs of the evaluation may be specific to a visualization such as its design decisions, or more general such as models and theories (e. g., theory formation based on grounded theory evaluation [START_REF] Isenberg | Grounded evaluation of information visualizations[END_REF] and perceptual and cognitive modeling based on controlled experiments), and metrics (e. g., metrics developments based on automatic evaluation of visual quality or salience).

HOW TO USE THIS PAPER

The purpose of this paper to suggest a process that can support the design of evaluations of visualizations through the following steps:

1) Setting a goal: In general, before thinking about evaluation methods, we recommend starting by determining a clear evaluation goal [START_REF] Corbin | Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory[END_REF][START_REF] Ellis | An exploratory analysis of user evaluation studies in information visualization[END_REF]. Section 2 lists a range of evaluation goals categorized by stages in visualization development; all of which are covered in our scenarios as described in Section 6. Each scenario is illustrated with a few common evaluation questions that can help to identify for which evaluation goals they may be most suitable. 2) Picking suitable scenarios: Having identified an evaluation goal, the seven scenarios can be used to help to identify close matches with the research goal and to provide descriptions of relevant information related to the goal. 3) Considering applicable approaches: These can be found in the Methods and Examples sections of Section 6. We recommend initially investigating several approaches that may suit a given evaluation need.

Each scenario is illustrated with examples of published evaluations, which can be used as references for additional details. 4) Creating evaluation design and planned analyses:

The design of an evaluation can be strengthened by considering benefits and limitations of each approach listed in the scenario selected. While we aimed to provide a diverse range of evaluation methods, the lists are not exhaustive. Also, in information visualization, research into evaluation methodologies themselves is still active and is still resulting in new methodologies and metrics. For these reasons, we encourage creativity in evaluation design starting from and also extending the work referenced here.

RELATED WORK

In this section, we review related work in the areas of evaluation taxonomies, systematic reviews, and evaluation methodologies and best practices.

Evaluation Taxonomies

Others have approached the problem of guiding researchers and practitioners in visualization evaluation by providing a high-level view of available methodologies and methods as taxonomies. The metrics used for classification have been diverse, ranging from research goals, to design and development stages in which the methodologies can be applied, to methods and types of data collected, to the scope of evaluation. Table 1 summarizes existing taxonomies and their respective foci.

The diversity exhibited in Table 1 reflects the complexity and richness of existing evaluation methodologies and the difficulty in deriving an all encompassing taxonomy. For example, using research goals as a taxonomy axis is challenging because the same evaluation method may be used for different purposes. One example is laboratory-based studies measuring task completion time to compare between interfaces (also known as "head-to-head" comparisons). Such a method can be used to summarize the effectiveness of an interface ("summative") or to inform design ("formative") [START_REF] Andrews | Evaluation comes in many guises[END_REF][START_REF] Ellis | An exploratory analysis of user evaluation studies in information visualization[END_REF]. Similar arguments apply to classifying methods based on design and development cycles-the same method may be used differently at different stages. For example, observational technique may be first used in the pre-design stage to gather background information [START_REF] Isenberg | Grounded evaluation of information visualizations[END_REF], but may also be used post-release to understand how the newly introduced technology affects user workflow. Given these difficulties, we decided on a different approach where we based our discussions on commonly encountered evaluation scenarios instead of methods. Across all the papers we examined, we explored how these scenarios relate to evaluation goals and questions (Section 5). Our goal is to encourage an approach to evaluation that is based on evaluation goals and questions instead of methods and to encourage our community to adopt and accept a more diverse range of evaluation methods.

Systematic Reviews

Our work here is closest in spirit to a subtype of systematic review known as narrative review, which is a qualitative approach and describes existing literature using narrative descriptions without performing quantitative synthesis of study results [START_REF] Shadish | Experimental and Quasi-Experimental Designs[END_REF]. Systematic reviews is itself a type of evaluation method with the purpose to provide snapshots of existing knowledge based on published study results, where "the researcher focuses on formulating general relations among a number of variables of interest" that "hold over some relatively broad range of populations", [48, p. 158]. To the best of our knowledge, two systematic reviews on evaluation methods have been conducted, both counted the number of papers in specific corpora based on the authors' classification scheme.

The first is Barkhuus and Rode's analysis on 24 years of publications in the proceedings of the SIGHCHI Conference on Human Factors in Computing Systems (CHI) [START_REF] Barkhuus | From mice to men: 24 years of evaluation in chi[END_REF]. The researchers found that while the proportion of papers with evaluations increased over time, the quality of the evaluation may not have improved, judging from the decreased median number of participants in quantitative studies, an over-reliance of students as participants, and lack of gender-balanced samples. The second is Perer and Shneiderman's analysis on three years of publications in the proceedings of the IEEE Symposium on Information Visualization (InfoVis) and one year of the IEEE Symposium on Visual Analytics Science and Technology (VAST) [START_REF] Perer | Integrating statistics and visualization for exploratory power: From long-term case studies to design guidelines[END_REF]. In these corpora, the researchers did not find an increase in proportion of papers with evaluation. Similar to Barkhuus and Rode, Perer and Shneiderman also expressed concerns over the quality of evaluation, as most evaluations conducted were controlled studies with non domain experts as test subjects. Our focus in contrast was to derive a common set of researcher questions and approaches to ground the development of our scenarios.

Evaluation Methodologies and Best Practices

There exists a large number of publications that reflect upon current practices in visualization evaluation and provide recommendations to improve our status quo. In fact, the BELIV workshop was created as a venue for researchers and practitioners to "explore novel evaluation methods, and to structure the knowledge on evaluation in information visualization around a schema, where researchers can easily identify unsolved problems and research gaps" [START_REF] Bertini | Beyond time and errors: novel evaluation methods for information visualization (beliv)[END_REF]. In short, providing a complete summary of publications on evaluation probably deserves a paper of its own. In this section, we briefly outline some of the commonly discussed challenges.

In terms of study design, many papers urge researchers to think about the goals of the evaluation [START_REF] Corbin | Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory[END_REF][START_REF] Ellis | An exploratory analysis of user evaluation studies in information visualization[END_REF]. The evaluation goal heavily influences the choice of research strategies, the types of data and methods of collection, and the methods of data analysis. For example, if the goal is to understand how a new technology affects user workflow, then realism is important. In other words, data collection should be from the field using non-intrusive collection mechanisms. Several researchers of these papers that reflect on evaluation commented on the lack of realism in the existing evaluation efforts, which are mostly laboratory based, using basic visual search tasks with non-target users. One way to ensure validity is to ensure realism in tasks, data, workflow, and participants [START_REF] Andrews | Evaluation comes in many guises[END_REF][START_REF] Ellis | An exploratory analysis of user evaluation studies in information visualization[END_REF][START_REF] Plaisant | The challenge of information visualization evaluation[END_REF]]. An alternative is to provide an understanding of situations where some of these requirements can be released, for example, using non-domain expert participants. Other commonly discussed topics of study design include the short durations of most study periods [START_REF] Plaisant | The challenge of information visualization evaluation[END_REF], the narrowness of study measurements [START_REF] Plaisant | The challenge of information visualization evaluation[END_REF], and possibly insufficient training of participants [START_REF] Andrews | Evaluation comes in many guises[END_REF]. In term of data analysis, concerns have been expressed on the narrowness of questions posed and statistical methods applied [START_REF] Ellis | An exploratory analysis of user evaluation studies in information visualization[END_REF]. Given that most of the existing evaluation studies are one-offs, researchers have suggested doing follow-up studies to further investigate unanswered questions [START_REF] Ellis | An exploratory analysis of user evaluation studies in information visualization[END_REF][START_REF] Lam | Increasing the utility of quantitative empirical studies for meta-analysis[END_REF].

In short, all aspects of evaluation require careful attention. This paper is therefore an effort to provide a different kind of guide for visualization researchers and practitioners through concrete scenarios illustrated with existing evaluations.

METHODOLOGY

Early in our project, we decided to take a descriptive rather than a prescriptive approach. In other words, our paper describes and comments on existing practices in evaluating visualizations, but we do not prescribe specific evaluation methods as we believe that the final decision on appropriate methods should be decided on a caseby-case basis. We identified seven evaluation scenarios most commonly encountered by visualization researchers which are meant to guide the development of appropriate evaluation strategies. The scenarios were derived from data collected through open coding [START_REF] Creswell | Research Design. Qualitative, Quantitative, and Mixed Methods Approaches[END_REF] of publications from four information visualization publication venues (see Table 2). Our approach included the following steps to derive the scenarios:

1-Compiling an evaluation dictionary. Initially, to gather a description of existing evaluation practices in the visualization community, we compiled a dictionary Usability.net [START_REF]Usability.net methods[END_REF] Design stages Concept design, detailed design, implementation, analysis Kulyk et al. [START_REF] Kulyk | Human-computered aspects[END_REF] Data and method Data collected (qualitative, quantitative), collection method (empirical, analytical)

Barkhuus and Rode [START_REF] Barkhuus | From mice to men: 24 years of evaluation in chi[END_REF] Data Data collected (qualitative, quantitative, mixed-methods) Creswell [START_REF] Creswell | Research Design. Qualitative, Quantitative, and Mixed Methods Approaches[END_REF] Evaluation scope Work environment, system, components Thomas and Cook [START_REF] Thomas | Illuminating the Path: The Research and Development Agenda for Visual Analytics[END_REF] TABLE 1 Taxonomies of evaluation methods and methodologies based on the type of categorization, the main categories themselves, and the corresponding references.

of terms of existing evaluation strategies and techniques and collected matching definitions and example evaluation publications. Our list was compiled based on information solicited by emails to participants of the BELIV 2008 workshop combined with our own knowledge and research (e. g., [START_REF] Bertini | Beyond time and errors: novel evaluation methods for information visualization (beliv)[END_REF][START_REF] Carpendale | Evaluating information visualizations[END_REF][START_REF] Isenberg | CoCoNutTrix: Collaborative retrofitting for information visualization[END_REF][START_REF] Isenberg | Grounded evaluation of information visualizations[END_REF][START_REF] Lam | Increasing the utility of quantitative empirical studies for meta-analysis[END_REF][START_REF] Plaisant | The challenge of information visualization evaluation[END_REF]). The process yielded a wealth of information which required additional structure but provided us with a richer understanding of the types of evaluations commonly used and helped to provide necessary context for us to perform the open coding and tagging of the evaluation papers.

2-Open coding and tagging. From the set of terms and examples collected in the first phase we derived an initial eight tags that classified evaluations in terms of evaluation goals. These tags included topics such as data analysis, decision making, or usability and were integrated into our final result. We selected four major visualization publication venues from which to identify commonly encountered evaluations:

• Eurographics/IEEE Symposium on Visualization (Eu-roVis)

• IEEE Information Visualization (InfoVis)

• IEEE Visual Analytics Science and Technology (VAST)

• Palgrave's Journal of Information Visualization (IVS)

From these sources, we collected 803 papers and conducted a first coding pass that culled papers that did not mention evaluation and left 345 evaluation papers for further consideration. Publication years and number of papers involved are summarized in Table 2. Three of us performed the open-coding [START_REF] Creswell | Research Design. Qualitative, Quantitative, and Mixed Methods Approaches[END_REF] on parts of the dataset. For each paper, we attached one or more tags from the initial set and recorded the reported evaluation goals and methods. As we proceeded in coding selected publications, each of us independently added new tags to the initial collection, which were then shared among all coders during the tagging period. At regular intervals we discussed the definition of each tag within the group and through consensus, adopted new tags from the other coders during the process and recoded papers with the new tags.

By the end of our publication coding, we had expanded our initial tag set to seventeen tags. Details of the tags can be found in Appendix A.

3-Developing Scenarios. We engaged in one final coding pass and grouped similar tags among the 17 tags to form 7 tags, excluding tags that relate to discussing methods to develop new evaluations as we considered these to be beyond the scope of this paper. The goal was to derive scenarios which would represent the main evaluation questions encountered when evaluating visualization tools. This consolidation provides a more manageable list of elements in order to facilitate their use in practice, as described in Section 3. Scenarios, tags, and paper numbers for each are summarized in Table 3 in the Appendix.

The building of scenarios is, thus, the result of an iterative process among coders where phases of individual grouping and collective consolidation alternated. In the next section, we present the final seven scenarios we derived.

SCENARIOS

In this section, we present our seven evaluation scenarios. For each scenario, we define the scenario, identify the popular goals and outputs, the common evaluation questions, and the applicable evaluation methods along with concrete examples. Note that our scenarios can be roughly classified into two broad categories based on their focus. We call these two categories process and visualization. In the process group, the main goal of the evaluation is to understand the underlying process and the roles played by visualizations. While evaluators may record specific user performance and feedback, the goal is to capture a more holistic view of the user experience. Scenarios that belong to this group are: Evaluating Environment and Work Practices (EWP), Evaluating Visual Data Analysis and Reasoning (VDAR), Evaluating Communication through Visualization (CTV), and Evaluating Collaborative Data Analysis (CDA). In contrast, evaluations can focus on the visualization itself, with the goal to test design decisions, explore design space, bench-mark against existing systems, or to discover usability issues. Usually in these evaluations, a slice of the visualization system or technique is tested. Scenarios that belong to this group include Evaluating User Performance (UP), Evaluating User Experience (UE), and Automated Evaluation of Visualizations (AEV).

Evaluating Environments and Work Practices (EWP)

Evaluations in the EWP group elicit formal requirements for design. In most software development scenarios it is recommended to derive requirements from studying the people for which a tool is being designed [START_REF] Shneiderman | Designing the User Interface: Strategies for Effective Human-Computer Interaction[END_REF] but, as noted by Munzner [51, p.7 ], "hardly any papers devoted solely to analysis at this level [problem characterization] have been published in venues explicitly devoted to visualization." Similarly, Plaisant [START_REF] Plaisant | The challenge of information visualization evaluation[END_REF] has argued that there is a growing need for information visualization designers to study the design context for visualization tools including tasks, work environments, and current work practices. Yet, in information visualization research studying people and their task processes is still rarely done and only few notable exceptions have published results of these analyses (e. g., [START_REF] Isenberg | An exploratory study of visual information analysis[END_REF][START_REF] Tory | Qualitative analysis of visualization: A building design field study[END_REF]).

EWP: Goals and Outputs

The goal of information visualization evaluations in this category is to work towards understanding the work, analysis, or information processing practices by a given group of people with or without software in use. The output of studies in this category are often design implications based on a more holistic understanding of current workflows and work practices, the conditions of the working environment itself, and potentially current tools in use. Studies that involve the assessment of people's work practices without a specific visualization tool typically have the goal to inform the design of a future visualization tool. Studies involving the assessment of work flow and practices with a specific tool in people's work environment try to assess factors that influence the adoption of a tool to find out how a tool has been appropriated and used in the intended work environments in order to elicit more specific design advice for future versions of the tool.

EWP: Evaluation questions

Questions in this scenario usually pertain to identifying a set of features that a potential visualization tool should have. For example:

• What is the context of use of visualizations?

• In which daily activities should the visualization tool be integrated?

• What types of analyses should the visualization tool support?

• What are the characteristics of the identified user group and work environments?

• What data is currently used and what tasks are performed on it?

• What kinds of visualizations are currently in use? How do they help to solve current tasks?

• What challenges and usage barriers can we see for a visualization tool?

EWP: Methods and Examples

There is a wealth of methods available for studying work environments and work practices. Most of these rely on qualitative data such as interviews or observational data, audio-visual, or written material: Field Observation. Observational methods are the most common way to elicit information on current work practices and visualization use. We further described the goals and basics of this method in Section 6.6. In information visualization, few published examples exist of this type of study, but more have been called for [START_REF] Isenberg | Grounded evaluation of information visualizations[END_REF]. In a study of automotive engineers Sedlmair et al. [START_REF] Sedlmair | Requirements for a mde system to support collaborative in-car communication diagnostics[END_REF] observed eight analysis experts at their workspace and derived information on which and how specific tools were used, why they were used, and when participants entered in collaborative analysis. The researchers then used the results of the study to derive a set of requirements for the design of data analysis systems for this domain. Both this study and another by Tory et al. [START_REF] Tory | Qualitative analysis of visualization: A building design field study[END_REF] combined observation with interviews.

Interviews. There are several types of interviewing techniques that can be useful in this context. Contextual inquiry [START_REF] Holtzblatt | Contextual Inquiry: A Participatory Technique for Systems Design[END_REF] is a user-centered design method in which people are first observed and then interviewed while engaged in their daily routines within their natural work environment. The researcher tries to interfere as little as possible. Picking the right person to interview is critical in order to gather useful results. Interviews can also be conducted within a lab context. These types of interviews are more common in information visualization: Pretorius and van Wijk interviewed domain experts about their own data to learn how they analyzed state transition graphs [START_REF] Pretorius | Visual inspection of multivariate graphs[END_REF], Brewer et al. [START_REF] Brewer | Collaborative geographic visualization: Enabling shared understanding of environmental processes[END_REF] interviewed geovisualization experts to learn about multi-disciplinary science collaboration and how it could be facilitated with collaborative visualization tools.

Laboratory Observation. Observational studies also sometimes occur in laboratory settings in order to allow for more control of the study situation. For example, two studies from the collaborative visualization field looked at how visualizations are used and shared by groups of people and how visualization results are integrated [START_REF] Isenberg | An exploratory study of visual information analysis[END_REF][START_REF] Robinson | Collaborative synthesis of visual analytic results[END_REF]. Both studies presented rich descriptions of how people interacted with visualizations and how these activities could be supported by technology.

Evaluating Visual Data Analysis and Reasoning (VDAR)

Evaluations in the VDAR group study if and how a visualization tool supports users in generating actionable and relevant knowledge in their domain. In general, VDAR evaluation requires fairly well developed and reliable software.

VDAR: Goals and Outputs

Evaluations in the VDAR group involve studies that assess how an information visualization tool supports visual analysis and reasoning about data to generate information about the users' domain. Outputs are both quantifiable metrics such as the number of insights obtained during analysis (e.g., [START_REF] Saraiya | An evaluation of microarray visualization tools for biological insight[END_REF][START_REF] Saraiya | An insight-based longitudinal study of visual analytics[END_REF]), or subjective feedback such as opinions on the quality of the data analysis experience (e.g., [START_REF] Seo | Knowledge discovery in highdimensional data: Case studies and a user survey for the rank-byfeature framework[END_REF]).

Even though VDAR studies may collect objective participant performance measurements, studies in this category look at how an integrated visualization tool as a whole supports the analytic process, rather than studying an interactive or visual aspects of the tool in isolation. We cover the latter case in our scenario Evaluating User Performance in Section 6.5. Similarly, VDAR is more process oriented than just to identify usability problems in the interface to refine the prototype, which is covered in Section 6.6. Here, we first focus on the case of a single user. Collaboration is discussed in Section 6.4, Evaluating collaborative data analysis.

VDAR: Evaluation Questions

Data analysis and reasoning is a complex and ill-defined process. Our sample questions are inspired by Pirolli and Card's model of an intelligence analysis process [START_REF] Pirolli | The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis[END_REF], considering how a visualization tool supports:

• Data exploration? How does it support processes aimed at seeking information, searching, filtering, and reading and extracting information?

• Knowledge discovery? How does it support the schematization of information or the (re-)analysis of theories?

• Hypothesis generation? How does it support hypothesis generation and interactive examination?

• Decision making? How does it support the communication and application of analysis results?

VDAR: Methods and Examples

Studying how a visualization tool may support analysis and reasoning is difficult since analysis processes are typically fluid and people use a large variety of approaches [START_REF] Isenberg | An exploratory study of visual information analysis[END_REF].

In addition, the products of an analysis are difficult to standardize and quantify since both the process and its outputs are highly context-sensitive. For these reasons, evaluations in VDAR are typically field studies, mostly in the form of case studies. These studies strive to be holistic and to achieve realism by studying the tool use in its intended environment with realistic tasks and domain experts. However, we also found experiments in which parts of the analysis process were controlled in a laboratory setting.

In this section, we focus on techniques that individual researchers can use, opposed to community wide evaluation efforts such as the Visualization Contest or the Visual Analytics Challenge [START_REF] Costello | Advancing user-centered evaluation of visual analytic environments through contests[END_REF]. The Visual Analytics Challenge provides a useful collection of data sets and analysis problems that can be used in wider VDAR evaluations, and a repository of examples that demonstrate how other tools have been used to analyse the data.

Case Studies. Case studies conducted in VDAR are mostly studies on domain experts interacting with the visualization to answer questions listed in 6.2.2. For example, Trafton et al. conducted an exploratory investigation in the field to answer questions such as "How are complex visualizations used, given the large amount of data they contain?" [84, p. 16 ]. The researchers recruited three pairs of meteorological forecasters and asked them to prepare a written information brief for a flight. The researchers opencoded video data to capture the type of visualizations used in various stages of the analysis.

In some cases, researchers collect data over a longer period of time (from weeks to months) with participants working on their own problems in their normal environments. Analysis activities may be captured by automated logging or by self-reporting methods such as the diary method [START_REF] Symon | Qualitative Research Diaries[END_REF]. Two examples of such long-term case studies in visualization evaluation are: Multi-dimensional In-depth Long-term Case studies (MILCs) [START_REF] Shneiderman | Strategies for evaluating information visualization tools: Multi-dimensional in-depth long-term case studies[END_REF] and insight-based evaluations [START_REF] Saraiya | An insight-based longitudinal study of visual analytics[END_REF].

MILC evaluations use multiple techniques such as observations, interviews, surveys, and automated logging to assess user performance, interface efficacy, and interface utility [START_REF] Shneiderman | Strategies for evaluating information visualization tools: Multi-dimensional in-depth long-term case studies[END_REF]. In MILC studies, researchers offer assistance to participants in learning the system, and may improve the systems based on participant feedback. MILC evaluations have been employed, for instance, to evaluate knowledge discovery tools [START_REF] Seo | Knowledge discovery in highdimensional data: Case studies and a user survey for the rank-byfeature framework[END_REF] and the integration of statistics and visualization [START_REF] Perer | Integrating statistics and visualization for exploratory power: From long-term case studies to design guidelines[END_REF]. The main question Seo et al. set out to answer in their MILC case studies using the Hierarchical Clustering Explorer (HCE) was "how do HCE and the rankby-feature framework change the way researchers explore their datasets" [71, p. 313]. To answer this data exploration question, Seo et al. used participatory observations [10, p. 36] and interviews, conducted weekly for a period of four to six weeks, during which time the participants were asked to use the tool in their everyday work.

Insight-based evaluations try to capture insight as "an individual observation about the data by the participant, a unit of discovery" [67, p. 4]. Data collection methods proposed are either the diary method or capturing video using a think-aloud protocol. For example, Saraiya et al. conducted a longitudinal study with biologists and bioinformaticans using real-life microarray data [START_REF] Saraiya | An insight-based longitudinal study of visual analytics[END_REF]. The goals of the study were to deepen understanding of the visual analytics process, to understand how existing tools were used in analysis, and to test out an evaluation methodology. Data was collected using a diary maintained by the participants to record the analysis process, the insights gained from the data, and which visualization and interaction techniques led to insights, and the successes and frustrations participants experienced with the software tools. Over the course of the study, debriefing meetings were held once every two to three weeks for the researchers to discuss data insights and participants' experience with the tools. Unlike the MILC studies, the researchers did not provide any help with the software tools or guide their participants' data analysis in any way to minimize the study's impact on the participants' normal data analysis process.

Controlled Experiment. Given the open-ended nature of exploration and the specificity of case studies, it may be beneficial to control some of the analysis process and study using laboratory experiments. For example, the Scented Widgets study measured how social navigation cues affected information foraging based on the number of revisits, unique discoveries, and user subjective preferences based on log data [START_REF] Willett | Scented widgets: Improving navigation cues with embedded visualizations[END_REF]. In some cases, experimenters may use a mixture of techniques to enrich the data collected in laboratory experiments. One example is an early insightbased evaluation [START_REF] Saraiya | An evaluation of microarray visualization tools for biological insight[END_REF]. The study used a think-aloud protocol and participants were asked to estimate the percentage of potential insight they would be able to obtain about the dataset with the tool every 15 minutes. In addition, Saraiya et al. coded all individual occurrences of insights from video recordings, with the characteristics of the insights coded by domain experts. Findings were expressed in five measures of insights: count, total domain value, average final amount learned, average time to first insight, and average total time spent before no more insight was felt to be gained. Both the hand-coded as well as the participantrecorded metrics helped to evaluate the most efficient of the five visualization techniques in supporting insight discovery and in influencing users' perception of data.

Evaluating Communication through Visualization (CTV)

Evaluations in the CTV group study if and how communication can be supported by visualization. Communication can pertain to aspects such as learning, teaching, and idea presentation as well as casual consumption of visual information as in ambient displays.

CTV: Goals and Outputs

Visualizations in this category have the goal or purpose to convey a message to one or more persons, in contrast to targeting focused data exploration or discovery. Their effectiveness is usually measured in terms of how effectively such a message is delivered and acquired. Ambient displays, for example, belong to this category as they are usually built to quickly communicate peripheral information to passers-by.

CTV: Evaluation questions

Studies in CTV are often interested in quantifying a tool's quality through metrics such as learning rate, information retention and accuracy, and qualitative metrics such as interaction patterns of the way people absorb information or approach the tool. Questions thus pertain to the quality with which information is acquired and the modalities with which people interact with the visualizations. Examples of questions are:

• Do people learn better and/or faster using the visualization tool?

• Is the tool helpful in explaining and communicating concepts to third parties?

• How do people interact with visualizations installed in public areas? Are they used and/or useful?

• Can useful information be extracted from a casual information visualization?

CTV: Methods and Examples

Controlled Experiments. Quantitative studies aiming at measuring improvement in communication or learning, employ traditional controlled experiments schemes. As an example, Sedig et al. studied how students used a mathematical visualization tool aimed at teaching basic concepts in geometry [START_REF] Sedig | Application of information visualization techniques to the design of a mathematical mindtool: A usability study[END_REF]. A similar study was performed in the context of a basic programming class, using a tool that visualized the role of variables in program animation [START_REF] Sajaniemi | Visualizing roles of variables in program animation[END_REF]. This last experiment is of special interest as it highlights how measuring learning may require the study to span several weeks or months and may, thus, take longer than other traditional evaluations.

Field Observation and Interviews. Qualitative methods like direct observation and interviews are often paired up with experiments in this context. The studies mentioned above, for instance, both complement their quantitative approach with observations of tools in use to understand how information is acquired and to better investigate the process that leads to concept learning. In the context of casual visualizations, that is, visualizations that "depict personally meaningful information in visual ways that support everyday users in both everyday work and nonwork situations" [START_REF] Pousman | Casual information visualization: Depictions of data in everyday life[END_REF], direct observation and interviews are common evaluation techniques. For example, Skog et al. [START_REF] Skog | Between aesthetics and utility: Designing ambient information visualizations[END_REF] study the use of an ambient visualization to convey real-time information of bus departure times in a public university area. The evaluation consists of interviewing people and spending enough time in the area to understand the people's interaction with the system. Viegas et al. [START_REF] Viégas | Artifacts of the presence era: Using information visualization to create an evocative souvenir[END_REF], studied a visual installation in a museum's gallery. The authors observed the reactions of people to the installation and collected the people's impressions to draw conclusions on the design. In a similar context Hinrichs et al. [START_REF] Hinrichs | EMDialog: Bringing information visualization into the museum[END_REF] used an observational and video coding approach to analyze how visitors in an art museum approach a visualization installation and derived design considerations for information visualization in the museum context. As noted by Pousman et al. [START_REF] Pousman | Casual information visualization: Depictions of data in everyday life[END_REF], this kind of observational evaluation is often needed in such context because it is necessary to capture evaluation data in a natural setting where people use the tools naturally.

Evaluating Collaborative Data Analysis (CDA)

Evaluations in the CDA group study whether a tool allows for collaboration, collaborative analysis and/or collaborative decision making processes. Collaborative data analysis differs from single-user analysis in that a group of people share the data analysis experience and often have the goal to arrive at a joint conclusion or discovery.

CDA: Goals and Outputs

Evaluations in this group study how an information visualization tool supports collaborative analysis and/or collaborative decision making processes. Collaborative systems should support both taskwork, the actions required to complete the task, and teamwork, the actions required to complete the task as a group [START_REF] Pinelle | Groupware walkthrough: Adding context to groupware usability evaluation[END_REF]. For collaborative visualization this means that systems must not only support group work well, but also be good data analysis tools (taskwork). We cover the evaluation of taskwork and its questions in the other scenarios (see Section 6.2). Studies in this category have varying goals and, thus, are defined by different types of outputs. Most commonly studies in this group aim to gain a more holistic understanding of group work processes or tool use during collaboration with the goal to derive concrete design implications. It is recognized that the study of teamwork is difficult due to a number of factors including a greater number of variables to consider, the complicated logistics of evaluation, or the need to understand and judge group work processes [START_REF] Neale | Evaluating computersupported cooperative work: Models and frameworks[END_REF]. Collaborative systems (or groupware) can be evaluated on a number of different levels such as the organization it will be embedded in, the team or group that will be using it, or the system itself. While there have been a number of papers concerned with the evaluation of groupware, only few examples of evaluations for collaborative information visualization systems exist.

CDA: Evaluation Questions

For the CDA evaluation of such systems any of or a combination of the following questions may be relevant to address:

• Does the tool support effective and efficient collaborative data analysis?

• Does the tool satisfactorily support or stimulate group analysis or sensemaking?

• Does the tool support group insight? [START_REF] Stahl | Group Cognition[END_REF] • Is social exchange around and communication about the data facilitated?

• How is the collaborative visualization system used?

• How are certain system features used during collaborative work? What are patterns of system use?

• What is the process of collaborative analysis? What are users' requirements?

CDA: Methods and Examples

As research on collaborative visualization systems has only recently begun to receive increased research attention, there are only few examples of studies in this area. We thus draw on results from both Computer-Supported Cooperative Work (CSCW) as well as the small set of recent studies in collaborative visualization. Within the field of CSCW a multitude of study and data collection methods have been applied to the analysis of group work [START_REF] Neale | Evaluating computersupported cooperative work: Models and frameworks[END_REF][START_REF] Pinelle | A review of groupware evaluations[END_REF]. The context of group work (e. g. group configuration, work environment) has been identified as a critical factor in the evaluation and acceptance of collaborative systems (e. g. [START_REF] Grudin | Why cscw applications fail: Problems in the design andevaluation of organizational interfaces[END_REF][START_REF] Neale | Evaluating computersupported cooperative work: Models and frameworks[END_REF][START_REF] Twidale | Situated evaluation for cooperative systems[END_REF]). Yet, several research papers have outlined the practicality of early formative evaluations in less authentic environments (e. g. [START_REF] Pinelle | Task analysis for groupware usability evaluation: Modeling shared-workspace tasks with the mechanics of collaboration[END_REF][START_REF] Twidale | Situated evaluation for cooperative systems[END_REF]). Coupled with later more situated fieldwork a clearer picture of collaborative systems in use and their influence on groups and organizations can be won. Here we highlight a number of possible evaluation techniques.

Heuristic Evaluation. Heuristic evaluation has been previously proposed for the evaluation of visualization systems [START_REF] Tory | Evaluating visualizations: Do expert reviews work?[END_REF][START_REF] Zuk | Heuristics for information visualization evaluation[END_REF]. Finding an appropriate set of heuristics is the main challenge for visualization systems not only to evaluate taskwork [START_REF] Zuk | Heuristics for information visualization evaluation[END_REF]. For the evaluation of teamwork a set of heuristics for the assessment of effectiveness and efficiency of collaboration has been proposed [START_REF] Baker | Heuristic evaluation of groupware based on the mechanicsof collaboration[END_REF]. These heuristics are based on the mechanics of collaboration [START_REF] Gutwin | The mechanics of collaboration: Developing low cost usability evaluation methods for shared workspaces[END_REF][START_REF] Pinelle | Task analysis for groupware usability evaluation: Modeling shared-workspace tasks with the mechanics of collaboration[END_REF] or lowlevel actions and interactions that a collaborative system must support in order for group members to be able to complete a task in a shared manner. Other sets include heuristics based on the locales framework to study the influences of locales (places) on social activities [START_REF] Greenberg | Adapting the locales framework for heuristic evaluation of groupware[END_REF] or awareness [START_REF] Drury | A framework for role-based specification and evaluation of awareness support in synchronous collaborative applications[END_REF].

Log Analysis. Analysis of logs and user traces were the main sources of information analyzed in studies of distributed collaborative web-based information visualization tools [START_REF] Heer | Voyagers and voyeurs: Supporting asynchronous collaborative information visualization[END_REF][START_REF] Viégas | Many Eyes: A site for visualization at internet scale[END_REF]. Both analyses resulted in descriptions and statistics of collaborative use of system features and suggestions for system improvement. Wattenberg [START_REF] Wattenberg | Baby names, visualization, and social data analysis[END_REF] used a slightly different approach while investigating the use of his web-based NameVoyager. He studied off-site reviews and comments and reported on a number of examples of social interaction around the use of the tool. Studies involving the investigation of logs or comments have the advantage to be relatively easy to conduct and evaluate. Little interaction with participants is used to analyze specific system features or a tool use overall. To elicit more user-specific data these evaluation have been combined with questionnaires or interviews (e. g. [START_REF] Heer | Voyagers and voyeurs: Supporting asynchronous collaborative information visualization[END_REF]). On the other hand, these studies cannot clearly evaluate interaction between participants, their work or other processes that do not generate a traceable log entry.

Field or Laboratory Observation. Qualitative user studies have a long tradition within CSCW [START_REF] Greenberg | Observing collaboration: Group-centered design[END_REF][START_REF] Neale | Evaluating computersupported cooperative work: Models and frameworks[END_REF]. Observational studies are often combined with logging of user activity, questionnaires, or interviews [START_REF] Mark | The effects of collaboration and system transparency on CIVE usage: An empirical study and model[END_REF][START_REF] Morris | Team-Tag: Exploring centralized versus replicated controls for co-located tabletop groupware[END_REF]. In an analysis of group activities with an information visualization system Mark and Kobsa [START_REF] Mark | The effects of collaboration and system transparency on CIVE usage: An empirical study and model[END_REF] used such a combination of techniques to analyze group coordination and analysis processes. Effectiveness and efficiency were also assessed by tracking errors and timings for group tasks. Isenberg et al. studied how effectively their collaborative social network analysis system CoCoNutTrix [START_REF] Isenberg | CoCoNutTrix: Collaborative retrofitting for information visualization[END_REF] supported the collaborative analysis process. They performed an observational study and post-session interview to assess how well the system supported the following factors of the collaboration: explicit communication, consequential communication, group awareness, coordination of actions, group insight, subjective work preferences, and general user reactions to the collaborative environment. Without digital systems, other more exploratory observational studies in visualization and visual analytics assessed group analysis processes [START_REF] Isenberg | An exploratory study of visual information analysis[END_REF] and collaborative information synthesis [START_REF] Robinson | Collaborative synthesis of visual analytic results[END_REF]. For collaborative systems studies of work processes are often seen as important prerequisites for estimating outcomes of tool use and to develop mature CSCW tools [START_REF] Neale | Evaluating computersupported cooperative work: Models and frameworks[END_REF].

In contrast to single user systems, collaborative visual analysis systems must also consider the groups interactions and possible harmony/dis-harmony as they proceed in their joint discovery efforts. Stahl [START_REF] Stahl | Group Cognition[END_REF] defines the notion of group cognition as "computer-supported collaborative knowledge building" and recommends the study of this collaborative knowledge building through discourse analysis and observation. It would be interesting to combine this approach with insight-based methodologies (e. g. [START_REF] Saraiya | An insight-based longitudinal study of visual analytics[END_REF]) for the study of group insight.

Evaluating User Performance (UP)

Evaluations in the UP group study if and how specific features affect objectively measurable user performance.

UP: Goals and Outputs

User performance is predominantly measured in terms of objectively measurable metrics such as time and error rate, yet it is also possible to measure subjective performance such as work quality as long as the metrics can be objectively assessed. The most commonly used metrics are task completion time and task accuracy. Outputs are generally numerical values analyzed using descriptive statistics (such as mean, median, standard deviations, and confidence intervals) and modeled by such methods as ANalysis Of VAriance (ANOVA) to partition observed variance into components.

UP: Evaluation questions

Questions addressed using evaluation methods in the UP group are generally narrow and determined prior to the start of the evaluation. There are basically two types of questions:

• What are the limits of human visual perception and cognition for specific kinds of visual encoding or interaction techniques?

• How does one visualization or interaction technique compare to another as measured by human performance?

UP: Methods and Examples

Controlled experiments. In order to answer evaluation questions with quantitative and statistically significant results, evaluations in the UP group require high precision. The most commonly used methodologies involve an experimental design with only a small number of variables changed between experiment conditions such that the impact of such variables can be measured ([10, p. 28]; [48, p. 156]). Such methods are commonly referred to as controlled experiments, quantitative evaluation, or factorial design experiments. A controlled experiment often requires the abstraction of real-life tasks to simple tasks that can be performed by a large number of participants repeatedly in each study session [START_REF] Plaisant | The challenge of information visualization evaluation[END_REF]. Due to the need of a relatively large number of participants, researchers often need to recruit non-experts. As a result, study tasks have to be further abstracted to avoid the need for domain knowledge.

Both types of task abstractions may sacrifice realism. One popular reason to study human perceptual and cognitive limits is to explore the design space for visualization and interaction techniques. The outcomes of these studies are usually design guidelines, and in some cases, models. For example, Tory et al. explored the design space of point displays and information landscape displays, dimensionality, and coloring method to display spatialized data [START_REF] Tory | Spatialization design: Comparing points and landscapes[END_REF].

Bartram et al. explored the design space of using motion as a display dimension [START_REF] Bartram | Filtering and brushing with motion[END_REF]. Heer and Robertson explored the use of animated transitions in linking common statistical data graphics [START_REF] Heer | Animated transitions in statistical data graphics[END_REF].

Another reason to study human perceptual limits is to study how people perform with specific visualization techniques under different circumstances such as data set sizes and display formats. The goal of the evaluation is to explore the scalability of particular visualization techniques. For example, Yost and North investigated the perceptual scalability of different visualizations using either a 2megapixel display or with data scaled up using a 32 megapixel tiled display [START_REF] Yost | The perceptual scalability of visualization[END_REF]. Another example is Lam et al.'s study to assess effects of image transformation such as scaling, rotation and fisheye on visual memory [START_REF] Lam | Effects of 2d geometric transformations on visual memory[END_REF]. In some cases, these experiments can be performed outside of the laboratories. An increasingly popular approach is crowdsourcing with Amazon's Mechanical Turk web service (http://aws.amazon.com/mturk/). Interested readers are directed to a number of validation studies of the method [START_REF] Heer | Crowdsourcing graphical perception: Using mechanical turk to assess visualization design[END_REF][START_REF] Kittur | Crowdsourcing user studies with mechanical turk[END_REF].

The second main evaluation goal in UP is to benchmark a novel system or technique with existing counterparts. These are sometimes known as head-to-head comparisons as participants perform the same tasks on all study interfaces. Interface effectiveness is usually defined by objective measurements such as time and accuracy. Examples of bench-marking studies in UP include the SpaceTree study, where a novel tree browser was compared with a hyperbolic tree browser and an Explorer-type interface to display tree data in a number of node-finding and navigation tasks [START_REF] Plaisant | Spacetree: Supporting exploration in large node link tree, design evolution and empirical evaluation[END_REF].

While most study metrics are time and accuracy, researchers are starting to look at different metrics. One example is memorability. Examples include a study on spatial location memory using Data Mountain in the short term [START_REF] Robertson | Data mountain: Using spatial memory for document management[END_REF], and six months later [START_REF] Czerwinski | The contribution of thumbnail image, mouse-over text and spatial location memory to web page retrieval in 3d[END_REF]. In cases where quality of work instead of objective measures are used as metrics, expert evaluators are required. One example is Hornbaek and Frokaer's study on document visualization, where authors of the documents were asked to determine quality of essays produced by participants [START_REF] Hornbaek | Reading of electronic documents: The usability of linear, fisheye and overview+detail interfaces[END_REF]. Individual differences may also play a role in user performance [START_REF] Chen | Empirical studies of information visualization: A meta-analysis[END_REF]. For example, in the evaluation of LifeLines, Alonso et al. looked at the interaction between participants' spatial visualization ability and display format (LifeLines vs. Tabular) in displaying temporal personal history information [START_REF] Alonso | Viewing personal history records: a comparison of tabular format and graphical presentation using lifelines[END_REF].

Field Logs. Systems can automatically capture logs of users interacting with a visualization. Evaluators analyze these logs to draw usage statistics or single out interesting behaviors for detailed study. This kind of evaluation, especially when performed in web-based environments, has the advantage of providing a large number of observations for evaluation. Also, participants can work in their own settings while data is collected, thus providing a good level of ecological validity. Two recent works used log-based evaluation. Mackinlay et al. [START_REF] Mackinlay | Show me: Automatic presentation for visual analysis[END_REF] used computer logs to evaluate the visual effectiveness of a function inserted into Tableau to suggest users' predefined visual configurations for the data at hand. Viegas et al. [START_REF] Viégas | Many Eyes: A site for visualization at internet scale[END_REF] examined how their design decisions in ManyEyes have been received after deployment.

Evaluating User Experience (UE)

Evaluations in the UE group study people's subjective feedback and opinions in written or spoken form, both solicited and unsolicited.

UE: Goals and Outputs

Evaluation of user experience seeks to understand how people react to a visualization either in a short or a long time span. A visualization here may interchangeably be intended as an initial design sketch, a working prototype, as well as a finished product. The goal is to understand to what extent the visualization supports the intended tasks as seen from the participants' eyes and to probe for requirements and needs. Evaluations in UE produce subjective results in that what is observed, collected, or measured is the result of subjective user responses. Nonetheless objective user experience measurements exist, for example, recording user reactions through the use of body sensors or similar means [START_REF] Mandryk | Modeling User Emotion in Interactive Play Environments: A Fuzzy Physiological Approach[END_REF]. Interestingly, several subjective measures simply mirror the measures we have in user performance, with the difference that they are recorded as they are perceived by the participant. Examples are: perceived effectiveness, perceived efficiency, perceived correctness. Other measures include satisfaction, trust, and features liked/disliked, etc. The data collected in such a study can help designers to uncover gaps in functionality and limitations in the way the interface or visualization is designed, as well as uncover promising directions to strengthen the system. In contrast to UP (Evaluating User Performance, see Section 6.5), the goal of UE is to collect user reactions to the visualization to inform design. Traditionally, studies in UP are more geared towards the production of generalizable and reproducible results whereas those in UE tend to be specific to the given design problem. While VDAR (Evaluating Visual Data Analysis and Reasoning, see Section 6.2) focuses on the output generated through the data analysis and reasoning process, UE looks more at the personal experience. EWP (Evaluating Environments and Work Practices, see Section 6.1) is similar to UE in that prolonged user observation may take place. Nonetheless, EWP focuses on studying users and their environment whereas UE focuses on a specific visualization.

UE: Evaluation Questions

The main question addressed by UE is: "what do my target users think of the visualization?" More specifically:

1) What features are seen as useful?

2) What features are missing? 3) How can features be reworked to improve the supported work processes? 4) Are there limitations of the current system which would hinder its adoption? 5) Is the tool understandable and can it be learned?

UE: Methods and Examples

Evaluations in this category can take varied forms: they can focus on understanding a small number of users' initial reactions, perhaps in depth (as in case studies) but they can also collect extensive qualitative feedback with statistical relevance, for example, in the form of questionnaires. Evaluations can be short-term to assess current or potential usage and long-term to assess the adoption of a visualization in a real usage scenario. The output consists of data recorded either during or after visualization's use. The data can be the result of indirect expert collection of user experience, as when the evaluator takes notes on observed behaviors, or of direct user feedback as when methods like structured interviews and questionnaires are used.

Informal Evaluation. An informal user feedback evaluation is performed by demoing the visualization to a group of people, often and preferably domain experts, letting them "play" with the system and/or observe typical system features as shown by representatives. The method is characterized by a very limited degree of formalism. For instance, it generally does not have a predefined list of tasks or a structured evaluation script as in usability tests. It is the simplest kind of evaluation and it is, probably for this reason, extremely common. These types of evaluations have been used to: assess "intuitiveness and functionality" [START_REF] Janoos | Activity analysis using spatio-temporal trajectory volumes in surveillance applications[END_REF], "probe for utility and usability" [START_REF] Eccles | Stories in geotime[END_REF], "identify design flaws and users' subjective preferences" [START_REF] Song | Multiple foci visualisation of large hierarchies with flextree[END_REF], "evaluate and improve [our] implementation of the ideas" [START_REF] Dwyer | Visualising changes in fund manager holdings in two and a half-dimensions[END_REF], or "to solicit ideas for improvements and enhancements" [START_REF] Weaver | Visual exploration and analysis of historic hotel visits[END_REF].

Usability Test. A usability test is carried out by observing how users perform a set of predefined tasks. For each session, the evaluators take notes of interesting observed behaviors, remarks voiced by the user, and major problems in interaction. The set of tasks is usually defined to address a subset of features the designer deems important for the project. What differentiates this method from the other methods is the careful preparation of tasks and feedback material like questionnaires and interview scripts. Its main goal is to perfect the design by spotting major flaws and deficiencies in existing prototypes [START_REF] Greenbery | Usability evaluation considered harmful (some of the time)[END_REF], nonetheless it can also serve the purpose of eliciting overlooked requirements. McGuffin et al. [START_REF] Mcguffin | Interactive visualization of genealogical graphs[END_REF] assigned "semi-structured navigation tasks" to a genealogist to evaluate and inform the design of a visual tool used to explore large genealogical trees. Hetzler et al. [START_REF] Hetzler | Turning the bucket of text into a pipe[END_REF] ran a usability test to "refine the details of the user interaction" of a visual analysis system designed to support constantly evolving text collections. The test was performed with 3 users and a series of assigned questions.

Field Observation. This method is similar to a usability test in that careful observation of users is involved. The observation however happens in a real world setting, where the system under study is used freely. The main goal of field observations is to understand how users interact with the tool in a real setting and thus to derive useful information on how it can be perfected. Often, the information extracted from this kind of study is a series of emergent patterns that can inspire new designs or improve the current one.

Sometimes, this kind of study can be followed by a more formal step of questionnaires or interviews to better understand the nature of the observed patterns. An example of field observation is the study of Vizster, a visualization that explore online communities [START_REF] Heer | Vizster: Visualizing online social networks[END_REF], where the authors observed usage in an "installation at a large party" where participants were free to use the developed tool.

Laboratory Questionnaire. The large majority of controlled experiments are followed by a subjective user experiment rating phase where participants typically fill out a questionnaire to solicit their opinions and reactions to the tested visualization. These questions may be closed ended with answers expressed in a five-or seven-point Likert Scale, or open-ended with free answers. While this phase of the evaluation is generally coupled with evaluating user experiment studies, we include it here as the method can be used alone. See Section 6.5 for examples of controlled experiments.

Automated Evaluation of Visualizations (AEV)

Evaluations in the AEV group study the aspects of visualization that can be measured automatically by a computational procedure.

AEV: Goals and Outputs

This class of evaluation scenarios comprises all methods that employ an automatic computer-based evaluation of visualization. The results of studies in this group usually consist of a series of numbers that represent the visualization quality or efficiency.

AEV: Evaluation questions

Questions in this scenario usually pertain to the visual effectiveness or computational efficiency with which data is represented. Typical questions in this domain are:

1) Is this layout algorithm faster than other state of the art techniques? Under what circumstances? 2) How does the algorithm perform under different volumes of data and number of dimensions? 3) What is the best arrangement of visual features in the visualization to optimize the detection of interesting patterns? 4) What is the extent to which the current visualization deviates from a truthful representation of underlying data? 5) What is the best ordering of visual items to speed up visual search?

AEV: Methods and Examples

Within this class, the evaluation of algorithmic performance plays a large role, especially when the performance of rendering algorithms is particularly relevant for the goals of the visualization. Other methods assess aspects of visual effectiveness by using some kind of computable metric.

The metric can for instance represent amount of clutter, the level of optimization in the use of screen space (e. g., to compare different solutions head-to-head), the degree of organization in a given arrangement or the degree to which the visualizations adhere to an optimal (virtual or real-world) model. Algorithmic Performance Measurement. The analysis of algorithmic performance is so common in the whole domain of Computer Science that a full discussion of its features is beyond the scope of the paper. Information visualization, by employing algorithms to display data in clever and efficient ways, is of course also often evaluated in terms of algorithmic efficiency. The two most common and established methods in this area are algorithm complexity measures and benchmark tests. In algorithm complexity usually the goal is to demonstrate that the time complexity of the proposed algorithm is better than the state of the art or that the provided solution can be computed in linear or logarithmic time. Less often, but equally important, the complexity of the algorithm is also evaluated in terms of allocated space. Benchmarking happens in a more experimental fashion. Normally the algorithm is evaluated in terms of running time and allocated space over a predefined set of cases (usually several variations of data size or dimensionality) as in the study published by Artero et al. on an algorithm to uncover clusters in parallel coordinates, where the algorithm is tested over several variations of data size and dimensionality. Sometimes the same approach is used to compare the performance of several alternative algorithms, as in the study of Peng et al. on clutter reduction [START_REF] Peng | Clutter reduction in multi-dimensional data visualization using dimension reordering[END_REF], where several algorithms are compared in terms of time performance over different data sizes. In many cases, standard benchmarking data sets are used for evaluation. One example is using graph data sets from the AT&T Graph Library (www.graphdrawing.org) to evaluate graph-drawing algorithms.

Quality Metrics. All studies where automatic evaluation is used share a fundamentally common model: one or more metrics must be devised to assess the quality of a given visualization against an absolute quality level or to compare alternatives. Some studies have proposed generic metrics that can be applied, in principle, to any kind of visualization. Edward Tufte's "data-ink ratio" is one notable example [START_REF] Tufte | The visual display of quantitative information[END_REF] that evaluates a visualization in terms efficient use of screen space. Another example is the work by Brath [START_REF] Brath | Metrics for effective information visualization[END_REF] which proposes high-level metrics for a wide set of visualizations to compare data features with visualization features as a way to find appropriate matching. More computationally intensive generic methods also exist. The clutter measure [START_REF] Rosenholtz | Feature congestion: A measure of display clutter[END_REF] permits evaluation of any generic digital image in terms of clutter. Haroz and Ma [START_REF] Haroz | Natural visualization[END_REF] proposed a method to measure the extent to which a visualization resembles the representation of a natural picture, arguing that natural visualizations are more aesthetically pleasing.

By contrast, some metrics are designed to evaluate a specific technique. Hao et al. [START_REF] Hao | Density displays for data stream monitoring[END_REF] use metrics to compare different design solutions in terms of "constancy of display" and "usage of display space" for a data stream monitoring tool. Constancy is measured in terms of changed pixels over time, display space in terms of used pixels in the avail-able space. A similar study [START_REF] Bederson | Ordered and quantum treemaps: Making effective use of 2d space to display hierarchies[END_REF] compared different sorting algorithms for ordered treemaps. The configurations were compared in terms of "the average aspect ratio of a treemap layout, and the layout distance change function, which quantify the visual problems created by poor layouts." [START_REF] Shneiderman | Ordered treemap layouts[END_REF]).

DISCUSSION

Evaluation is becoming increasingly important in the field of information visualization. The scenarios presented in this paper show the wide range of evaluation goals and questions in which the information visualization research community is currently engaged. These scenarios, and their associated evaluation goals and questions, provide an overview of the types of questions the community has asked of its tools and representations. We also provide information for each scenario about how different evaluation methodologies have been used by different researchers as they work towards discovering answers for their research questions. In this paper, we have contributed a descriptive analysis of the state of evaluation in information visualization. By analyzing several hundred papers, using tags to distinguish between approaches and categorizing and grouping the tagged evaluations into scenarios, goals, and research questions we provide a systematic overview of the diversity of evaluations and research questions that are relevant to information visualization research community.

We started this investigation by tagging evaluation papers from different information visualization venues to get a broader understanding of the types of evaluation goals present in our community. Table 3 lists the 17 tags that were used to code the papers and Appendix A provides descriptions for each tag in more detail. These tags are organized into groups according to the scenario each tag was assigned to. The scenarios are listed by acronym in the order that they are discussed in Section 6. In these scenarios we found two main categories of visualization evaluation focus: (1) the process of data analysis, and (2) the assessment of visualization use.

When analyzing this data numerically one can see a skewed distribution of papers across the different scenarios. The large majority of evaluations fall into our visualization evaluation group: Evaluating User Performance-UP (27%), Evaluating User Experience-UE(21%), and Automated Evaluation of Visualization-AEV (37%), together contributing to 85% of all evaluation papers. This is in sharp contrast to the 15% in the process scenarios.

The fact that the process visualization group is much less represented in the literature is somewhat surprising as the questions in these group are of high relevance to the field: how can visualization tools be integrated and used in everyday work environments (EWP), how are tasks such as reasoning, knowledge discovery, or decision making supported (VDAR), how does a visualization support communication and knowledge transfer (CTV), and how does a visualization support collaborative analysis (CDA). These questions are of high practical value beyond specific individual tools and can benefit both researchers and practitioners in all areas of visualization.

Several reasons could explain our current evaluation focus. Evaluation in the information visualization community has been following the traditions of Human-Computer Interaction (HCI) and Computer Graphics (CG), both of which also have traditionally focused on controlled experiments, usability evaluations, and algorithm evaluations [START_REF] Greenbery | Usability evaluation considered harmful (some of the time)[END_REF]. Possible questions include: [START_REF] Alonso | Viewing personal history records: a comparison of tabular format and graphical presentation using lifelines[END_REF] are experiments in the process group simply not being conducted as frequently? (2) does the fact that these types of evaluations are often lengthy requiring field studies, case studies, and extensive qualitative data analysis contribute to their under representation? (3) are we as a community less welcoming to these different-often qualitative-types of evaluations? The lack of evaluations in this group raises questions about whether we as a community should take steps to encourage more evaluations in these groups to be conducted and published.

In the wider HCI community it is comparatively common that publications solely focus on evaluation, often using field and long-term evaluation approaches. In the process evaluation group we frequently used examples from venues outside of the four publications we coded (EuroVis, InfoVis, IVS, Vast) to illustrate scenarios in which these types of methodologies are more common (e. g. [START_REF] Isenberg | An exploratory study of visual information analysis[END_REF][START_REF] Perer | Integrating statistics and visualization for exploratory power: From long-term case studies to design guidelines[END_REF][START_REF] Trafton | Turning pictures into numbers: Extracting and generating information from complex visualizations[END_REF][START_REF] Twidale | Situated evaluation for cooperative systems[END_REF]). As our community continues to grow we need to think critically about what types of evaluations we would like to see more of and how they can benefit our community. If the current trend continues we will likely see process and longer term qualitative evaluations published at other venues.

For this article, we have coded four main visualization venues and arrived at the codes we used through discussions and several coding passes. Yet, we encourage others to extend our coding or to re-code our results at a later point in time to see how the community has evolved in terms of what kind of evaluation papers are published. Since our coding is based on the published literature it is entirely possible that further coding can reveal new scenarios and questions which we may not have considered here. We encourage others to publish these findings and help to expand our evolving understanding of evaluation in information visualization.

CONCLUSION

Our seven evaluation scenarios encapsulate the current state of evaluation practices in our surveyed papers. From the over 800 papers we surveyed in the EuroVis, InfoVis, and VAST conferences as well as the IVS journal, we found 345 papers that included evaluations. We coded these evaluations according to seventeen tags (see Table 3) and condensed these tags into seven scenarios. The seven scenarios are:

• Evaluating environments and work practices: to derive design advice through developing a better understanding of the work, analysis, or information processing practices by a given group of people with or without software use.

• Evaluating visual data analysis and reasoning: to assess how an information visualization tool supports analysis and reasoning about data and helps to derive relevant knowledge in a given domain.

• Evaluating communication through visualization: to assess the communicative value of a visualization or visual representation in regards to goals such as teaching/learning, idea presentation, or casual use.

• Evaluating collaborative data analysis: to understand to what extent an information visualization tool supports collaborative data analysis by groups of people.

• Evaluating user performance: to objectively measure how specific features affect the performance of people with a system.

• Evaluating user experience: to elicit subjective feedback and opinions on a visualization tool.

• Automated evaluation of visualizations: to automatically capture and measure characteristics of a visualization tool or algorithm. These scenarios can be used as a practical context-based approach to exploring evaluation options. To briefly reiterate we recommend:

1) Setting a goal: start by choosing an evaluation goal. In Section 6, each scenario includes a range of evaluation goals that are characterized by evaluation questions. 2) Picking suitable scenarios: the choice of a goal helps identify a relevant scenario. 3) Considering applicable approaches: from the scenarios possible methods can be investigated. Referenced examples for each method are provided. 4) Creating evaluation design and planned analyses:

these methods and examples provide a spring board for designing evaluation methods that fit your research and your research goals. However, since evaluation is still in flux, it is important to keep abreast of new evaluation methods in your considerations. Our scenario approach can, thus, be used as a starting point for expanding the range of evaluation studies and opens new perspectives and insights on information visualization evaluation. In contrast to other overview articles on evaluation, a major contribution of our work is that we based our evaluation categorization of evaluation questions and goals instead of on existing methods. Our intention is to encourage the information visualization community to reflect on evaluation goals and questions before choosing methods. By providing a diverse set of examples for each scenario, we hope that evaluation in our field will employ a more diverse set of evaluation methods.

TABLE 2

 2 Venues included in the open-coding stage to identify commonly encountered evaluation goals, which were then distilled into scenarios.

TABLE 3

 3 Original coding tags, the number of papers classified, and the final scenario to which they were assigned. 11) Potential usage: Solicit users' opinions on how the visualization may be useful. 12) Proposed Evaluation Methodologies: Propose new methodologies on evaluation. 13) Visualization-analytical operation: Study how visualization affects users' performance of simple visual tasks. 14) Causal information acquisition: Study how users causally acquire information, especially in ambient displays. 15) Evaluation metrics development: Propose new evaluation metrics. 16) Algorithm quality: Evaluate algorithms when compared to accepted gold standards such as human judgments. 17) Knowledge management: Evaluate how effectively does the system support management of knowledge generated in the sense-making loop.

ACKNOWLEDGMENTS

We would like to thank Adam Perer and Amy Voida for early discussions on structuring evaluation methodologies. We would also like to thank the participants of Beliv 2008 who have contributed material to our early data collection on evaluation methodologies in information visualization.

APPENDIX A TAGS USED IN OPEN CODING

We developed our seven scenarios based on the following 17 tags. These tags were used to open code publications from four venues. The distribution of publication by tags and venue is listed in Table 3.

1) Data analysis: Evaluate how visualization is used in exploratory data analysis to generate hypotheses. 2) Decision making: Evaluate how visualization is used to confirm or refute solutions to problems or hypotheses. 3) Collaboration: Evaluate how visualization supports collaboration activities. 4) Adoption: Observe how a visualization is adopted after deployment.