Russ Burtner

Alex Endert

Toward a Deeper Understanding of Data Analysis, Sensemaking, and Signature Discovery

Keywords: Analytic process, sensemaking, signature discovery, visual analy tics, data analysis

Data analysts are tasked with the challenge of transforming an abundance of data into knowledge and insights. This complex cognitive process has been studied, and models created to describe how the process works in specific domains. Two popular models used for this generalization are the sensemaking and signature discovery models, which apply a cognitive and computational focus to describe the analytic process, respectively . This work seeks to deepen our understanding of the data analysis process in light of these two models. We present the results of interviews and observations of analysts and scientists in four domains (Biology, Cyber Security, Intelligence Analysis, and Data Science).

Our results indicate that specific aspects of both models are exhibited in the analysts from our study, but neither describe the holistic analysis process.

Introduction

Data analysts are tasked with coming about insightful conclusions based on large amounts of unstructured information provided to them. As such, understanding and generalizing the process of gaining such insight has been widely st udied. For example, the literature on sensemaking depicts the iterative process of developing insight through foraging information and synthesizing knowledge by mapping features from data to a sets of hypotheses, and then performing assessments to understand the resulting complex relationships [START_REF] Russell | The cost structure of sensemaking[END_REF]. Sensemaking is a cognitive activity that entails the iterative development, re-assessment, and refinement of knowledge structures formed by prior experiences and data. Two popular models of sensemaking, Pirolli and Card's sensemaking loop [START_REF] Pirolli | Sensemaking Processes of Intelligence Analysts and Possible Leverage Points as Identified Though Cognitive Task Analysis[END_REF] and the data/frame model presented by Klein et al. [START_REF] Klein | Making Sense of Sensemaking 2: A Macrocognitive Model[END_REF], present frameworks by which this process can be thought of. The emphasis of both of these models is on the cognitive aspects of users performing such tasks.

Another model, called the signature discovery process [START_REF] Baker | Research towards a systematic sign ature discovery process[END_REF], is a model aimed at understanding the data-centric counterpart to such tasks. This approach (shown in Figure 1) takes a more quantitative and computational approach to describing the analysts' processes. Signature discovery describes the process of developing a signature based on an insightful understanding of the data gathered from both computational and co gnitive reasoning. The signature discovery process requires analysts to specify a pro blem, and categorize the necessary data that needs to be inventoried, assessed, and explored. From this data, analysts can find a unique or distinguishing measurement, pattern, or collection of the data that can be used to detect, characterize, or predict a target phenomenon (e.g., state, object, action, or behavior) of interest. As a result, the signature represents a mathematical quantification of the insight. For example, a signature can be realized as a parameterized classifier used to generate a cluster of images, where that cluster represents an interesting finding as determined by the user.

In this paper, we present the findings of a user study consisting of semi-structured interviews and observations of data analysts in four domains: Biology, Cyber Security, Intelligence Analysis, Data Science. For this paper, we broadly define data analyst and analyst to describe a person who is tasked with gaining insights from data, not as a specific job title or formal description of their task. The study seeks to understand how the practice of data analysis across these four domains compares to two current models describing such processes: the sensemaking loop and signature discovery. Our results indicate that while there exists overlap with current models, as the fluidity and personalization of the participants' workflows carries inherent complexity. The primary contributions of our work are:

 Providing a deeper understanding of the data analysis processes in the four domains studied (Cyber Security, Intelligence Analysis, Biology, Data Science)  Discovering that both sensemaking and signature discovery models depict specific aspects of data analysis, but neither fully describe the holistic process  Discussion of findings to inform design of future data analysis tools

RELATED WORK

Signature Discovery

Baker et al. define a signature to be a unique or distinguishing measurement, pa ttern, or collection of data that detects, characterizes, or predicts a target phenomenon (state, object, action, or behavior) of interest as well as the transformation from events to measurements to features to categorical labels and associated uncertainties [START_REF] Baker | Research towards a systematic sign ature discovery process[END_REF]. The steps of the signature discovery process are diagrammed in Figure 1. The signature discovery process is a linear process that entails problem specification, identification of observable data, specification of measurements for the observable data, feature extraction, and finally signature development with the option to re -assess specific steps. The result of signature discovery, along with a signature, is a signature system; a reusable function, or piece of insight that can be used on similar inputted datasets. Once applied, the process calls for users to test the suitability of the created signature for the task and domain [START_REF] Baker | Research towards a systematic sign ature discovery process[END_REF]. This work seeks to further understand the definition of signatures, signature sy stems, and the signature discovery process from the viewpoints of analysts from the four domains chosen. Further, these interviews and observations will help define the relationship between signature discovery and other sensemaking literature.

Sensemaking

Sensemaking, as defined by Russell et al., is the process of searching for a representation to answer task-specific questions [START_REF] Russell | The cost structure of sensemaking[END_REF]. There are two major loops of activities in sensemakinga foraging loop that involves processes aimed at seeking info rmation, searching and filtering it, and reading and extracting information and secon dly, a sensemaking (or synthesis) loop that involves iterative development of a mental model from the schema that best fits the evidence [START_REF] Russell | The cost structure of sensemaking[END_REF], [START_REF] Pirolli | Sensemaking Processes of Intelligence Analysts and Possible Leverage Points as Identified Though Cognitive Task Analysis[END_REF], [START_REF] Pirolli | Information foraging in information access environments[END_REF]. A model of the sensemaking (the sensemaking loop) is shown in Figure 2. The foraging loop progresses analysts from external, raw data sources to evidence files that refer to a more casespecific subset of information. The synthesis portion of this loop describes the steps involved in going from an evidence file to a final presentation of the hypothesis d eveloped through schematizing, building a case, searching for evidence and support, and re-evaluating output, making this loop more of a cognitive, synthesizing activity.

The emphasis of sensemaking is on creating a mental model and does not explicitly contain components correlating to scientific tools, such as visualizations [START_REF] Pohl | The User Puzzle: Explaining the Interaction with Visual Analytics Systems[END_REF].

Another model of sensemaking, presented by Klein et al. [START_REF] Klein | Making Sense of Sensemaking 2: A Macrocognitive Model[END_REF], shows another descriptive model for sensemaking. Their model depicts an exchange of information between the human and the data in terms of frames . That is, a user has an internal "frame" that represents her current understanding of the world. Then, as she continues to explore a particular dataset, her frames of the world are mapped against the info rmation she uncovers. If the information supports a specific frame, that frame is thought to strengthen. However, when evidence is discovered through exploration that refutes the existence of such a mental frame, the frame can either be augmented or a new one created. Their data/frame model consists of reframing data through filtering, making connections, and defining characteristics, which can be equated to the fora ging loop from Pirolli and Card's work. Other steps and cycles elaborate frames, refine frames through questioning and preserve frames, and can be said to parallel aspects of the sensemaking loop defined in Pirolli and Card's work.

Workflow Management Tools

It is assumed that there are computational steps in the analytic process. To carry out these computational steps, workflow management tools have been developed. It is expected that during the analytic process, that workflow management tools be used to automate computationally attractive steps. The steps used within these tools can p otentially map to steps in the sensemaking and signature discovery processes.

For example, Taverna is a workflow management tool, first used in the area of bioinformatics, to assist in the procedural computation steps of conducting experiments [START_REF] Oinn | Taverna: a tool for the comp osition and enactment of bioinformatics workflows[END_REF]. The tool also provides provenance information, which is information that ident i-fies the source and processing of data by recording the metadata and intermediate results associated with the workflow. The provenance tracking capability enables users to conduct sensemaking activities, specifically those found in the foraging loop and the beginning of the sensemaking loop, through the use of visualizations and a graphical user interface. Steps of the s ignature discovery process where data is processed, such as feature extraction and measurement specification, can be paralleled to the calculation and data processing steps that a workflow management tool performs.

Similar, VisTrails visualizes the steps and interactions of the analytic process [START_REF] Callahan | VisTrails: visualization meets data management[END_REF]. The tool captures the provenance of the visualization proces ses and the data they manipulate as well, which can enable the reevaluation steps within the sensemaking model. The tool also provides the ability to apply a dataflow instance to different data, which can be equated to the feature extraction aspect of sign ature discovery.

In this work, the importance of these tools and how they coincide with the sign ature discovery process and sensemaking is investigated.

Observing Analysts and Domain Experts

Analysis and assessments of the analytic process have been conducted to find out the challenges faced by analysts, design implications for analytic tool development, and future trends of the analytic process. For example, Fink et al. and Endert et al. observed how cyber analysts use visualizations and large high -resolution displays to perform a sensemaking task, producing findings about their processes [START_REF] Endert | Profes sional Analysts using a Large, High-Resolution Display[END_REF]- [START_REF] Andrews | Space to think: large high-resolution displays for sensemaking[END_REF]. The findings presented enhancements for analyst's performance, such as desire to save the state of their investigation at certain points, a need to provide rich linkages among multiple visualization tools, means of keeping a visual history of steps in the analysis, and deep access to the data provided. Similarly, Kandel et al. interviewed analysts to gain insight on their analytic process within the social and organizational context of companies. They were able to create archetypal patterns to categorize analysts and organizational schemes of workflow processes for enterprise workers [START_REF] Kandel | Enterprise Data Analysis and Visualization: An Interview Study[END_REF]. Evaluations of analytic processes were also investigated by Scholtz et al., who (through the use of the VAST challenge) found the importance of assessing the process of the analysis separate from the accuracy of the resulting analysis [START_REF] Scholtz | Evaluation of visual analytics environments: The road to the Visual Analytics Science and Technology challenge evaluation methodology[END_REF].

The work presented in this paper builds upon the knowledge gained through these studies. First, we study additional domains to gain insights into how data analysis occurs across domains. Second, we map our findings to two existing models for data analysis (signature discovery and sensemaking).

Method

The study consisted of a series of semi-structured interviews and observations of participants performing analysis to illuminate domain-specific information from our subject matter experts. The questions asked during the interviews include what type of problems they are tasked with, what types of data they use, which tools they use in their analytic process, and how they measure success . The observations were con-ducted by having the investigators passively watch as the analysts walked through one of their recent tasks. The sporadic (and at time chaotic and time-sensitive) nature of their actual analysis forced us to observe past data analysis processes.

Research Questions

We seek to address the following three research questions (RQ1 -RQ3), and their associated hypotheses (H1 -H3):

RQ1: What are the similarities and differences between the signature discovery process from Baker et al., the model of sensemaking from Pirolli and Card, and the analytic process as described by analysts? H1 : Our hypothesis for RQ1 is that the signature discovery process will reflect a more quantitative and sequential process, whereas sensemaking will be shown to be more fluid (as described in the related work). It is likely that the two may complement each other.

RQ2:

What are tangible outcomes/products of the analytic process? H2 : We hypothesize that the outcome of the analytic processes is a signature as d efined in [START_REF] Russell | The cost structure of sensemaking[END_REF] with little variance between domains and that a combination o f the signature and signature system will form the analyst's presentation. That is, analysts seek to find repeatable mathematical descriptions of their findings.

RQ3:

How do analysts use workflow management tools in their analytic process? H3 : Our hypothesis for RQ3 is that analysts use workflow management tools wit hin their analytic process for feature extraction and signature construction (i.e., the more computationally-focused aspects of their work).

Data Analysis

From the semi-structured interviews, the investigators collected answers to the seed questions, captured on paper in natural language. These were later transcribed to digital versions, to ease indexing, searching, and annotating. The observations were conducted by having the investigators passively watch as the analysts walked through one of their recent tasks. The sporadic (and at time chaotic and time-sensitive) nature of their actual analysis forced us to observe past data analysis processes (instead of active analysis tasks). We collected notes on general observations, steps taken, and comments made by the analysts during these sessions. These notes were also tra nscribed to digital versions, similar to the interview responses.

Each of the three investigators performed an open coding process of binning the digital transcripts to reveal higher-level themes. While there were no formal, predefined topics or themes to code for, the investigators were aware of the previous models being tested (signature discovery and sensemaking). Thus, as each investigator individually analyzed the transcripts, caution was taken to mark instances where the transcripts describe or comment on aspects of either sensemaking or signature discovery. After each individual investigator saturated the individual th emes (or bins), we met collaboratively to compare, contrast, and final set of labels for observed instances. These were then mapped into the stages of the sensemaking and signature discovery models, shown in Figure 3. Note that this set does not consist of all the stages of sensemaking and signature discovery, as some stages or aspects were not observed.

Participants

Semi-structured interviews were conducted with sixteen analysts (P1-P16) in the areas of Biology [START_REF] Oinn | Taverna: a tool for the comp osition and enactment of bioinformatics workflows[END_REF], Cyber Security (3), Intelligence Analysis (2) and Data Science (4). Of these sixteen, one had to be discarded for analysis due to classification and data sensitivity concerns (P7).

Biology

The biologists that we interviewed covered a wide scope of work from Epidemio logy to Genetics. Six of the seven hold a Ph.D. in Biology/Computational Biology, and have five or more years of experience in the field with strong publishing credentials.

Cyber Security

The cyber security analysts had strong IT backgrounds and have been established in the field for 10 or more years (two were only in their current job for 2 years). Their scope of work covered smaller local networks to broader more national investigative cyber networks. Their levels of education are Bachelors of Science to Masters in Computer Science or Computer Engineering.

Intelligence Analysts

The intelligence analysts we interviewed worked for multiple government agencies including the Department of Energy. Each had a diverse background in either a nuclear science related field (thus the "Nuke" label in figures in this paper) or comput ational science. Both have been in intelligence for more than ten years.

Data Science

The data scientist we talked to are all traditional computer science or formal statisticians. Their level of expertise varied from over 10+ years of experience to only two years out of school. Their level of education also varied from Master's degree to Bachelors of Science.

Findings

Through the thematic coding approach used to analyze the data collected from the interviews and data analysis observation sessions , we present the following findings .

First, we present details regarding the characteristics of the analytic process based on the data sources, tools used, and methods for recording their analytic provenance (or process). Second, we compare the self-reported and observed processes to existing models of sensemaking and signature discovery. Third, we analyze the analysts' methods for reporting the findings and insights .

Analytic Process Characteristics

Data sources

All of the domains we studied utilized databases as a primary method for storing and retrieving their data. Among the intelligence analysts, data sources commonly used were media such as images and text. Our biology users accessed many scientific databases shared among other scientists in their discipline or clinical study. The Data Science analysts were mostly users of a variety of observational sensor data gathered from databases, log files, or data streamed directly from a sensor or system. These datasets were described to be rather large and high dimensional. The cyber security analysts primarily accessed locally-hosted databases of log files. One challenge faced by all of our analysts we spoke with was how to subset the large quantity of data, so that a smaller amount of it can be worked on locally.

Tools used

Most of the analysts across the four domains used a data analytic tool, like R or Matlab, as well as a variety of visualization tools. A commonality among the analysts across domains was the use of spatialization tools, which are visualizations that clu ster like data together based on specific attributes of the data. Spatialization tools such as Starlight [START_REF] Risch | The STARLIGHT information visualization system[END_REF], IN-SPIRE [START_REF] Wise | Visualizing the non-visual: spatial analysis and interaction with information for text documents[END_REF], Clique and Traffic Circle [START_REF] Abdullah | Visualizing network data for intrusion detection[END_REF], and ARGIS were used by the analysts that we interviewed. These spatialization tools were semiautonomous processes that allowed manipulations by the analysts to occur in order to perform feature extraction. A myriad of custom scripts were also used by analysts as well as other feature extracting processes, which were used in a more batch/sequential manner in comparison to the spatialization tools focused on exploration.

To perform any of their scripts and queries, it was assumed that analysts utilized workflow management tools, such as Taverna, but that did not prove to be true among the majority of analysts (only two stated they use such tools). For example, P11 stated that Taverna "provided too many options I do not need." This analyst preferred to maintain a detailed record of inputs and outputs of scripts and queries to recall the process and execute the sequence of scripts for the task. P9 detailed that the purpose of using the Galaxy workflow management tools was to "replay workflows" as well as "share and collaborate on workflows", something that was not desired and thus added un-needed overhead to using the tool. Other participants detailed natural progressions in their work, which may have been complimented by a workflow manag ement tools. While they commented that these sequences of scripts could be described as a workflow, they are created in-situ, and thus they prefer the flexibility of performing these computational steps as scripts, often executed in standard command line interfaces. These findings were particularly interesting with regards to our third research questions and hypothesis (R3 and H3), in that workflow management software, in the computational sense, was not used by our participants.

Analytic Provenance Recording and Documentation.

Upon coming across phenomena of interest, analysts look to capture/document it in some way in order to recall it later. These moments are captured in two ways: the process and the finding. P11 captures the process by using a "configuration file to specify what to run", so that it can be repeated when needed. P2, an analyst that pe rformed most work manually, filed pages (printed hard copies) to capture findings. P3 stated that findings are best captured and described through publications. From the interviews, it was seen that there is an importance set on documenting and filing ph e-nomenon for sharing and revisiting findings. P15 stated that he and any collaborators kept an online journal that was used to keep track of important findings throughout their processes for easy reference later on and to provide an opportunity for peer reviewing. In general, we observed that the provenance or process of our analysts was maintained primarily via inputs and outputs of data to and from scripts.

Comparisons to Existing Models

Our hypothesis (H1) for comparing the observed processes with both sensemaking and signature discovery is that signature discovery would be more structured (even, sequential), compared to a more fluid sensemaking process. Below, we describe our findings in more detail.

Comparison to the Signature Discovery Process

Figure 1 describes the signature discovery process presented by Baker et al. [START_REF] Baker | Research towards a systematic sign ature discovery process[END_REF]. Through the interviews, we found that analysts did not follow the linear design of this model. One noticeable difference between the analytic process described by participants and the model of the signature discovery process was the exclusion of certain steps that are present within the signature discovery model. For example, the majority of analysts interviewed did not take inventory of observables; typically the step where data intended for use is defined. In the case of most analysts, the data provided to them for analysis did not require a definition or schema to define the atomic units in the data. Another difference was the lack of assessment and exploration of data, mainly seen amongst the intelligence analysts interviewed. P11 stated that there were "no feature detection models" needed, as the data was structured upon reception by the analyst. P2 and P12 noted how the feature extraction process was not computational and expressed the desire to have "entity extraction in the future." We also saw that there was a lack of explicit description of assessments, detail in the discussion section.

The participants also emphasized the cyclic nature of the analytic process . P5 described the process as the "refinement of a filter to understand the tradeoffs in the data". This cyclic process follows the loop between signature suitability and extrac ting features within the signature discovery process.

Within the Biology domain, we saw that all analysts performed most of the dataintensive tasks within the signature discovery process, mainly feature extraction , and measuring, and collecting data. We found that the Cyber and Data Science analysts did not perform feature extraction for the most part because the data provided to them was typically already structured and mostly quantitative data, so their main concern was to gain insight from what was provided. This is a similar case for the Intelligence analysts. However, Intelligence analysts differ from the other domains in that they identified observables as a critical part in their process . Analysts in all domains came to an endpoint or what could be called a signature, based on the data provided to them. There were a variety of endpoints that came about, such as patterns to be co mpared to databases, feature sets, and points of interest.

There are two forms of outcomes described by the analysts we interviewed that can be described in terms of the signature discovery process. One is a signature, which is a unique or distinguishing measurement, pattern, or collection of data that detects, characterizes, or predicts a target phenomenon of interest. In the case of the Cyber domain, this echoes prior work by Fink et al. that found the final produce of cyber security analysts to be a complex query [START_REF] Fink | Visualizing Cyber Security: Usable Workspaces[END_REF]. Another form of result is a signature system, which describes a sequence of steps taken to attain the insight . For example, P6 stated how the findings attained through the use of scripts, filters, and other data transformations are "not valuable without remembering how all of these steps fit t ogether".

Comparison to the Sensemaking Loop

There are differences based on the sensemaking loop presented by Pirolli and Card [START_REF] Pirolli | Information foraging in information access environments[END_REF], and the processes observed. One difference that was persistent throughout most of the participants was the lack of top-down reassessment steps performed.

Another component of the sensemaking process that was not present in the analytic process of most of our analysts was the steps prior to obtaining a shoebox of ev idence. Most analysts were provided scoped, domain -specific data so there was no need for searching and filtering external, unrelated information. Most analysts did work to approach a presentation as a deliverable in order to find if the analytic process should be continued or if the final result had been sufficiently reached. Kang and Stasko's findings are similar to these findings, in that they identified different categ orizations of analyses: structured and intuitive [START_REF] Kang | Characterizing the intelligence analysis process: Informing visual analytics design through a longitudinal field study[END_REF].

For the Cyber and Data science domain, we found the synthesis stages often lac king. In the Cyber domain, based on their described analytic techniques, there is less information synthesis. Their analytic processes were more focused on foraging. P8 commented that the "findings of our queries are sent off and evaluated" by other co llaborators. Similarly, the participants from the data science domain commented that they were sometimes required to simply forage through data. That is, to transform data from one (or a set of) structured or unstructured sources to a single, clean, unified spreadsheet or database.

The biology and intelligence analysts, on the other hand, performed more of their work in the synthesis part of the sensemaking loop, specifically schematizing. The biologist exhibited this in the form of generating research questions and hypotheses. The intelligence analysts often created hypothetical scenarios and observe how they panned out with respect to the current and new data.

Products and Results of the Analytic Process

We analyzed the data collected from the interviews and observations to determine what constituted as "products" from the analyses of our participants. These are primarily the insights our participants found, and also the repeatable processes that allowed them to acquire the insights. The details below help address our second hypothesis (H2), regarding the outcomes and products.

Insights

A majority of the analysts consider a valuable outcome of their process to be broadly categorized as novel insight. P10 stated that a goal is to "represent data in a meaningful way" and that this is often through the detection of a "pattern found in the data." P8 stated that a valuable finding was the "unique identification for features," which can be considered a derived facet from the original data to gain insight. P4 stated that findings typically come in the form of behavior of extracted features (biomarkers). P9 defined a the valuable insights as a "piece of info that stands out from background" and "that gives you meaning."

We found that the insights of the analysts in our study were also focused on fe atures or characteristics in the data, rather than the data objects themselves. For exa mple, P3 states that characteristics of "a panel of genes"', such as "how it behaves", "a set of features/markers", and "abundance level[s]", are valuable results of analysis. This form of an analytic product maps well onto the prior work of defining what a signature is in different contexts. For example, signatures have b een described as nucleotide sequences [START_REF] Vieira | A novel approach for the identification of bacterial taxa-specific molecular markers[END_REF][19], features of network communication packets [START_REF] Han | Data mining aided signature discovery in network-based intrusion detection system[END_REF], or a set of proteins and their measured abundance levels [START_REF] Bock | Photoaptamer arrays applied to multiplexed proteomic analysis[END_REF].

Repeatable Processes

Analysts also described their findings in terms of the process (rather than the ou tcome). P5 stated that findings are typically a "sequence of up-front data processing, followed by decision making unique to the data" to come about a set of answers . These steps and decisions made are the critical findings , as they help inform collaborators of how the findings were generated. P5 also described the process as "hypothesis generation, not hypothesis answering." P11 stated that 75 percent of the process was processing data through mass spectrometry and matching it to a database , and "the other 25 percent is generating p-values".

Collaboration

The analysts exhibited variations of informal collaboration during their analysis. P6 often performs "pair analytics" [START_REF] Arias-Hernandez | Pair Analytics: Capturing Reasoning Processes in Collaborative Visual Analytics[END_REF], in which P6 and a domain expert work alongside an expert in the area they're performing analy sis for. P3 leverages expert knowledge in combination with "multiple sensor sources to find patterns" with the goal of providing "trained models to get experts further along … to make better distinctions in features." However, there was no explicit collaboration between analysts observed or reported (with the exception of handing off results and steps in a seque ntial manner to collaborators). For example, the "identify and select potential observ ables" step of the signature discovery process was not performed because another analyst had previously completed this step and communicated the resulting data to the next analyst.

Discussion

Sensemaking and Signature Discovery

The entirety of the analytic process can be said to be composed of aspects of both sensemaking and signature discovery. Figure 3 shows a summary of how the analytic processes of analysts interviewed relate to the signature discovery and sensemaking processes based on our data analysis. Full circles indicate steps explicitly mentioned or observed. Half circles indicate steps that were generally mentioned, but not explicitly performed. Empty circles indicate steps that were not observed or mentioned du ring the interviews. As can be seen by these figures, it is rare for individual analysts to conduct every step explicitly.

The signature discovery process is computationally based procedure. From the interviews, it was shown that data processing was a necessary part of the analytic process. The goal of signature discovery is quantitative and well defined, where a signature and signature system are the ideal results of the process . In contrast, sensemaking is primarily a cognitive process and is not as rigidly structured as the signature discovery process. There are more avenues for assessing previously performed tasks in the process compared to signature discovery. As a result, we saw similarities with the sensemaking process in the more synthesis -focused analysis tasks, and more similarities to signature discovery when data-centric methods were used.

Further, while specific steps in the foraging loop map directly to some of the steps in the signature discovery process , a link between the synthesis stages from sens ema king to signature discovery are not as apparent. For example, the task of transitioning from external data sources to an evidence file maps to the steps in the signature discovery process steps of taking inventory of observables to feature extraction. From there, the two processes appear to branch off into independently .

As a result, we found that both models do not encompass the process of analysis across all of the domains fully. The steps they performed were mostly based on the data sources and tasks given. Signature discovery and sensemaking have been identified as independent processes, focused on the computational and cognitive aspects of data analysis, respectively. The differences are also illuminated in observing the result of the models. Signature discovery has the signature as a result, while sensemaking suggest a presentation to share or present findings, processes, and knowledge to someone else. From our study, we find that the distinction between these two is d ependent primarily on the domain. In Cyber Security, for example, a repeatable query or classifier is often desired, whereas intelligence analysis emphasizes the shared knowledge (i.e., the intelligence being discovered) to give to someone else.

Hypotheses and Assessments

The formation and assessment of hypotheses played a central role in the analytic processes of our participants. The position of the hypothesis in the analytic process can vary, depending on the initial starting point of the analysis. Proving or disproving a hypothesis can be the driving force behind the analytic process , or developing a hypothesis can be the purpose of the analysis. It was found that some of the analysts' processes were based on either hypothesis generation or hypothesis verification that has an impact on how the analytic process is conducted. P1 stated the measure of success for their work was "hypothesis confirmation and verifying a prediction," which correlates to the scientific method. P5 stated that analysis was about "hypothesis generation" and P2 stated that the goal for their work is a hypothesis. Among the data scientists interviewed, the placement of hypothesis formation within their analy tic processes significantly varied depending on the question that needed to be asked of the data and the type data being dealt with. Typically, when dealing with cyber data, the Data analysts had a hypothesis near the earlier steps of their process with an end goal of verifying their hypothesis. Hypothesis generation can also arise when there is no specific initial problem. In this case, the analytic process is used to develop more refined problems as opposed to solving a succinct problem or question.

Our participants also exhibited tactics of hypothesis assessment . As seen in Figure 3, many analysts do not explicitly state that they conduct assessments of suitability in signature discovery, or any of the top-down analytical steps within the sensemaking loop. However, during the observation of analysis, it appears these steps are inherent to the analytic process that analysts do not consider them to independently coincide with the signature discovery or sensemaking process. This is reflected in Pirolli and Card's "dual search" activity in the sensemaking model [START_REF] Pirolli | Information foraging in information access environments[END_REF], and may evidence the importance of that aspect of the sensemaking process. Similarly, the analytic proces ses observed can be generally fit either the "top-down" or "bottom-up" terminology [START_REF] Pirolli | Sensemaking Processes of Intelligence Analysts and Possible Leverage Points as Identified Though Cognitive Task Analysis[END_REF], [START_REF] Pirolli | Information foraging in information access environments[END_REF], as hypothesis-driven analysis, or the process of forming hypotheses.

Implications for Design

The results of this study can help inform the design and implementation of systems created to support data analysis. Through gaining a deeper un derstanding of the cognitive and computational processes involved in gaining insights in data -driven domains, we can develop design guidelines and considerations.

There is an inherent importance of capturing and tracking the analytic provenance [START_REF] North | Analy tic provenance: process+ interaction+ insight[END_REF]. Analytic provenance encompasses the interactive steps, intermediate results, and hypotheses considered throughout analysis. The participants of our study commented that recalling their process after a focused data analysis session is difficult. Analytic provenance support in such tools need s to strike a fine balance between allowing users to maintain this "cognitive zone" [START_REF] Green | Building and applying a human cognition model for visual analytics[END_REF] during analysis, and encouraging users to record and annotate their process. Our participants commented that such provenance support should not interfere with the actual data analysis process. Thus, there is an open challenge for the field to determine how formal and explicit such capturing and tracking techniques should be. In comparison, passive capture and interpretation of user interaction may be a more well-suited approach [START_REF] Endert | Semantic Interaction for Visual Analytics: Toward Coupling Co gnition and Computation[END_REF], [START_REF] Brown | Finding Waldo: Learning about Users from their Interactions[END_REF].

Further, the analysts from the domains we studied emphasized the importance of hypotheses during their analytic processes. This included generation, testing, and validation of hypotheses. However, hypotheses were not typically formal (or mathematically grounded). For example, they came in the form of "hunches" or "moments of interest" for following one's curiosity about the data. This also came through in the observations, where the investigators often asked how the analyst decided to try one approach over another. Often, the analysts stated that they did not know, or that they had seen something like it before. This finding echoes the importance of researching and developing more formal methods of tracking hypotheses, but also of evaluating exploratory data analysis tools.

Conclusion

In this paper, we present the results of a user study of professional analysis in four domains (Biology, Cyber Security, Intelligence Analysis, and Data Science). We analyzed the tasks and processes of analysts to get a deeper understanding of the cognitive and computational aspects of data analysis. In order to do this, we conducted a series of semi-structured interviews and analysis observations.

Our results indicate that the data analysis processes exhibited by our study partic ipants was a combination of the known sensemaking and signature discovery models. Generally, the cognitively-focused aspects of synthesizing the information into knowledge was represented in the sensemaking models. In contrast, the computatio nally-focused data foraging and features extraction stages were better represented in the signature discovery model. The diversity of analytic processes, among those we interviewed suggests a combination of sensemaking and signature discovery as a viable model for data-driven analytic processes. These findings are further discussed, including implications for design, which can inform the design and implementation of future data analysis systems.

Figure 1 .

 1 Figure 1. Diagram of the S ignature Discovery Process (adapted from [1]).

Figure 2 .

 2 Figure 2. The sensemaking loop, presented by Pirolli and Card [5], depicting cognitive stages of data analysis.

Figure 3 .

 3 Figure 3. S ummary of how the analytic processes of analysts interviewed relate to the signature discovery and sensemaking processes based on our data analysis. Full circles indicate steps explicitly mentioned or observed. Half circles indicate steps that were ge nerally mentioned, but not explicitly performed. Empty circles indicate steps that were not observed or mentioned during the interviews. The areas of expertise are N=Intelli gence Analyst, B=Biology, C=Cyber S ecurity, and D=Data S cience. P7 was removed for data classification and sensitivity concerns.

Acknowledgements

This research is part of the Signature Discovery Initiative at Pacific Northwest National Laboratory, conducted under the Laboratory Directed Research and Develo pment Program at PNNL, a multi-program national laboratory operated by Battelle for the U.S. Department of Energy.


Christian Kray
email: c.kray@uni-muenster.de

Dennis Wilhelm
email: d.wilhelm@uni-muenster.det.fechner@uni-muenster.de

Thore Fechner

Morin Ostkmap
email: m.ostkmap@uni-muenster.de

Natural Interaction with Video Environments Using Gestures and a Mirror Image Avatar

Keywords: gestural interaction, mirror image, avatar, video

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Motivation

There are many scenarios, where a realistic visual simulation of a real environment is needed: training, gaming, entertainment, or remote collaboration are some examples. In addition, the design and evaluation of ideas and (contextdependent) systems can strongly benefit from such simulations. The design process of ubiquitous systems such as public display networks can also benefit from realistic simulations of the intended deployment area: instead of implementing a fully functional system or deploying screens in the real world, designers can create mock-ups using simulations and thus gather feedback from users early on in the development process. Immersive video environments are one option to convincingly simulate the real world. While they are easy to create and very realistic, they do not include semantic or geometric information. Movement in the depicted 3D space and interaction with objects shown in the footage is thus not realized easily.

In this paper, we propose an approach to overcome these issues. The approach also enables the injection of new content into the video footage and the subsequent experiencing of this content. Our system combines gestural interaction with a mirror image of the user that serves as an avatar within the video environment. It thereby enables the intuitive selection of 3D locations shown in video environments as well as the placement of virtual objects inside the 3D space depicted by the video footage.

Related Work

There are different approaches to realize visually convincing simulations. One is to use virtual environments (VEs), i.e., computer generated scenarios based on a detailed 3D model. Another approach is to use photographs or video footage to generate an immersive experience. Synthetic 3D models allow for fine grained details and interaction, while the actual modeling requires a lot of work. While parts of the process can be automated, e.g., using 3D scanners [START_REF] Izadi | Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera[END_REF], the overall effort is still considerable. Conversely, photographs or video footage provide a realistic (audio-)visual simulation and can be captured quite easily and effortlessly but interaction with the shown objects is limited.

Different means of interaction are available to move within the simulation, e.g., treadmills, and to manipulate objects shown in the simulation (such as gloves or voice commands). Gestures can also be used for this purpose. Särkelä et al. [START_REF] Särkelä | The movement patterns and the experiential components of virtual environments[END_REF] analyzed different gestures and notions to let users navigate in a VE. Vogel and Balakrishnan [START_REF] Vogel | Distant freehand pointing and clicking on very large, high resolution displays[END_REF] define a design space for freehand pointing and clicking interaction. Nancel et al. [START_REF] Nancel | Mid-air pan-andzoom on wall-sized displays[END_REF] investigated how to implement mid-air pan-and-zoom gestures on wall-sized displays. Benko and Wilson [START_REF] Benko | Multi-point interactions with immersive omnidirectional visualizations in a dome[END_REF] analyzed multi-point mid-air gestures for omnidirectional immersive environments.

A key issue in creating immersive experiences is the lack of haptic feedback. Vogel and Balakrishnan suggest to compensate for this by using additional visual and auditory cues [START_REF] Vogel | Distant freehand pointing and clicking on very large, high resolution displays[END_REF]. In general, humans perceive visual stimuli more pronounced than auditory or tactile ones [START_REF] Del Valle | The persuasive mirror: Computerized persuasion for healthy living[END_REF]. One possible approach to provide additional visual feedback is to employ a mirror metaphor. Mirrors are commonly used, and most users are thus familiar with how to operate them. This metaphor has been used in a variety of contexts [START_REF] Ahn | Large display interaction using video avatar and hand gesture recognition[END_REF][START_REF] Iwabuchi | Smart makeup mirror: Computer-augmented mirror to aid makeup application[END_REF][START_REF] Morikawa | Hypermirror: toward pleasant-to-use video mediated communication system[END_REF][START_REF] Del Valle | The persuasive mirror: Computerized persuasion for healthy living[END_REF][START_REF] Vera | Augmented mirror: Interactive augmented reality system based on kinect[END_REF]. Uses include raising awareness of health issues, motivating behavior change, or simply using a mirror image to focus attention during interaction.

Similar to the approach presented by Ahn et al. [START_REF] Ahn | Large display interaction using video avatar and hand gesture recognition[END_REF], the approach proposed in this paper uses the user's mirror image as a video avatar. The avatar can be used to navigate within the VE and to manipulate virtual objects. In contrast to previous work, our system does not merely substitute a cursor with a mirror image. Our approach is focussed on providing an immersive experience by using intuitive gestures in combination with the video avatar. It supports both the creation of augmented scenes, where virtual objects are inserted into video footage and the exploration of such scenes. Gotardo and Price [START_REF] Gotardo | Integrated space: authoring in an immersive environment with 3d body tracking[END_REF] aimed at a similar workflow and developed a system that is comparable to the one presented here. However, their approach is based on a much more complex hardware setup. Moreover, the user interface designed by Gotardo and Price is based on a heads-up-display (HUD) rather than on gestures to let users select actions (e.g., selecting or scaling objects). A purely gestural interface may be experienced as a more natural way to interact with an immersive video environment (IVE), both by designers and participants. In addition, the system proposed here allows for a quick and easy creation of scenes from video footage and does not require custom and expensive hardware.

Approach

The main goals of our approach were to enable intuitive interaction with video environments and to immerse people watching video footage as much as possible into the real world scene being shown. Our aim was thus to enable users to perform various actions in the simulated environment while providing them with a strong feeling of presence. The basic idea underlying our proposed interaction technique is to create a realtime mirror image of the user and overlay it over the video footage. Using a simple, layer-based depth model and a small set of gestures, users can place their mirror avatar inside the depicted real-world scene and interact with the environment via the avatar (e.g., to place virtual objects). In the following paragraphs, we describe all essential components of the approach in detail and provide an overview of our prototypical implementation.

Mirror image avatar

A live mirror image of the user constitutes the focal point of interaction, see Figure 2(a)-(d). The background behind the user, which is also captured by the camera pointed at the user, is eliminated in real-time, e.g., using standard difference image or chroma-key techniques. The cut-out mirror image serves as an avatar or proxy for the user: its location in 3D space defines where interaction can take place.

In particular, the avatar defines the depth layer that is currently selected. Since both the user and the system's depth camera know how tall the user is, the actual size of the avatar as depicted on screen defines its depth position inside the video footage. The size of the avatar is used as a depth cue to inform the user about the layers and objects that can be selected. For example, by placing the avatar on the third layer three, the user can interact with objects located on that layer and can inject virtual objects on that layer.

Gestures

Avatar control. In order to move the avatar within the 3D space defined by the video footage and the layer model, users can perform a number of gestures. When one foot is placed in front of the other, users can move their avatar along the X-, Y-, and Z-axis. To control the movement in X-and Y-direction, users use a one-handed gesture. By extending one arm in one of the cardinal directions, a user can specify in which direction the avatar should move, see Figure 1 (left). For example, extending the arm to the left moves the avatar in that direction. Movement continues while the arm is extended. Movement stops when users bring their arm close to their bodies, or when they perform a different gesture.

A two-handed gesture controls movement along the Z-axis (depth). Putting both hands closely together in front of the body shrinks the avatar. This corresponds to moving it deeper into the image, i.e., it increases its distance from the camera. By spreading both arms, users can increase the size of the avatar and thus decrease its distance to the camera, see Figure 1 (middle). The scaling stops when users either return their arms to a relaxed position, or when they perform another gesture. The size of the avatar determines which depth layer is selected: the depth information specified for each layer and the actual height of the user enables the system to compute the best match, i.e., to find the layer which corresponds best to the current size of the avatar.

In order to evaluate these gestures, we carried out a comparison study that contrasted it with an alternative set of gestures, where movement was controlled by walking in place while orienting one's body in the target direction. Twenty participants (ten male, ten female, average age 22.35 years, SD: 1.927) were recruited via word of mouth and other means from around the university. They were asked to navigate several immersive video scenes in the IVE with the help of the mirror image avatar using two different sets of movement gestures. All participants had to use both methods, but the order of exposure was randomized. Due to space constraints we cannot report on the entirety of the results here but can only summarise the key findings. Participants were largely successful in navigating the avatar to the target locations using either method. We frequently observed people stopping when their avatar reached a street and avoiding "collisions with cars," which indicates a high degree of immersion. The gesture set depicted in Figure 1 was rated more favorably (SUS score of 74.25, SD: 12.76) than the comparison set (SUS score of 64.63, SD: 23.13). 70% of the participants also preferred the static gesture shown in Figure 1 overall. Object manipulation. In addition to moving the mirror image avatar, we defined a set of gestures to select objects depicted in the video footage (e.g., buildings or signs), to inject virtual objects (e.g., public displays, new buildings, or audio sources) as well to move and scale those objects. In addition, gestures to select content to inject into the scene or other content-related activities can be defined. For example, in our prototypical implementation we included gestures to select an item to inject from a list of options.

In order to select an object in the video footage, users first need to place their avatar on the corresponding layer. Users then select an object by simply pointing in the direction of the object. The gestures to place an object in 3D space are the same ones as those used for moving the mirror image avatar. Users can switch between moving their avatar and moving objects by putting their hand together, extending their arms in front of them and then maintaining this pose for a short time, see Figure 1 (right). Visual feedback, i.e., a progress bar, indicates the switch from one set of actions to another.

Experiencing augmented video scenes. Once video footage has been augmented with a number of virtual objects, people can experience the new "scenario" in the following way: by moving their mirror image avatar through the 3D space defined by the video footage and the layer model (as described above), they can interact with the objects augmenting the video scene. The layer model provides means to, for example, measure the distance to a public display and realize proxemic interaction [START_REF] Ballendat | Proxemic interaction: designing for a proximity and orientation-aware environment[END_REF]. The next section describes an example scenario, which demonstrates how users can experience augmented video scenes.

We created a prototypical implementation of our approach using predominately web-based technologies and two cameras (a webcam and a depth camera), which runs in real-time inside a standard browser on a desktop PC. In order to demonstrate the use of our approach, we propose the following example scenario, see Figure 2. Designers want to create a public display system that consists of a series of screens distributed throughout the city. These screens are meant to react to passersby and provide them with personalized information. Instead of installing real displays at the planned deployment sites, short video clips of those sites can be recorded and be used to prototype the system.

The first step is to construct the layer model: for each scene, the designers need to define a number of layers that are located at different distances (depth levels) from the plane defined by the camera. In addition, they can mark up a number of objects shown in the video footage and link them to a specific layer. Layers and objects define how people can later interact with the video footage. Once layers (and optionally objects) are specified, people can interact with the footage. Designers can now use the gestures shown in Figure 1 to move their avatar around the video scene. As they move their avatar, previously defined objects are highlighted, see Figure 2 (a)-indicating, which objects can be "reached" from the current position of the avatar. When designers have positioned their avatar at the desired location, they can point at objects, which are associated with the layer the avatar is located on, see Figure 2 (b). Additionally, the designers can inject virtual objects into the footage. In the example scenario, the designers insert, move, and scale a mockup of a public display, see Figure 2 (c). The virtual object is associated with the layer on which the avatar is located.

The resulting augmented video footage can then be explored by other people, e.g. stakeholders or the people who commissioned the public display system. Figure 2 (d) shows a stakeholder who is exploring the scene that the designers created previously. Using the gestures shown in Figure 1, the stakeholder moves they avatar around the video scene. The public display reacts to the avatar, and when its within its activation area, the display content changes and shows directions targeted at the user. The stakeholder can thus experience the design in a realistic way and explore, for example, whether the activation area of a public display or its content are fit for the intended deployment location.

Discussion

The initial prototypical implementation suffers from a number of limitations. It currently only supports a small set of objects that can be placed inside the video scene, and at the moment, their orientation in space cannot be changed. In addition, only one user can interact with the system at the same time. Furthermore, movement of the avatar is not restricted so that users can place it in physically impossible positions (e.g., floating above ground), which could break immersion. Finally, both the avatar and virtual objects are simply overlaid over the video footage: moving objects such as cars that intersect with these simply disappear behind them regardless of where they are supposed to be in the 3D space defined by the video. This is another aspect that can negatively affect immersion.

Most of these limitations can be addressed by improving the current implementation. A more generic import mechanism for assets to be injected could use a different set of gestures (or a mobile device) to allow for the use of arbitrary virtual objects. A more sophisticated and robust gesture recognition system would allow for rotation gestures and enable multi-user interaction. A more sophisticated layer model could also specify permissible locations of the avatar on each layer to prevent avatars from being moved to physically impossible locations. Realizing physically correct occlusions involving the avatar would require a deeper analysis of the video and a more sophisticated spatial model.

Finally, several limitations relate to the gestures we used. While the gestures we defined were learned quickly by the participants and positively received in our user study, further studies are required to identify the most immersive/intuitive set of gestures. So far, we only assessed a subset of all the gestures, i.e., movement control. In addition, we did not test whether the use of devices, e.g., mobile phones, to carry out certain actions, such as injecting virtual objects, would be more immersive/intuitive, neither on their own nor in combination with gestures. Further studies on these aspects are desirable as well.

Generally speaking, mirror image avatars could also be used with photographs or true 3D environments, i.e., virtual worlds. We chose to use video footage as we expected the real-time motion of the avatar to blend in more naturally with the movement naturally occurring on video footage, and would thus create a strong sense of presence and immersion. Using true 3D environments would allow for correct occlusions but constructing realistic virtual worlds requires a lot of effort. Compared to a desktop scenario, where a user would place objects and experience augmented video scenes, we argue that the gesture-based approach combined with a large screen provides a more realistic and immersive experience. Initial informal feedback from people seeing the system in action as well as observations from the initial user study on the movement gestures seem to confirm this, but we intend to carry out a series of user studies to investigate these aspects in more detail. Clearly, further studies are also needed to identify the most suitable gesture sets for different tasks.

Conclusion

In this paper, we proposed a novel approach to interact with video environments in an immersive and intuitive way. Using their avatar, users can move inside the footage in three dimensions, and place virtual objects inside the scene depicted on the video. Knowledge about the height of the user and the layer model enable the system to place the video avatar in three dimensions. The system can be used for various applications, for example, the prototyping and evaluation of ubiquitous and situated systems. We presented an example scenario, where designers first placed an interactive public display in a video environment and a stakeholder then explored the resulting scenario.

The initial prototype, though limited, used web technologies to illustrate the feasibility of the approach. A first study provided initial evidence for a high degree of immersion and the usability of the proposed approach. Future work will focus on carrying out a series of further user studies. With the latter, we plan to explore properties of mirror image avatars as a means of interaction in simulated environments, to compare this approach to alternatives, e.g., 3D controllers, and to investigate the integration of mobile devices with the system, e.g., as a secondary controller for content selection.

Fig. 1 .

 1 Fig.1. Key gestures used to control the avatar, interact with the video scene, and to manipulate virtual objects: moving gestures (left), scaling gesture (middle), switching gesture (right).

Fig. 2 .

 2 Fig. 2. Example scenario: (a) moving and scaling the avatar, (b) selecting scene element, (c) placing virtual object, (d) experiencing virtual object (e.g., public display).