Johannes Fuchs

Petra Isenberg

Anastasia Bezerianos

Daniel Keim

A Systematic Review of Experimental Studies on Data Glyphs

Keywords: Survey, Glyphs, Quantitative Evaluation, Glyph Design. !

We systematically reviewed 64 user-study papers on data glyphs to help researchers and practitioners gain an informed understanding of tradeoffs in the glyph design space. The glyphs we consider are individual representations of multi-dimensional data points, often meant to be shown in small-multiple settings. Over the past 60 years many different glyph designs were proposed and many of these designs have been subjected to perceptual or comparative evaluations. Yet, a systematic overview of the types of glyphs and design variations tested, the tasks under which they were analyzed, or even the study goals and results does not yet exist. In this paper we provide such an overview by systematically sampling and tabulating the literature on data glyph studies, listing their designs, questions, data, and tasks. In addition we present a concise overview of the types of glyphs and their design characteristics analyzed by researchers in the past, and a synthesis of the study results. Based on our meta analysis of all results we further contribute a set of design implications and a discussion on open research directions.

INTRODUCTION

R EPRESENTING multi-dimensional data is a common task in data visualization and, thus, a multitude of techniques has been developed [START_REF] Munzner | Visualization Analysis & Design[END_REF]. Data glyphs are one such technique, in which single data points are encoded individually by assigning their dimensions to one or more marks and their visual variables. Data glyphs have a long history, going back to the 1950s, with metroglyphs being one of the first designs using line length to encode data [START_REF] Anderson | A Semigraphical Method for the Analysis of Complex Problems[END_REF]. A somewhat infamous, and thus well researched, example of data glyphs are Chernoff faces [START_REF] Chernoff | The Use of Faces to Represent Points in K-Dimensional Space Graphically[END_REF] which encode data values in 2D facial features such as the length of the nose or the orientation of eyebrows. Star Glyphs are another type of glyphbased encoding that has received research attention [START_REF] Siegel | The Surgical Implications of Physiologic Patterns in Myocardial Infarction Shock[END_REF] and has been used in various applications [START_REF] Peng | Clutter Reduction in Multi-Dimensional Data Visualization Using Dimension Reordering[END_REF], [START_REF] Yang | Interactive Hierarchical Dimension Ordering, Spacing and Filtering for Exploration of High Dimensional Datasets[END_REF], [START_REF] Chan | The Generalized Sensitivity Scatterplot[END_REF].

Over the years, many different glyph variations were introduced to better fit certain data types, or to solve specific tasks more effectively. Yet, while many designs have already been explored, the mapping possibilities of data dimensions to visual glyph encodings is nearly endless [START_REF] Munzner | Visualization Analysis & Design[END_REF] and many more designs are certainly imaginable. This flexibility allows designers to come up with new and innovative glyph representations for specific data, tasks, or contexts. However, without any guidance, this freedom and large design space can become overwhelming. Knowledge of when and which types of designs work best or are preferred by viewers, could aid designers and practitioners in creating new designs or in selecting among existing ones. Yet, while many user studies in the literature have investigated different data glyph designs and their variations, a systematic overview of these studies and what they pertain to, is so far missing.

With this paper we contribute such a systematic overview of the user-study literature on data glyphs, focusing on the analysis of 64 papers with quantitative controlled studies. In contrast to their qualitative counterpart, controlled experiments are more easily comparable and summarizable, as they test concrete hypotheses regarding design choices and isolate factors in the glyph designs [START_REF] Cunningham | Experimental Design: From User Studies to Psychophysics[END_REF]. We focus on data glyphs that: use different visual channels to encode multiple attribute dimensions, that can be independently spatially arranged, and that can vary in size (see Section 3.1). We categorize the studies we found according to a number of criteria that are meant to help researchers and practitioners choose amongst the most relevant literature to read, and ultimately to make informed choices about glyph use, design, and potential future studies. These criteria include glyph types (see Fig. 1 for an overview), presentation settings, datasets, tasks, and study goals. We extend this characterization with a summary of study outcomes to help practitioners select the most appropriate data glyphs according to different criteria like visual design, data density, or task. By tabulating existing studies according to these criteria we provide a novel user-study-centric view on the design space and study outcomes, and contribute a discussion on open research areas that will help to further our understanding of data glyphs.

RELATED WORK

In this section we provide an overview of glyph-related surveyor meta-papers that, similar to ours, have attempted to structure existing glyphs according to various criteria. Since a large number of glyph designs have been proposed in the past, a tabulation of existing individual designs is beyond the scope of this section.

Ward [START_REF] Ward | A taxonomy of glyph placement strategies for multidimensional data visualization[END_REF], [START_REF] Ward | Multivariate data glyphs: Principles and practice[END_REF] was one of the first to provide a structure of the glyph design space by classifying different layout possibilities into data-driven and structure-driven layouts. In a data-driven arrangement spatial position is determined by data: this can either be the raw data used as in a scatterplot, or a projection-based approach such as PCA. A structure-driven layout makes use of relations between the data points to calculate a layout. For example, hierarchical information can be used to lay out glyphs in a tree structure. Ward extended this work with a categorization of the visual characteristics of data glyphs. He structured data glyphs based on their mapping of data to visual attributes into three different classes: a many-to-one mapping where each data dimension is mapped to the same visual variable (e.g., profile glyph [START_REF] Du Toit | Graphical Exploratory Data Analysis[END_REF]); a one-to-one mapping showing each data attribute with a different visual variable (e. g., Chernoff faces [START_REF] Chernoff | The Use of Faces to Represent Points in K-Dimensional Space Graphically[END_REF]); and a one-tomany mapping representing the data dimensions redundantly with many different visual attributes (e. g., compound glyph [START_REF] Pearlman | Visualizing Network Security Events using Compound Glyphs from a Service-oriented Perspective[END_REF]). In this paper, we use this categorization to structure our own categorization of data glyph user-studies.

In contract, Chung et al. [START_REF] Chung | Glyph-based multi-field visualization[END_REF] proposed a categorization based on the visual channels used to represent the data and the spatial dimensionality of the glyph (2D, 2.5D, and 3D). The authors also discussed critical design aspects and guidelines for glyph visualizations, such as the normalization of data input for each dimension, the use of redundant mappings, and the visual orthogonality of different glyph components to ensure best performance. Since some of these guidelines cannot be followed for a high number of dimensions, designers have to choose between few single complex glyph designs, or many simple designs. Additionally, they suggested using halos to limit the negative effect of overplotting. In our survey we extend this list of guidelines based on our review of experimental results and provide further open research questions.

An extensive survey on data glyphs was presented by Borgo et al. [START_REF] Borgo | Glyph-based Visualization: Foundations, Design Guidelines, Techniques and Applications[END_REF]. The authors cover different glyph representations and propose guidelines for designing data glyphs based on a collection of design principles in the literature. While Borgo et al. also include several empirical studies in their survey, their focus is on design study papers showing the applicability of data glyphs to different data sets and tasks. In contrast to this work, we provide an overview of performance assessments from quantitative user studies.

A more data-specific survey on glyphs in the medical domain was presented by Ropinski et al. [START_REF] Ropinski | Survey of Glyph-based Visualization Techniques for Spatial Multivariate Medical Data[END_REF]. The authors classified glyph-based visualizations for medical data into two groups: preattentively and attentively identifiable glyph designs. Based on this grouping the authors further derive design guidelines for developing glyphs for this domain, but provide no additional empirical results from user studies.

While there is no systematic assessment of glyph user-studies that we know of, some researchers have categorized subsets of the study design space. Nelson [START_REF] Nelson | The Face Symbol: Research Issues and Cartographic Potential[END_REF], for example, discusses the history of Chernoff faces [START_REF] Chernoff | The Use of Faces to Represent Points in K-Dimensional Space Graphically[END_REF] with its many variations such as the Flury-Rydwiel [START_REF] Flury | Graphical Representation of Multivariate Data by Means of Asymmetrical Faces[END_REF] or Kabulov faces [START_REF] Kabulov | A Method for Constructing Chernoff Faces Oriented Toward Interval Estimates of the Parameters[END_REF]. She also discusses studies investigating performance changes for different data types or visual variations. We took this work as inspiration, but provide a much more comprehensive view on the study design space. Ware's [Ch. 5] [19] discussion on "Glyphs and Multivariate Discrete Data" is related to our work in that he categorizes two types of user study tasks for glyphs. He focuses on tasks designed to find out which display dimensions are perceived holistically (integral) or perceived separately (separable): restricted classification tasks and speeded classification tasks. Among others, we include both types of tasks in our discussion based on slightly different terminology [START_REF] Andrienko | Exploratory Analysis of Spatial and Temporal data[END_REF]: similarity search tasks (related to restricted classification) and lookup tasks (similar to speeded classification).

In summary, while there are several overview papers on data glyphs, we know of no prior work, such as ours, that provides a systematic review of glyph user studies, study designs, and results.

METHODOLOGY

For our systematic review we focused on sampling user studies in which participants performed controlled, quantitatively measured tasks with data glyphs. These quantitative measurements could (but did not have to) be accompanied by a subjective assessment of the tested glyphs (e. g., according to aesthetics, confidence, etc.) Next, we categorized the found studies according to the criteria discussed in this section.

Data Glyphs: Sampling Characteristics

During our previous work on data glyphs we noticed that definitions of data glyphs vary in the literature. Yet, we needed to base our systematic literature search on a more formal, grounded understanding of existing data glyph definitions. We, thus, extracted definitions from survey-and meta-papers on glyphs, several books on visualization, many papers in the literature that used the word "glyph," and interviewed several visualization experts on their understanding of the term glyph.

We found that in early years the term "glyph" was often used as a synonym for the metroglyph [START_REF] Du Toit | Graphical Exploratory Data Analysis[END_REF], [START_REF] Jacob | Facial Representation of Multivariate Data[END_REF] but that with increasing numbers of different glyph designs published, more abstract definitions of a data glyph emerged. To compare how researchers have defined the term, we extracted descriptive keywords from published definitions, and summarized in which publications they appeared in Table 1. While Table 1 is certainly not exhaustive, it serves to show the wide variety of ways researchers think of, and define, data glyphs, an observation corroborated recently by Munzner [START_REF] Munzner | Visualization Analysis & Design[END_REF].

After a careful assessment of these definitions, we approached the paper sampling as follows: first, we included all user studies that used the word "glyph". In a second step we excluded all papers which used the term glyph as a synonym for simple data marks (e.g., [START_REF] Miller | Interactive visualization of intercluster galaxy structures in the horologium-reticulum supercluster[END_REF], [START_REF] Micallef | Assessing the effect of visualizations on bayesian reasoning through crowdsourcing[END_REF]), such as points in a scatterplot. We chose this exclusion criteria because we wanted to focus our assessment on glyphs that encode multiple attribute dimensions at once. This notion of a data glyph as a multi-dimensional encoding also aligns with the majority of historical data glyph definitions. We, thus, chose studies in which glyphs fit the following general definition:

Data glyphs are data-driven visual entities, which make use of different visual channels to encode multiple attribute dimensions. They can be independently spatially arranged and can vary in size.

After this filtering we categorized the tested glyph designs according to the main descriptive criteria we found in the literature:

Data glyph characteristics small/compact encodes different attribute dimensions

uses different visual channels [14] X X X [24] X X - [25] X X - [26] - X X [27] - X X [28] - X X [29] - X X [1] - X X [15] - X X [19] - X X [10] - X - [30] - - X [31] - - X
visual channels, encoded dimensions, and presentation size (see Table 1).

Paper Sampling and Collection

To find relevant papers for our review we used a snowball sampling technique in which we first searched for the keyword "glyph" in the title, abstract, and keywords in the the ACM digital library (leading to 80 potential results), the IEEE Xplore digital library (leading to 255 potential results), the EG digital library (leading to 66 potential results), and the DBLP computer science bibliography (leading to 134 potential results).

In a next step we excluded papers that did not include at least one user study with quantitative measures or did not study glyphs that fit our definition. This filtering step removed 505 of the 535 candidate papers, leaving 30 relevant papers for our survey. From this initial set of papers we recursively scanned references for further user studies about data glyphs. Using this approach we collected 64 papers from the visualization literature as well as work from statistics and psychology.

Analyzed Study Characteristics

In the design of any quantitative user study several characteristics are important: the tasks to be performed, the collected measures, the presentation of the stimuli (glyphs), the size and type of data visualized, the general presentation setting, and the study goals (or main research questions) [START_REF] Cunningham | Experimental Design: From User Studies to Psychophysics[END_REF]. We categorized the 64 study papers using these characteristics as explained in more detail next.

Glyph Types and Data Encoding

We used Ward's data mapping taxonomy [START_REF] Ward | Multivariate data glyphs: Principles and practice[END_REF] to distinguish between glyphs using many-to-one and one-to-one mappings (see Section 2). The rows and columns of Fig. 3 give an overview of this categorization. We only found two occurrences of Ward's third group: one-to-many mappings. Thus, we do not highlight this group as a category in our result table. The two studies we found ( [START_REF] Dick | Distance Visualization for Interactive 3D Implant Planning[END_REF] and [START_REF] Klippel | Colour-Enhanced Star Plot Glyphs: Can Salient Shape Characteristics Be Overcome?[END_REF]) are, however, discussed throughout the paper.

Since the many-to-one group encodes multiple data point dimensions using the same visual variable, we further split this group into categories based on the visual variables used: position/length, color saturation, and orientation/angle (see Fig. 3).

We also distinguished whether or not a linear or circular layout was chosen to lay out the dimensions.

The category of one-to-one mapping was structured slightly differently as it includes a wide variety of design choices. As we mostly found facial glyph representations or three-dimensional designs, the result table includes these two categories: Faces and 3D Glyphs. A third category on car glyphs was added, since in one paper [START_REF] Surtola | The Effect of Data-Relatedness in Interactive Glyphs[END_REF] faces were compared against unique car glyph representations. Car glyphs are abstract two dimensional representations of vehicles, which use unique characteristics (size of the trunk or hood) to encode data.

We additionally found twelve studies that tested unique glyph designs that were not compared to alternative representations: PlanningLines [START_REF] Aigner | Planninglines: novel glyphs for representing temporal uncertainties and their evaluation[END_REF], weather vanes [START_REF] Martin | Results of a User Study on 2D Hurricane Visualization[END_REF], [START_REF] Pilar | Representing flow patterns by using streamlines with glyphs[END_REF], shapes [START_REF] Yost | Single complex glyphs versus multiple simple glyphs[END_REF], [START_REF] Healey | Large datasets at a glance: Combining textures and colors in scientific visualization[END_REF], roses [START_REF] Li | Metaphoric transfer effect in information visualization using glyphs[END_REF], themes [START_REF] Chan | Visualizing the semantic structure in classical music works[END_REF], arrows [START_REF] Wittenbrink | Glyphs for visualizing uncertainty in vector fields[END_REF], Motifs [START_REF] Dunne | Motif simplification: Improving network visualization readability with fan, connector, and clique glyphs[END_REF], [START_REF] Cayli | Glyphlink: An interactive visualization approach for semantic graphs[END_REF], flowers [START_REF] Chau | Visualizing Web Search Results Using Glyphs: Design and Evaluation of a Flower Metaphor[END_REF], and MILSTD2525 glyphs [START_REF] Siva | Evaluation of MILSTD 2525 Glyph Features in a Visual Search Paradigm[END_REF]. Rather they were either compared against textual information, tested on varying backgrounds (changes in the topological level of detail), or against different types of visualizations. Since they were not compared to other designs in the table, we positioned them slightly apart in the "One-to-One Mapping" category.

Glyph Presentation Setting

For the examined studies, we categorized how many glyphs were presented to a viewer on the screen: individual glyphs, multiple glyphs of fixed number, or multiple glyphs of varying numbers. In the category of multiple glyphs we further noted how the glyphs were arranged on the screen, as grids, scatterplots, node-link diagrams, on geographic maps, or other layouts.

Datasets

The glyphs used in the studies all encoded either multi-dimensional data of a general nature, or time-series data. Additionally, we noted how many dimensions a glyph encoded. The number of dimensions is related to the visual complexity of a glyph. Independent of data type and density we further recorded whether the data was synthetically created, or if real data was used in the study.

Tasks and Measures

Important for understanding any study results is the nature of the task participants had to perform. We group tasks in broad categories, differentiating between tasks involving the glyph as a whole (synoptic tasks [START_REF] Andrienko | Exploratory Analysis of Spatial and Temporal data[END_REF]) and tasks where participants had to focus on single specific characteristics of a glyph (elementary tasks [START_REF] Andrienko | Exploratory Analysis of Spatial and Temporal data[END_REF]). An elementary task is typically a lookup task during which participants focus on single dimensions of a glyph and read individual values.

We further subdivided synoptic tasks into three categories: 1) visual search where participants had to find a glyph differing from others, or tell whether a specific glyph is present or not; 2) similarity search where participants had to compare the overall structure of glyphs and group similar representations; and 3) trend detection tasks where participants had to keep track of the development of data values across dimensions.

Study Goals

We found three different general study goals: 1) a comparison of various glyph designs according to their performance and a ranking of designs based on it; 2) a comparison of different variations of a single glyph, to detect visual features improving a specific glyph design; and 3) a comparison of single glyphs vs. data tables, to motivate the use of these visual objects over textual representations.

Study Results

We summarized study outcomes on a high level, reporting findings on the impact of presentation settings, number of data points and dimensions on the tested glyphs. We further report overall ranking of different glyph types, offering explanations to seemingly contradictory results across studies. We do not enter into detail on findings regarding variations of a single glyph type. Our goal is to provide researchers and practitioners with a better grasp of the overall picture of the performance of different glyphs, and to point to individual papers for detailed study results.

RESULTS: STATE-OF-THE-ART IN GLYPH EVALUA-TION

In this section we discuss the findings from our systematic review based on the characteristics discussed in the previous section. A summary of the results is presented in individual tables and in highlighted paragraphs throughout the section. Many study descriptions did not include all information needed for our characterization and subsequently our counts do not always add up to 64-the total number of papers examined.

Study Goals

We found three higher-level study goals-all related to different types of comparisons: a) comparison of glyph designs, b) comparison of glyph variations, and c) comparison of glyphs with data tables or text. As can be seen in the diagonal of Fig. 3 1 , the majority of studies (39/64, 60.94%) tested case b) or c). Design variations within a glyph category were more frequently tested against each other (32/39, 82.05%) than glyphs vs. a common data table or text description (7/39, 17.95%). The latter group was most often used to motivate the use of visuals over text descriptions or data tables [START_REF] Jacob | Facial Representation of Multivariate Data[END_REF], [START_REF] Boulette | Classification of Multidimensional Data under Time Constraints: Evaluating Digital and Configural Display Representations[END_REF], [START_REF] Moriarity | Communicating Financial Information Through Multidimensional Graphics[END_REF], [START_REF] Stock | Human Judgment Accuracy, Multidimensional Graphics, and Humans Versus Models[END_REF], [START_REF] Saxena | Comparison of Chernoff-type Face and Non-Graphical Methods for Clustering Multivariate Observations[END_REF].

To measure participant performance all studies but one [44] recorded accuracy scores, additionally 65.63% measured completion time (42/64), and 29.69% collected qualitative feedback (19/64) as well. It is interesting to note that participants' preferences did not always match with their performance [START_REF] Caban | An evaluation of visualization techniques to illustrate statistical deformation models[END_REF], [START_REF] Fuchs | The Influence of Contour on Similarity Perception of Star Glyphs[END_REF], [START_REF] Weigle | Visualizing Intersecting Surfaces with Nestedsurface Techniques[END_REF], [START_REF] Feng | Evaluation of Glyph-based Multivariate Scalar Volume Visualization Techniques[END_REF]. Therefore, a preferred design was not always a guarantee for a good user performance.

1. The flower [START_REF] Chau | Visualizing Web Search Results Using Glyphs: Design and Evaluation of a Flower Metaphor[END_REF] and theme [START_REF] Chan | Visualizing the semantic structure in classical music works[END_REF] glyphs also fit into this category, however, due to their visual encoding they are not represented on the diagonal Summary: We found similar study goals across many experiments, yet varied were factors like number of data points and dimensions, task, or glyph design. These variations make individual study outcomes hard to compare. Thus, we will discuss the individual factors in the following sections before discussing the study outcomes in Section 4.6.

Glyph Types and Data Encoding

Fig. 3 summarizes evaluated glyph types and their encodings based on Ward's data mapping taxonomy outlined in Section 3. The table is meant to be read like a matrix. The intersections of rows and columns show which glyph types and encodings a particular study compared against each other. The diagonal (top left to bottom right) of the table contains references to studies that tested design variations of the same glyph category, or an evaluation of one specific design against plain text or data tables (marked with a * in the table). Empty cells indicate new research possibilities.

Fig. 2 shows that face glyphs were evaluated most frequently (39.06%), followed by glyphs with position/length encodings (linear: 17.19%, circular: 18.75%), and 3D glyph designs (14.06%). We note that from the studies involving position/length encodings or 3D glyphs (27 in total), 8 were in fact compared to faces (Fig. 3). Color (linear: 9.38%, circular: 3.13%) and orientation encodings (linear: 3.13%, circular: 0%) have received little research attention.

The high number-28.13%-of user studies on face variations , stands out compared to studies that only focus on other variations, e. g., circular position/length encodings (14.06%), 3D glyphs (7.81%), linear color (6.25%), or linear orientation encoding (1.56%). A possible reason for this imbalance are the many ways one can design faces and their data mappings (e. g., Chernoff faces [START_REF] Chernoff | The Use of Faces to Represent Points in K-Dimensional Space Graphically[END_REF], Rydwiel-Flury faces [START_REF] Flury | Graphical Representation of Multivariate Data by Means of Asymmetrical Faces[END_REF], Kabulov faces [START_REF] Kabulov | A Method for Constructing Chernoff Faces Oriented Toward Interval Estimates of the Parameters[END_REF]).

We found only two studies [START_REF] Albers | Task-driven evaluation of aggregation in time series visualization[END_REF], [START_REF] Goffin | Exploring the effect of word-scale visualizations on reading behavior[END_REF] that compared different linear position/length design variations. This is an interesting research gap given that profile glyphs that use this encoding are well established in practice (i. e., sparklines [START_REF] Tufte | Beautiful Evidence[END_REF], profiles [START_REF] Du Toit | Graphical Exploratory Data Analysis[END_REF]).

In general, we only found three main categories of visual variables used to encode data in glyphs with many-to-one mappings (Position/Length, Color, Orientation). Almost all glyph designs in these studies mapped quantitative information to visual variables. The only exception was Lee et al.'s work [START_REF] Lee | An Empirical Evaluation of Chernoff Faces, Star Glyphs, and Spatial Visualizations for Binary Data[END_REF] which compares star glyphs, faces, and 2D projected data points using bivariate data. Here bivariate information, however, was still mapped either to the length of the whiskers (star glyph) or to different face characteristics.

Summary: Faces and circular profiles have been investigated in detail, in contrast to color value and orientation encodings on glyphs that only few studies investigated. Surprisingly, we found only two studies comparing different variations of linear profiles.

Glyph Presentation Settings

Presentation settings can be characterized by the number of glyphs presented to viewers, as well as by how the glyphs are layed out in space. We identified three types of studies when considering the number of glyphs presented (Table 3): those that presented only individual glyphs to the viewers (7/64, 10.94%), those that presented a fixed number of more than one glyph at a time (46/64, 71.88%), and those in which the number of presented glyphs varied but was always higher than one (11/64, 17.19%). Seven papers did not report the exact number of glyphs represented on the screen: [START_REF] Aigner | Planninglines: novel glyphs for representing temporal uncertainties and their evaluation[END_REF], [START_REF] Pilar | Representing flow patterns by using streamlines with glyphs[END_REF], [START_REF] Chan | Visualizing the semantic structure in classical music works[END_REF], [START_REF] Dunne | Motif simplification: Improving network visualization readability with fan, connector, and clique glyphs[END_REF], [START_REF] Cayli | Glyphlink: An interactive visualization approach for semantic graphs[END_REF], [START_REF] Chau | Visualizing Web Search Results Using Glyphs: Design and Evaluation of a Flower Metaphor[END_REF], [START_REF] Caban | An evaluation of visualization techniques to illustrate statistical deformation models[END_REF].

One-to-One Mapping Many-to-One Mapping Since PlanningLines [START_REF] Aigner | Planninglines: novel glyphs for representing temporal uncertainties and their evaluation[END_REF], weather vanes [START_REF] Martin | Results of a User Study on 2D Hurricane Visualization[END_REF], [START_REF] Pilar | Representing flow patterns by using streamlines with glyphs[END_REF], shapes [START_REF] Yost | Single complex glyphs versus multiple simple glyphs[END_REF], [START_REF] Healey | Large datasets at a glance: Combining textures and colors in scientific visualization[END_REF], roses [START_REF] Li | Metaphoric transfer effect in information visualization using glyphs[END_REF], themes [START_REF] Chan | Visualizing the semantic structure in classical music works[END_REF], arrows [START_REF] Wittenbrink | Glyphs for visualizing uncertainty in vector fields[END_REF], Motifs [START_REF] Dunne | Motif simplification: Improving network visualization readability with fan, connector, and clique glyphs[END_REF], [START_REF] Cayli | Glyphlink: An interactive visualization approach for semantic graphs[END_REF], flowers [START_REF] Chau | Visualizing Web Search Results Using Glyphs: Design and Evaluation of a Flower Metaphor[END_REF], and MILSTD2525 glyphs [START_REF] Siva | Evaluation of MILSTD 2525 Glyph Features in a Visual Search Paradigm[END_REF] use a unique encoding and are not compared to other glyphs, we positioned them slightly apart in the "One-to-One Mapping" category.

For the 46 studies that tested a fixed number of multiple glyphs at a time, we found five types of layouts. The most frequent was a common small-multiples grid (65.22%), followed by geographic maps (17.39%), scatterplots (6.52%), node-link diagrams (4.35%), and other layouts (6.52%) like different 3D environments (see Fig. 4).

The goal of most of the studies with varying number of glyphs was to investigate changes in performance when increasing the number of visible data points in grid layouts [START_REF] Morris | Experimental Analysis of the Effectiveness of Features in Chernoff Faces[END_REF], [START_REF] Siva | Visual Search Efficiency for Features in Chernoff Faces[END_REF], [START_REF] Sivagnanasundaram | Evaluation of the presence of a face search advantage in chernoff faces[END_REF], geographic maps [START_REF] Wittenbrink | Glyphs for visualizing uncertainty in vector fields[END_REF], [START_REF] Nelson | Visual Search Processes and the Multivariate Point Symbol[END_REF], and node-link diagrams [START_REF] Wu | Visualizing multivariate networks: A hybrid approach[END_REF]. The amount of glyphs visible to participants changed from 5-50 [START_REF] Morris | Experimental Analysis of the Effectiveness of Features in Chernoff Faces[END_REF]; 5-15 [START_REF] Siva | Visual Search Efficiency for Features in Chernoff Faces[END_REF], [START_REF] Sivagnanasundaram | Evaluation of the presence of a face search advantage in chernoff faces[END_REF]; 6-18 [START_REF] Siva | Evaluation of MILSTD 2525 Glyph Features in a Visual Search Paradigm[END_REF]; 9-23 [START_REF] Nelson | Visual Search Processes and the Multivariate Point Symbol[END_REF]; 4-300 [START_REF] Wittenbrink | Glyphs for visualizing uncertainty in vector fields[END_REF]; and 30-48 [START_REF] Wu | Visualizing multivariate networks: A hybrid approach[END_REF]. In all seven studies participants were affected negatively by an increasing number of data points, as we discuss in Section 4.6.2. In the studies conducted by Aigner et al. [START_REF] Aigner | Planninglines: novel glyphs for representing temporal uncertainties and their evaluation[END_REF], Dunne et al. [START_REF] Dunne | Motif simplification: Improving network visualization readability with fan, connector, and clique glyphs[END_REF], Cayli et al. [START_REF] Cayli | Glyphlink: An interactive visualization approach for semantic graphs[END_REF], and Zhang et al. [START_REF] Zhang | Glyphbased comparative visualization for diffusion tensor fields[END_REF] the varying number of data points was not treated as a factor in the analysis.

In seven studies (10.94%) a single data point was shown to participants at a time [START_REF] Casey | Visual Display Representation of Multidimensional Systems: The Effect of System Structure: And Display Integrality[END_REF], [START_REF] Jones | The Display of Multivariate Information: An Experimental Study of an Information Integration Task[END_REF], [START_REF] Macgregor | Graphic Representation of Judgmental Information[END_REF], [START_REF] Albers | Task-driven evaluation of aggregation in time series visualization[END_REF], [START_REF] Goffin | Exploring the effect of word-scale visualizations on reading behavior[END_REF], [START_REF] Boulette | Classification of Multidimensional Data under Time Constraints: Evaluating Digital and Configural Display Representations[END_REF], [START_REF] Jankun-Kelly | An Evaluation of Glyph Perception for Real Symmetric Traceless Tensor Properties[END_REF]. These studies tried to control all parameters and avoid possible confounding factors, so as to better reason about changes in performance when modifying specific aspects of the same design [START_REF] Jankun-Kelly | An Evaluation of Glyph Perception for Real Symmetric Traceless Tensor Properties[END_REF], or when comparing it with other representations [START_REF] Casey | Visual Display Representation of Multidimensional Systems: The Effect of System Structure: And Display Integrality[END_REF], [START_REF] Jones | The Display of Multivariate Information: An Experimental Study of an Information Integration Task[END_REF], [START_REF] Macgregor | Graphic Representation of Judgmental Information[END_REF], [START_REF] Albers | Task-driven evaluation of aggregation in time series visualization[END_REF], [START_REF] Boulette | Classification of Multidimensional Data under Time Constraints: Evaluating Digital and Configural Display Representations[END_REF], or layouts [START_REF] Goffin | Exploring the effect of word-scale visualizations on reading behavior[END_REF].

Regardless of whether a fixed or changing number of glyphs was tested, the vast majority of studies (56.25%) arranged glyphs in a grid layout, followed by geographic arrangements (15.63%), nodelink diagrams (7.81%), and scatterplots (4.69%) (see Fig. 4). The choice of a grid layout for quantitative studies is understandable. Grids can help to avoid confounding factors in visual search, comparison, or classification tasks. For example, the information provided by a background, e. g., an underlying geographic map, may influence the perception of glyphs. The background color, for example may influence the perception color hues [START_REF] Stone | A field guide to digital color[END_REF], while topology (e. g., rivers, mountains, land borders) may act as grouping enclosures or as reference structures for reading data values of glyphs. We only found a single study [START_REF] Martin | Results of a User Study on 2D Hurricane Visualization[END_REF] that examined the influence of reading data glyphs with different geographic backgrounds; and one [START_REF] Healey | Large datasets at a glance: Combining textures and colors in scientific visualization[END_REF] that studied how the reading of a glyph is affected by the presence of other glyphs around it. We discuss their results in Section 4.6.1.

Summary: Only a small number of user studies varied the amount of data glyphs as a study factor. Most studies were conducted with a fixed number of glyphs arranged in a grid layout. Surprisingly, only four papers investigated the influence of different background information and layout on reading data glyphs [START_REF] Martin | Results of a User Study on 2D Hurricane Visualization[END_REF], [START_REF] Healey | Large datasets at a glance: Combining textures and colors in scientific visualization[END_REF], [START_REF] Goffin | Exploring the effect of word-scale visualizations on reading behavior[END_REF], [START_REF] Frisson | Audiometro: Directing search for sound designers through content-based cues[END_REF].

Datasets & Number of Dimensions

The number of data dimensions tested can help us compare results across studies, and inform us of the imagined use-case setting for data glyphs. Only four studies (6.25%) used the number of dimensions itself as a study factor and thus varied between glyphs with different dimension counts [START_REF] Fuchs | Evaluation of alternative glyph designs for time series data in a small multiple setting[END_REF], [START_REF] Wilkinson | An Experimental Evaluation of Multivariate Graphical Point Representations[END_REF], [START_REF] Fuchs | The Influence of Contour on Similarity Perception of Star Glyphs[END_REF], [START_REF] Wu | Visualizing multivariate networks: A hybrid approach[END_REF]. The remaining 60 studies tested glyphs with various fixed numbers

Dataset Availability

Anthropometrical data about twins [START_REF] Flury | Graphical Representation of Multivariate Data by Means of Asymmetrical Faces[END_REF], [START_REF] Hamner | Comparisons of Several Graphical Methods for Representing Multivariate Data[END_REF] [95]

Patients rated by psychiatrists [START_REF] Jacob | Facial Representation of Multivariate Data[END_REF] Minnesota Multiphasic Personality Inventory

Medical data [START_REF] Dick | Distance Visualization for Interactive 3D Implant Planning[END_REF] unknown

Medical images [START_REF] Caban | An evaluation of visualization techniques to illustrate statistical deformation models[END_REF] unknown Cars dataset [START_REF] Surtola | The Effect of Data-Relatedness in Interactive Glyphs[END_REF] http://davis.wpi.edu/xmdv/datasets/cars.html

Project plans [START_REF] Aigner | Planninglines: novel glyphs for representing temporal uncertainties and their evaluation[END_REF] unknown MM5 weather information [START_REF] Martin | Results of a User Study on 2D Hurricane Visualization[END_REF] unknown

Weather information [START_REF] Pilar | Representing flow patterns by using streamlines with glyphs[END_REF] NCEP forecast model Weather information [START_REF] Wittenbrink | Glyphs for visualizing uncertainty in vector fields[END_REF] Operational Regional Atmospheric Prediction System Modified U.S. census data [START_REF] Yost | Single complex glyphs versus multiple simple glyphs[END_REF] unknown

Financial data [START_REF] Li | Metaphoric transfer effect in information visualization using glyphs[END_REF] Investment in education USA (2008)

Financial data [START_REF] Moriarity | Communicating Financial Information Through Multidimensional Graphics[END_REF] Wall Street Journal Index (1974 and 1975) Financial data [START_REF] Stock | Human Judgment Accuracy, Multidimensional Graphics, and Humans Versus Models[END_REF] Standard and Poor's firm list (1974 and 1975) Classical music data [START_REF] Chan | Visualizing the semantic structure in classical music works[END_REF] unknown

Network data [START_REF] Dunne | Motif simplification: Improving network visualization readability with fan, connector, and clique glyphs[END_REF] Lostpedia wiki edits

Network data [START_REF] Cayli | Glyphlink: An interactive visualization approach for semantic graphs[END_REF] 10 best ranked movies (IMDb)

Google search results [START_REF] Chau | Visualizing Web Search Results Using Glyphs: Design and Evaluation of a Flower Metaphor[END_REF] http://www.google.de

Marathon runners [START_REF] Macgregor | Graphic Representation of Judgmental Information[END_REF] unknown Power plant statistics [START_REF] Petersen | Performance-based Evaluation of Graphic Displays for Nuclear Power Plant Control Rooms[END_REF] unknown

Audio information [START_REF] Frisson | Audiometro: Directing search for sound designers through content-based cues[END_REF] One laptop per child sound library Biological data [START_REF] Saraiya | Comparing benchmark task and insight evaluation methods on timeseries graph visualizations[END_REF] unknown

Economic variables [77] U.S. Department of commerce & labor

Tensor data [START_REF] Zhang | Glyphbased comparative visualization for diffusion tensor fields[END_REF] DTI dataset of dimensions. Of these, 44 tested less than 10 dimensions. An overview of different dimensionality settings is provided in Table 4. Three papers did not report about the number of dimensions encoded by the glyph designs [START_REF] Dunne | Motif simplification: Improving network visualization readability with fan, connector, and clique glyphs[END_REF], [START_REF] Goffin | Exploring the effect of word-scale visualizations on reading behavior[END_REF], [START_REF] Frisson | Audiometro: Directing search for sound designers through content-based cues[END_REF].

In the vast majority of studies (54/64, 84.38%) glyphs encoded general multi-dimensional data, both real and synthetic. Eight studies tested glyphs encoding time-series data (8/64, 12.5%) and in two experiments [START_REF] Dunne | Motif simplification: Improving network visualization readability with fan, connector, and clique glyphs[END_REF], [START_REF] Cayli | Glyphlink: An interactive visualization approach for semantic graphs[END_REF] glyphs were used to represent network topologies.

Only a small number of studies (24/64, 37.5%) used real data to investigate the performance of different glyph designs. The respective papers and real datasets can be found in Table 2. For the other experiments (41/64, 64.06%) the data was created synthetically.

Summary: Overall, most studies used synthetically created multi-dimensional data (41/64, 64.06%). The majority (44/64, 68.75%) of studies used glyphs encoding less than 10 dimensions.

Task Space

We used the Andrienko & Andrienko task taxonomy [START_REF] Andrienko | Exploratory Analysis of Spatial and Temporal data[END_REF] to distinguish between two higher-level tasks as discussed in Section 3. Synoptic tasks (i.e., similarity search, visual search, trend detection) were the most common type of task used in the studies (44/64, 68.75%). This is perhaps not surprising as glyphs are often meant to provide quick overviews over a large number of varying multidimensional data points-and the use of synoptic tasks may reflect the authors' desire to test glyphs in a realistic use context.

As shown in Table 5, we found the following classes of synoptic tasks: similarity search (23/44, 52.27%), followed by visual search [START_REF] Klippel | Colour-Enhanced Star Plot Glyphs: Can Salient Shape Characteristics Be Overcome?[END_REF], [START_REF] Klippel | Star Plots: How Shape Characteristics Influence Classification Tasks[END_REF].

In contrast to these synoptic tasks, 26/64 studies (40.63%) used elementary tasks, i.e., lookup (25/26, 96.15%) and 3D distance calculation (1/26, 3.85%). These studies focused on more perception-related questions such as the reading accuracy for visual variables used to encode a data value. In these studies, participants did not focus on reading the entire shape of the glyph, but on single glyph characteristics. For example in the user study conducted by MacGregor and Slovic [START_REF] Macgregor | Graphic Representation of Judgmental Information[END_REF] participants had to read the completion time of 48 marathon runners from bar chart glyphs, faces, and star glyphs. Faces performed best, followed by bar chart glyphs and star glyphs.

Summary: Most studies used a similarity search or a direct lookup task to measure the performance of glyph designs.

Study Outcomes

While we cannot discuss the study results individually for all 64 papers, we collected higher-level observations on study outcomes. Results on the study of factors such as number of dimensions and datapoints tested, is consistent across experiments. Nevertheless, when it comes to a general ranking, experimental results apply to a study's specific setting and should be generalized with caution. We discuss these results next.

Influence of Background Information and Layout

Understanding the influence of layout strategies or additional context information is crucial since data glyphs can be arranged in various different ways and settings. Four studies investigated the influence of positioning or background information on the performance of data glyphs [START_REF] Martin | Results of a User Study on 2D Hurricane Visualization[END_REF], [START_REF] Healey | Large datasets at a glance: Combining textures and colors in scientific visualization[END_REF], [START_REF] Goffin | Exploring the effect of word-scale visualizations on reading behavior[END_REF], [START_REF] Frisson | Audiometro: Directing search for sound designers through content-based cues[END_REF].

A common setting for data glyphs was the positioning in scatterplots, or projections from a high dimensional dataset to a two dimensional space. Frisson et al. used a visual search task to examine the benefits of a two dimensional projection compared to a grid layout used in small multiple settings [START_REF] Frisson | Audiometro: Directing search for sound designers through content-based cues[END_REF]. Performance was lower for the two dimensional projection, since after projection, some data glyphs ended up overlapping each other, which caused a loss of information making it difficult to detect the stimulus. In a follow-up study, the authors added a proximity grid [START_REF] Rodden | Does organisation by similarity assist image browsing?[END_REF] as an additional layout to the study setting. Results indicated that participants performed best in a visual search task when using the proximity grid.

Glyphs were also used in textual documents to communicate statistical data not only with words but visually. Sparklines are a famous example of such small visual representations [START_REF] Tufte | Beautiful Evidence[END_REF], which are usually positioned in the reading direction next to the statistics (e. g., on the right hand side). To backup this design decision, Goffin et al. conducted a user study to compare different layout possibilities of glyphs within sentences [START_REF] Goffin | Exploring the effect of word-scale visualizations on reading behavior[END_REF]. Surprisingly, there was no significant effect on accuracy or reading performance for the different layouts. However, participants preferred the glyph being positioned above the words.

The influence of reading data glyphs with different geographic backgrounds was investigated in only one study conducted by Martin [START_REF] Martin | Results of a User Study on 2D Hurricane Visualization[END_REF]. He measured the performance of participants working with weather vane glyphs while varying the underlying geographic map. Surprisingly, his results indicated the background had no influence on the performance of reading data-glyphs. However, the glyphs in his study were arranged in a grid on top of a map, and not according to their geographic position. Using different glyph designs or an irregular layout may, nevertheless, influence their performance.

Healey and Enns conducted an experiment to compare the interaction of different visual features in the surroundings of the glyph stimulus for a visual search task [START_REF] Healey | Large datasets at a glance: Combining textures and colors in scientific visualization[END_REF]. Results indicated that color variations due to the presence of other glyphs in the neighborhood of the stimulus glyph, caused a significant interference effect when participants had to judge heights of glyphs or density patterns. However, different densities in the surroundings of the stimulus or heights of neighboring glyphs had no effect on the detection of colored glyphs.

Summary: The influence of background and layout on reading data glyphs has so far received little research attention. The limited evidence from this work suggests that the background and neighborhood of a glyph did not affect glyph readability. Nevertheless more work is needed to determine the perceptual difficulties of reading glyphs depending on their background and layout. Seven studies varied the number of visible data points as a factor. The glyphs used in these experiments were either faces [START_REF] Morris | Experimental Analysis of the Effectiveness of Features in Chernoff Faces[END_REF], [START_REF] Nelson | Visual Search Processes and the Multivariate Point Symbol[END_REF], [START_REF] Siva | Visual Search Efficiency for Features in Chernoff Faces[END_REF], [START_REF] Sivagnanasundaram | Evaluation of the presence of a face search advantage in chernoff faces[END_REF], unique glyph designs (i.e., MILSTD2525 [START_REF] Siva | Evaluation of MILSTD 2525 Glyph Features in a Visual Search Paradigm[END_REF], arrow glyphs [START_REF] Wittenbrink | Glyphs for visualizing uncertainty in vector fields[END_REF]), or star glyphs [START_REF] Wu | Visualizing multivariate networks: A hybrid approach[END_REF].

For the studies involving face glyphs, participants had to perform visual search tasks and find a certain stimulus in a growing set of data points. The researchers tested whether pre-attentive identification was possible, in which case search time would not have been seriously impacted by increasing the number of glyphs. Yet, in all studies the performance dropped with an increasing number of data points independent from the mapping of data to face characteristics. Based on this outcome Siva and co-authors concluded that participants performed a serial search and were not able to pre-attentively identify the stimulus [START_REF] Siva | Visual Search Efficiency for Features in Chernoff Faces[END_REF], [START_REF] Sivagnanasundaram | Evaluation of the presence of a face search advantage in chernoff faces[END_REF]. Therefore, the perception of abstract data glyph faces compared to human faces was shown to be different. This is an interesting finding, which lessens the basic motivation for using abstract faces. However, researchers could also show that a redundant visual mapping of data to face characteristics improved the performance [START_REF] Morris | Experimental Analysis of the Effectiveness of Features in Chernoff Faces[END_REF].

Summary: Increasing the number of data points negatively affects search within a set of data glyphs, indicating that theyeven face glyphs-cannot be read pre-attentively.

Influence of Number of Dimensions

The results of studies varying the number of dimensions as a factor showed that different designs were impacted to different extents. In a study by Fuchs et al., for example, the performance of star glyphs dropped significantly in a lookup task when increasing the number of dimensions from 24 to 96, whereas the performance of line glyphs stayed stable [START_REF] Fuchs | Evaluation of alternative glyph designs for time series data in a small multiple setting[END_REF].

Wilkinson also varied the number of dimensions to investigate changes in performance for different glyph representations. His results indicated that increasing the number of dimensions had no significant effect on the ranking of tested glyph designs [START_REF] Wilkinson | An Experimental Evaluation of Multivariate Graphical Point Representations[END_REF], although there was a drop in performance overall.

However, it is interesting to note that even slight variations of a glyph design can be affected differently by the number of dimensions. Fuchs et al. tested the effect of increasing the number of dimensions on whisker glyphs (star glyphs without a contour line), traditional star glyphs and polygon variations. Although the performance dropped for all variations, whisker glyphs were affected the least [START_REF] Fuchs | The Influence of Contour on Similarity Perception of Star Glyphs[END_REF].

Summary: Increasing the number of dimensions negatively affects the performance of data glyphs [START_REF] Fuchs | Evaluation of alternative glyph designs for time series data in a small multiple setting[END_REF], [START_REF] Wilkinson | An Experimental Evaluation of Multivariate Graphical Point Representations[END_REF], [START_REF] Fuchs | The Influence of Contour on Similarity Perception of Star Glyphs[END_REF], [START_REF] Wu | Visualizing multivariate networks: A hybrid approach[END_REF].

Influence of Tasks and Visual Encoding

The outcome of individual user studies often involved a ranking of data glyphs based on their performance in the study. These rankings were not always consistent for the same designs tested, and they changed, for example, based on tasks and details of the visual encoding. Table 6-Table 9 summarize the outcomes of the different experiments. The ">" symbol indicates that the glyph on the left outperforms the design on the right (either in terms of completion time or accuracy).

Seven studies compared faces against circular position/length encodings [START_REF] Wilkinson | An Experimental Evaluation of Multivariate Graphical Point Representations[END_REF], [START_REF] Casey | Visual Display Representation of Multidimensional Systems: The Effect of System Structure: And Display Integrality[END_REF], [START_REF] Macgregor | Graphic Representation of Judgmental Information[END_REF], [START_REF] Mezzich | A Comparison of Graphical Representations of Multidimensional Psychiatric Diagnostic Data[END_REF], [START_REF] Jacob | The Face as a Data Display[END_REF], [START_REF] Nelson | An evaluation of multivariate, quantitative point symbols for maps[END_REF], [START_REF] Lee | An Empirical Evaluation of Chernoff Faces, Star Glyphs, and Spatial Visualizations for Binary Data[END_REF] (Table 6). In four, faces performed best [START_REF] Wilkinson | An Experimental Evaluation of Multivariate Graphical Point Representations[END_REF], [START_REF] Macgregor | Graphic Representation of Judgmental Information[END_REF], [START_REF] Mezzich | A Comparison of Graphical Representations of Multidimensional Psychiatric Diagnostic Data[END_REF], [START_REF] Jacob | The Face as a Data Display[END_REF], while circular position/length encodings performed best in the remaining three [START_REF] Casey | Visual Display Representation of Multidimensional Systems: The Effect of System Structure: And Display Integrality[END_REF], [START_REF] Nelson | An evaluation of multivariate, quantitative point symbols for maps[END_REF], [START_REF] Lee | An Empirical Evaluation of Chernoff Faces, Star Glyphs, and Spatial Visualizations for Binary Data[END_REF]. These seemingly contradictory results are reconciled when we consider the tasks participants had to perform and how the glyphs were designed. In five of these studies the participants performed a synoptic task [START_REF] Wilkinson | An Experimental Evaluation of Multivariate Graphical Point Representations[END_REF], [START_REF] Mezzich | A Comparison of Graphical Representations of Multidimensional Psychiatric Diagnostic Data[END_REF], [START_REF] Jacob | The Face as a Data Display[END_REF], [START_REF] Nelson | An evaluation of multivariate, quantitative point symbols for maps[END_REF], [START_REF] Lee | An Empirical Evaluation of Chernoff Faces, Star Glyphs, and Spatial Visualizations for Binary Data[END_REF], in the other three a lookup task [START_REF] Casey | Visual Display Representation of Multidimensional Systems: The Effect of System Structure: And Display Integrality[END_REF], [START_REF] Macgregor | Graphic Representation of Judgmental Information[END_REF], [START_REF] Nelson | An evaluation of multivariate, quantitative point symbols for maps[END_REF]. From the five synoptic task studies, in the three where faces performed best, the circular position/length encoding was a polygon (i.e., star glyph without whiskers, but only a contour) [START_REF] Wilkinson | An Experimental Evaluation of Multivariate Graphical Point Representations[END_REF], [START_REF] Mezzich | A Comparison of Graphical Representations of Multidimensional Psychiatric Diagnostic Data[END_REF], [START_REF] Jacob | The Face as a Data Display[END_REF], while in the remaining two where faces performed worst the circular encoding was a star glyph with [START_REF] Lee | An Empirical Evaluation of Chernoff Faces, Star Glyphs, and Spatial Visualizations for Binary Data[END_REF] and without contour line (i.e., whisker glyph) [START_REF] Nelson | An evaluation of multivariate, quantitative point symbols for maps[END_REF]. The remaining three studies with lookup tasks also compared faces against polygons (with polygons performing best [START_REF] Casey | Visual Display Representation of Multidimensional Systems: The Effect of System Structure: And Display Integrality[END_REF]), faces against star glyphs (with faces performing best [START_REF] Macgregor | Graphic Representation of Judgmental Information[END_REF]), and faces against whisker glyphs (with whisker glyphs performing best [START_REF] Nelson | An evaluation of multivariate, quantitative point symbols for maps[END_REF]). It seems that star glyphs compared to faces are more suitable for synoptic tasks. However, the whiskers glyph had the best performance independent from the underlying task. This finding has partially been confirmed for a similarity search [START_REF] Fuchs | The Influence of Contour on Similarity Perception of Star Glyphs[END_REF] but not for lookup tasks.

Another example where glyph rankings change based on study characteristics can be found when comparing faces against linear profiles (Table 7). In three studies faces performed best [START_REF] Wilkinson | An Experimental Evaluation of Multivariate Graphical Point Representations[END_REF], [START_REF] Macgregor | Graphic Representation of Judgmental Information[END_REF], [START_REF] Mezzich | A Comparison of Graphical Representations of Multidimensional Psychiatric Diagnostic Data[END_REF], in the fourth study, profiles [START_REF] Casey | Visual Display Representation of Multidimensional Systems: The Effect of System Structure: And Display Integrality[END_REF]. Again, the four studies used different tasks: lookup tasks [START_REF] Casey | Visual Display Representation of Multidimensional Systems: The Effect of System Structure: And Display Integrality[END_REF], [START_REF] Macgregor | Graphic Representation of Judgmental Information[END_REF] and a similarity search task [START_REF] Wilkinson | An Experimental Evaluation of Multivariate Graphical Point Representations[END_REF], [START_REF] Mezzich | A Comparison of Graphical Representations of Multidimensional Psychiatric Diagnostic Data[END_REF]. When comparing the two lookup tasks the ranking of the two glyph designs is still different although they use a similar number of dimensions (4 [START_REF] Macgregor | Graphic Representation of Judgmental Information[END_REF] and 5 [START_REF] Casey | Visual Display Representation of Multidimensional Systems: The Effect of System Structure: And Display Integrality[END_REF] dimensions), and just show one data point at a time. Yet, a major difference can be Elementary Task Synoptic Task [START_REF] Casey | Visual Display Representation of Multidimensional Systems: The Effect of System Structure: And Display Integrality[END_REF] [47], [START_REF] Mezzich | A Comparison of Graphical Representations of Multidimensional Psychiatric Diagnostic Data[END_REF], [START_REF] Jacob | The Face as a Data Display[END_REF] [65]

[65]

[50]

[66] Elementary Task Synoptic Task > [START_REF] Macgregor | Graphic Representation of Judgmental Information[END_REF] [47], [52] > [START_REF] Casey | Visual Display Representation of Multidimensional Systems: The Effect of System Structure: And Display Integrality[END_REF] found when reading the task description more carefully. Although both tasks are a lookup task, participants had to either read a one-dimensional value [START_REF] Macgregor | Graphic Representation of Judgmental Information[END_REF] or detect when one dimension changes significantly compared to the other dimensions for a single data point [START_REF] Casey | Visual Display Representation of Multidimensional Systems: The Effect of System Structure: And Display Integrality[END_REF].

When comparing linear and circular position/length encodings, we found glyph ranking differences in 8 studies [START_REF] Fuchs | Evaluation of alternative glyph designs for time series data in a small multiple setting[END_REF], [START_REF] Wilkinson | An Experimental Evaluation of Multivariate Graphical Point Representations[END_REF], [START_REF] Casey | Visual Display Representation of Multidimensional Systems: The Effect of System Structure: And Display Integrality[END_REF], [START_REF] Jones | The Display of Multivariate Information: An Experimental Study of an Information Integration Task[END_REF], [START_REF] Macgregor | Graphic Representation of Judgmental Information[END_REF], [START_REF] Carswell | Information Integration and the Object Display: An Interaction of Task Demands and Display Superiority[END_REF], [START_REF] Mezzich | A Comparison of Graphical Representations of Multidimensional Psychiatric Diagnostic Data[END_REF], [START_REF] Petersen | Performance-based Evaluation of Graphic Displays for Nuclear Power Plant Control Rooms[END_REF] (Table 8). In four, the linear design outperformed the radial [START_REF] Casey | Visual Display Representation of Multidimensional Systems: The Effect of System Structure: And Display Integrality[END_REF], [START_REF] Macgregor | Graphic Representation of Judgmental Information[END_REF], [START_REF] Mezzich | A Comparison of Graphical Representations of Multidimensional Psychiatric Diagnostic Data[END_REF], [START_REF] Petersen | Performance-based Evaluation of Graphic Displays for Nuclear Power Plant Control Rooms[END_REF], while in two, circular designs were better [START_REF] Jones | The Display of Multivariate Information: An Experimental Study of an Information Integration Task[END_REF], [START_REF] Carswell | Information Integration and the Object Display: An Interaction of Task Demands and Display Superiority[END_REF], and in the last one performance varied according to the underlying task [START_REF] Fuchs | Evaluation of alternative glyph designs for time series data in a small multiple setting[END_REF]. However, only 3 out of these 7 had a similar experimental setting with respect to design variations, presentation setting, number of dimensions and task [START_REF] Casey | Visual Display Representation of Multidimensional Systems: The Effect of System Structure: And Display Integrality[END_REF], [START_REF] Jones | The Display of Multivariate Information: An Experimental Study of an Information Integration Task[END_REF], [START_REF] Carswell | Information Integration and the Object Display: An Interaction of Task Demands and Display Superiority[END_REF]. These three all compare bar charts with polygons in a lookup task, using low dimensional data and presenting only one data point at a time. Surprisingly, the performance was still different : polygons ranked best in two of them [START_REF] Jones | The Display of Multivariate Information: An Experimental Study of an Information Integration Task[END_REF], [START_REF] Carswell | Information Integration and the Object Display: An Interaction of Task Demands and Display Superiority[END_REF] and bar charts performed best in the third [START_REF] Casey | Visual Display Representation of Multidimensional Systems: The Effect of System Structure: And Display Integrality[END_REF]. Again, we have to look at the studies more carefully to come to a conclusion. In the two studies where polygons performed best, the bars in the bar charts were shown without a common baseline. This was not true for the third study were bar charts outperformed the polygons. We assume that a common baseline increases the performance of the linear profiles, a finding which is proposed as a design guideline from a study by Fuchs et al. [START_REF] Fuchs | Evaluation of alternative glyph designs for time series data in a small multiple setting[END_REF]. However, a user study comparing linear profiles with and without a common baseline has, to the best of our knowledge, not yet been conducted.

Additionally, it is interesting to note that there were changes in performance depending on the kind of elementary task. For reading exact data values linear profiles outperformed star glyphs, however, when reading the position of an attribute dimension (e. g., a certain point in time for time-series data) star glyphs ranked first. [START_REF] Fuchs | Evaluation of alternative glyph designs for time series data in a small multiple setting[END_REF] Data glyph designs using color saturation to encode data values have not received much attention. We only found two papers, which report on results from quantitative experiments comparing these glyphs against alternative representations [START_REF] Fuchs | Evaluation of alternative glyph designs for time series data in a small multiple setting[END_REF], [START_REF] Nelson | An evaluation of multivariate, quantitative point symbols for maps[END_REF] (Table 9).

For overview visualizations focusing on the overall appearance of a glyph, color value encodings were not found to be effective. In three out of four user studies participants performed better using [24] a position encoding (i. e., linear and circular profiles) in synoptic tasks. In the fourth experiment the color encodings were visually enhanced to help participants solve certain tasks and outperformed linear profiles. Only faces performed worse. However, it is more difficult to draw conclusions for elementary tasks. When pursuing a direct lookup task (e. g., reading data values) radial color value encodings have outperformed star glyphs and faces [START_REF] Fuchs | Evaluation of alternative glyph designs for time series data in a small multiple setting[END_REF], [START_REF] Nelson | An evaluation of multivariate, quantitative point symbols for maps[END_REF]. Whisker glyphs on the other hand have been shown to be as accurate as color value encodings but more efficient [START_REF] Nelson | An evaluation of multivariate, quantitative point symbols for maps[END_REF]. However, linear profiles were most accurate and, therefore, the best choice for direct lookup tasks [START_REF] Fuchs | Evaluation of alternative glyph designs for time series data in a small multiple setting[END_REF]. Summary: Study results differed based on individual factors like number of dimensions, task, number of data points, or slight variations to the designs. Our summary tables can be considered as a performance overview pinpointing to relevant literature.

Influence of Metaphoric Glyph Design

One goal of information visualization is to present the underlying data in a way that can be easily understood by users. Thus, researchers have tried to improve intuitive understandability of visualizations, by using metaphors when mapping data to visual representations. One such example can be found for weather forecasts. In such a scenario weather status is communicated with small icons on top of a geographic map. These icons are metaphoric representations of the real environment to facilitate their understanding. Small cloud icons represent cloudy areas, rain drops encode rainy areas, and little suns illustrate sunshine in specific regions.

While data glyphs are different from icons [START_REF] Borgo | Glyph-based Visualization: Foundations, Design Guidelines, Techniques and Applications[END_REF], the general concept of representing the underlying data using metaphors can also be applied here. Since the visual representation of a glyph is data driven the idea is not to use a different glyph design for each individual data point (like in the weather forecast example), but to use certain glyph characteristics to display the data while being consistent with the metaphor. However, it is not clear whether such metaphor-based representations are better than more abstract ones. Siirtola has attempted to provide an answer to this problem by introducing metaphoric glyph designs and comparing them with more abstract ones [START_REF] Surtola | The Effect of Data-Relatedness in Interactive Glyphs[END_REF]. In his experiment he visualized car related data with abstract face representations, and with metaphoric car glyphs [START_REF] Surtola | The Effect of Data-Relatedness in Interactive Glyphs[END_REF]. Car glyphs were created by mapping data to parts of the glyph with related meaning. For example the attribute horsepower was mapped to the size of the engine of the car, which is metaphorically reflected in a bigger hood. In his user study participants had to answer car related questions when working with either faces or car glyphs. The metaphor helped the participants in understanding the data. As a result, they performed better when working with car glyphs compared to faces.

Li et al. [START_REF] Li | Metaphoric transfer effect in information visualization using glyphs[END_REF] provided another example where metaphors were used. In their quantitative experiment they compared RoseShape glyphs against abstract polygons to visualize multi-dimensional data about the education level in the US. The glyphs were positioned on top of a geographic map and participants had to either read data values or search for certain characteristics. Results suggest that participants were more accurate and more confident of their answers when working with the metaphoric designs.

In a study conducted by Flury and Riedwyl, data collected about monozygotic and dizygotic twins, such as their height or weight was mapped to two types of face glyphs [START_REF] Flury | Graphical Representation of Multivariate Data by Means of Asymmetrical Faces[END_REF]. Using abstract face representations (i. e., Chernoff faces) or more realistic faces (i. e., Flury Riedwyl faces) participants had to look at a glyph for each twin and rate whether or not the two glyphs showed data about monozygotic twins. The results indicated that participants were more accurate when working with the more realistic faces.

Jacob [START_REF] Jacob | Facial Representation of Multivariate Data[END_REF] gave another example where he tested the performance of a single metaphoric glyph design. He displayed data from patients having a certain psychological condition (e. g., depression, paranoia etc.) using faces. The abstract faces were created to show facial expressions resembling those of the human faces of the patients. Participants in his study had to judge which face corresponds to which behavior without being trained or knowing the patients. The results indicated that people were able to name the correct psychological illness without knowing the mapping criteria of data to face representations.

Metaphors may help to explain the results obtained in a study conducted by Fuchs et al. [START_REF] Fuchs | Evaluation of alternative glyph designs for time series data in a small multiple setting[END_REF]. The researchers ran a quantitative study using time-series data. Participants had to locate specific points in time using glyphs with either a linear dimension layout (e. g., sparklines) or a radial arrangement (e. g., star glyphs). Surprisingly, participants were more accurate when working with circular glyphs. This is interesting since the visual variable position (used in linear layouts) is considered more accurate compared to orientation (used in circular glyphs) [START_REF] Cleveland | Graphical Perception: Theory, Experimentation, and Application to the Development of Graphical Methods[END_REF]. However, participants argued that they were reminded of a clock when working with radial glyph designs, which facilitated locating certain points in time.

Summary: A small number of previous studies suggest that metaphors may help to better understand the underlying data.

Summary

While we found and reported on 64 papers, the vast design space of data glyphs and the possibility to test only a limited set of factors in a controlled user study makes it difficult to recommend a single best-of glyph design. Glyph performance depends on many different factors, such as the task used, the number of data points, or slight variations to the designs used across studies. Our analysis in Section 4.6.4 presents a summary of rankings from the articles we analyzed, and discusses how these factors can explain seemingly contradictory results.

We were able to draw general conclusions when it comes to number of dimensions and glyphs. Some study results indicate that increasing the number of data dimensions affects the performance of glyph designs negatively [START_REF] Fuchs | Evaluation of alternative glyph designs for time series data in a small multiple setting[END_REF], [START_REF] Wilkinson | An Experimental Evaluation of Multivariate Graphical Point Representations[END_REF], [START_REF] Fuchs | The Influence of Contour on Similarity Perception of Star Glyphs[END_REF], [START_REF] Wu | Visualizing multivariate networks: A hybrid approach[END_REF] with position encodings (linear and circular profiles) being more robust compared to color encodings in high-density situations [START_REF] Fuchs | Evaluation of alternative glyph designs for time series data in a small multiple setting[END_REF]. As with the number of dimensions, there is evidence that performance drops with increasing the amount of visible glyphs on the screen [START_REF] Wittenbrink | Glyphs for visualizing uncertainty in vector fields[END_REF], [START_REF] Siva | Evaluation of MILSTD 2525 Glyph Features in a Visual Search Paradigm[END_REF], [START_REF] Wu | Visualizing multivariate networks: A hybrid approach[END_REF], [START_REF] Morris | Experimental Analysis of the Effectiveness of Features in Chernoff Faces[END_REF], [START_REF] Nelson | Visual Search Processes and the Multivariate Point Symbol[END_REF], [START_REF] Siva | Visual Search Efficiency for Features in Chernoff Faces[END_REF], [START_REF] Sivagnanasundaram | Evaluation of the presence of a face search advantage in chernoff faces[END_REF]. This seems like a logical conclusion due to the required additional effort in visual search involving a higher number of entities. In addition, a small number of past studies indicate that metaphoric glyph designs increase performance.

Finally, it has to be noted that our analysis was made difficult by a lack of standard for reporting study details on glyphs. For example specific information (e. g., stimuli size, viewing distance, number of visible data points, etc.), that could shed light on differences across experiments, were often missing.

IMPLICATIONS FOR DATA GLYPH DESIGN

In this section we abstract and summarize the study outcomes in the form of design considerations. We indicate the number of papers supporting each design consideration in brackets to indicate the extent of generalizability, and order them based on this support.

Do not expect glyphs to be perceived pre-attentively [START_REF] Chan | The Generalized Sensitivity Scatterplot[END_REF]. All seven studies considering the scalability as a factor come to the conclusion that the performance drops significantly when increasing the number of data points. However, these studies only included faces, arrows, star glyphs and MILSTD2525 glyphs.

Faces have possible value [START_REF] Peng | Clutter Reduction in Multi-Dimensional Data Visualization Using Dimension Reordering[END_REF]. Many experiments with different study designs have been conducted comparing faces against other data glyph design. Based on the underlying study setting the ranking of the tested designs was different. However, in at least five studies faces outperformed the alternatives [START_REF] Wilkinson | An Experimental Evaluation of Multivariate Graphical Point Representations[END_REF], [START_REF] Macgregor | Graphic Representation of Judgmental Information[END_REF], [START_REF] Mezzich | A Comparison of Graphical Representations of Multidimensional Psychiatric Diagnostic Data[END_REF], [START_REF] Jacob | The Face as a Data Display[END_REF], [START_REF] Brown | Methods for the Graphic Representation of Systems Simulated Data[END_REF].

Consider metaphoric designs [START_REF] Peng | Clutter Reduction in Multi-Dimensional Data Visualization Using Dimension Reordering[END_REF]. Five studies investigated the usefulness of metaphoric data glyph designs. The results indicate that metaphors helped participants in analyzing data. However, the type of metaphors and type of data are closely related, making the past studies somewhat limited. This a fruitful area for further research.

Linear data glyph designs profit from a similar baseline (3). In three studies comparing linear against circular profiles, linear designs were most effective when the dimensions where aligned to a common baseline. However, this finding has not been directly investigated by an experimental comparison of linear profiles with and without a common baseline.

In linear designs color encodings should be avoided, whereas they seem to be beneficial for circular layouts (3).

Since only four studies have been conducted using color saturation to encode data values, this guideline has to be considered with caution [START_REF] Fuchs | Evaluation of alternative glyph designs for time series data in a small multiple setting[END_REF], [START_REF] Nelson | An evaluation of multivariate, quantitative point symbols for maps[END_REF]. It seems that in circular layouts with position/length encodings (i.e. star glyps) the mental rotation required for comparisons affects participants more strongly than color comparisons that can be conducted without a common axis. For linear designs the ranking of visual variables from Cleveland and McGill seems to hold [START_REF] Cleveland | Graphical Perception: Theory, Experimentation, and Application to the Development of Graphical Methods[END_REF].

Pick line glyphs for lookup tasks with a large number of dimensions (2): Studies suggest that data glyph designs are influenced negatively by an increasing number of dimensions. However, the line glyph seems to be affected the least. For similarity search tasks one study found the whiskers glyph to be most effective in a high-dimensional setting.

DISCUSSION AND OPEN RESEARCH AREAS

In this section we identify and discuss directions for future research based on our analysis. The proposed research directions are ordered roughly according to their scope.

Types of User Studies:

Even though we focused on user studies with quantitative components for this paper, we found only a few qualitative studies that considered how glyphs are used in practice within real applications. One such exception is the experiment conducted by Sreng et al. [START_REF] Sreng | Using visual cues of contact to improve interactive manipulation of virtual objects in industrial assembly/maintenance simulations[END_REF] where participants used a 3D automotive assembly tool and answered questions about the perceived usefulness of the embedded glyphs. Although this study provided qualitative observations in the form of questionnaires, we can envision more elaborate field experiments and observational studies on real use of glyphs. Observers could thus gather information on how people use glyphbased visualizations in real contexts, for which tasks, and with what kind of results. Such studies could inform our understanding of how glyph-based applications are adopted and used in practice and could, thus, provide new insights on which to base design choices.

Summary: Adding qualitative evaluations observing analysts working with different glyph designs, datasets, and tasks, would help to better understand the glyphs design space. In particular, information about subjective preferences and the applicability of specific glyph designs in practice would be useful. It would be interesting to capture which design analysts choose to solve which analysis task.

Data to be Tested:

There are several pros and cons for choosing real vs. synthetic datasets for a study. On the one hand, real data has the advantage that it can demonstrate which visual representation performs best in realistic situations, providing valuable results for analysts of this data. However, real data often contains unique characteristics (e. g., size, structure, number of dimensions), that make the results noisy and hard to generalize.

On the other hand, one may argue that synthetic data does not always represent a real world scenario or problem well (ecological validity), making results again hard to generalize. However, artificial data can be easily controlled and focused on answering specific questions. Additionally, possible confounding factors due to the underlying data are excluded (e. g., visual search time according to the number of data points).

Given the above pros and cons, it seems an interesting open research question to see how glyphs behave when they undergo study using both synthetic and real data, similar to the approach taken by Caban [START_REF] Caban | An evaluation of visualization techniques to illustrate statistical deformation models[END_REF].

Summary: Running quantitative experiments, using both datasets from synthetic to real world and vice versa will enhance our knowledge on the behavior of data glyphs in different situations.

Study Tasks and Measures:

In the majority of studies participants had to perform synoptic tasks (Table 5). This is not surprising given that glyphs are often used to provide quick overviews over a large number of multi-dimensional data points. Nevertheless, there are glyph designs (e. g., some 3D glyphs) that have not or rarely been looked at for synoptic tasks, an interesting topic for further study.

Although results from specific tasks, such as these synoptic ones, are valuable, a common visualization task is free exploration, insight generation and hypothesis forming. Inspired by recent work on insight based evaluation [START_REF] Saraiya | An Insight-Based Methodology for Johannes Fuchs is a research scientist and lecturer at the University of Konstanz[END_REF], it would be worthwhile to investigate the performance of different glyph designs in such contexts.

Summary: Adding exploration tasks or extracting insights from an unknown dataset are realistic real-world analysis tasks. They should, therefore, be added to the repertoire of user study tasks in glyph evaluation to further reason about the practical applicability of data glyphs.

Glyph Presentation Setting:

A large number of studies presented glyphs as small multiples using a grid layout. There were no studies on glyphs nested inside treemaps, or other types of representations apart from maps, scatterplots, node-link diagrams, and two 3D representations in the medical domain. This is interesting, as it is not clear that grid layouts present the most commonly assumed usage context for glyphs. For example, in the area of scientific visualization, glyphs are often used on 3D volumetric surfaces or to represent 2D flow fields in order to indicate data at specific sampling points. These glyphs are approximately uniformly spaced apart, but this relative spacing changes depending on the view's magnification factor, making them appear more or less densely packed together. There is very little to no guidance from controlled user studies on how this apparent density affects their performance.

Moreover, we know little about the influence of the background information on the performance of glyphs. Only one study investigated performance changes for glyph designs when placed on top of different geographic maps [START_REF] Martin | Results of a User Study on 2D Hurricane Visualization[END_REF], and one other their performance close to neighboring glyphs [START_REF] Healey | Large datasets at a glance: Combining textures and colors in scientific visualization[END_REF]. Many questions remain unanswered, for example, we do not know if glyphs are perceived differently when arranged in uniform grids compared to other arrangements, such as treemaps, that vary their relative distance.

It is also unclear what effect the glyphs have on the understanding of the underlying visualization itself: for example, it would be interesting to investigate if rectangular treemaps are more effective compared to circular treemaps when adding glyph designs; or if people are distracted by the additional context information in the form of glyphs in 3D environments. There is certainly much space for further research.

Summary: Since data glyphs cannot only be positioned in small multiple grids, evaluating different arrangements of more complex layouts (i. e., treemaps, etc.), would help to better understand the influence of specific data glyph designs on the context and vice versa.

Glyph Types and Data Encodings:

Understand redundant encodings: Using Ward's glyph design categorization [START_REF] Ward | Multivariate data glyphs: Principles and practice[END_REF], we found only two studies that used glyphs with a one-to-many mapping (i. e., a redundant encoding). Ware [START_REF] Ware | Information Visualization: Perception for Design[END_REF], however, discusses interesting perceptual study approaches to learn how dimension encodings can be separable or integral. A better understanding of how redundant encodings work together, and could enforce data reading, would prove beneficial to glyph design. Study missing mappings: In Fig. 3 we refer to 50 of the 64 studies examined, having left out the two one-to-many mappings [START_REF] Dick | Distance Visualization for Interactive 3D Implant Planning[END_REF], [START_REF] Klippel | Colour-Enhanced Star Plot Glyphs: Can Salient Shape Characteristics Be Overcome?[END_REF] and the twelve that were not compared to other designs [START_REF] Aigner | Planninglines: novel glyphs for representing temporal uncertainties and their evaluation[END_REF], [START_REF] Martin | Results of a User Study on 2D Hurricane Visualization[END_REF], [START_REF] Pilar | Representing flow patterns by using streamlines with glyphs[END_REF], [START_REF] Yost | Single complex glyphs versus multiple simple glyphs[END_REF], [START_REF] Healey | Large datasets at a glance: Combining textures and colors in scientific visualization[END_REF], [START_REF] Li | Metaphoric transfer effect in information visualization using glyphs[END_REF], [START_REF] Chan | Visualizing the semantic structure in classical music works[END_REF], [START_REF] Wittenbrink | Glyphs for visualizing uncertainty in vector fields[END_REF], [START_REF] Dunne | Motif simplification: Improving network visualization readability with fan, connector, and clique glyphs[END_REF], [START_REF] Cayli | Glyphlink: An interactive visualization approach for semantic graphs[END_REF], [START_REF] Chau | Visualizing Web Search Results Using Glyphs: Design and Evaluation of a Flower Metaphor[END_REF], [START_REF] Siva | Evaluation of MILSTD 2525 Glyph Features in a Visual Search Paradigm[END_REF]. Looking at the table there is still clearly an imbalance in what kind of data encodings have been comparatively tested. Many cells remain empty, and there are several sparsely populated ones. One of the first things to notice is that there is no single study on circular orientation encodings, although they are used in visualization applications: representatives of this category are the compound glyph used in network graphs [START_REF] Pearlman | Visualizing Network Security Events using Compound Glyphs from a Service-oriented Perspective[END_REF], pie chart glyphs for analyzing multi-dimensional data (e.g., global material composition [START_REF] Amirkhanov | Inspectr: Multi-modal exploration, visualization, and analysis of spectral data[END_REF], or biological binding properties ), or as provided in visualization toolkits (e. g., JIT 2 ). Perhaps this type of encoding is a-priori deemed inferior based on Cleveland and McGill's [START_REF] Cleveland | Graphical Perception: Theory, Experimentation, and Application to the Development of Graphical Methods[END_REF] work that ranks orientation low for quantitative data representation. Given past use of these encodings however, it is certainly worthwhile to confirm that Cleveland and McGill's ranking does hold for circularorientation encodings in glyphs, in particular in the context of real multi-dimensional data.

Similarly, several other cells of Fig. 3 are empty or populated by studies from a single paper. As discussed in Section 4.6, the ranking of glyph designs or their variations often depends on tasks and encodings, and as such more studies are needed to be able to provide reliable guidance for general glyph use and design. Especially glyph designs, which have not received much research attention but are used in practice (i.e., pie chart glyphs, or variations of linear profiles) should be prioritized in future studies. Replicate studies on face glyphs: Many studies have been conducted investigating the performance of faces. Most of these studies were conducted in the 70s, and 80s when faces were newly introduced. In recent years face glyphs have been considered inferior but there are no recent studies or replications of earlier studies to confirm this. Given that some past studies showed good performance, it may be worthwhile to try and reproduce some earlier studies to confirm that they are indeed not as good as their current reputation in the community suggests. Test larger number of dimensions: In addition to the data encoding, the number of glyph dimensions may highly influence performance. As we saw in Table 4 the vast majority of studies only examined glyphs under a fixed number of dimensions, often less than 10 data dimensions. Only four varied the number of dimensions systematically in their studies. To reliably understand how glyph performance scales, we need to further explore how glyph designs fare under different dimensions.

Summary: Quantitative user studies should be conducted to compare data glyph designs which have not yet received much research attention (i. e., pie chart glyphs). The number of 2. JavaScript InfoVis Toolkit http://philogb.github.io/jit/ dimensions should be varied during the experiment and considered as a factor for analysis, to better understand glyph scalability.

Summary:

This section motivated promising open research directions for future experiments on data glyphs. In this summary, we revisit the most important gaps we identified and most promising research directions. Firstly, we need to give priority to experiments investigating glyph designs, which have not received much research attention, yet. For example, there is only little knowledge about the performance of radial layouts, such as pie chart glyphs. Having more evaluations about data glyph designs will help to better generalize the outcomes and argue about the performance of visual variables.

Additionally, different presentation settings need to be tested in more detail, since a big advantage of data glyphs is their flexible arrangement on the screen. In most experiments the data glyphs were positioned in a regular grid layout, however, data glyphs can also be arranged in more complex layouts like treemaps. Currently, there is only little guidance whether the performance of data glyphs will change according to context information or layout.

A wider variety of experimental factors should be considered such as: multiple datasets (i. e., synthetic data and real world data), different analysis tasks (e. g., exploration or insight generation), and different study types (i. e., qualitative and quantitative) to get a deeper understanding of the utility and performance of data glyphs.

CONCLUSION

In this paper we contribute a systematic review of research papers focusing on the evaluation of data glyphs in quantitative user studies. We organize this work using several criteria, such as glyph types, study presentation settings, datasets and tasks used. An overall summary of the most interesting findings can be found in Table 10. Our goal is to: first, help researchers and practitioners identify relevant previous studies that give insights into glyph design tradeoffs, and get inspired by previous study setups; second, provide a meta analysis of the study outcomes; and third, pinpoint open research directions for the study of data glyphs.

In summary, we found that faces and their variations were the most studied glyphs, followed by circular position encoding glyphs (e. g., star glyphs), which were often also compared to faces. Our analysis showed that at first glance performance rankings differ across studies. Yet, we discuss how some of these seemingly contradictory results can be explained by differences in the study criteria, such as the tasks, density and variations of glyphs tested. Our categorization provides readers with references to past studies that can not only serve as specific design considerations but also be an inspiration to design one's own user study.

Our work also aims to highlight gaps in the literature on data glyph evaluation. Few papers have evaluated variations of glyphs using color encoding, even though such glyphs are used in practice [START_REF] Yang | Value and Relation Display for Interactive Exploration of High Dimensional Datasets[END_REF]. Moreover, only two works [START_REF] Albers | Task-driven evaluation of aggregation in time series visualization[END_REF], [START_REF] Goffin | Exploring the effect of word-scale visualizations on reading behavior[END_REF] have compared design variations of glyphs using linear position or length encodings, that are well established in practice (i. e., sparklines [START_REF] Tufte | Beautiful Evidence[END_REF] or profiles [START_REF] Du Toit | Graphical Exploratory Data Analysis[END_REF])-although some more have been compared to faces. We were also unable to find any study on circular orientation/angle encodings as already used in applications (e.g. [START_REF] Pearlman | Visualizing Network Security Events using Compound Glyphs from a Service-oriented Perspective[END_REF]). At the present time we caution against making overly general recommendation for using one type of glyph over another given in particular the many criteria we needed to use to distinguish and categorize past Faces and circular profiles have been investigated in detail, in contrast to color value and orientation encodings on glyphs that only few studies investigated. Surprisingly, we found only two studies comparing different variations of linear profiles.

4.3

Only a small number of user studies varied the amount of data glyphs as a study factor. Most studies were conducted with a fixed number of glyphs arranged in a grid layout. Surprisingly, only four papers investigated the influence of different background information and layout on reading data glyphs [START_REF] Martin | Results of a User Study on 2D Hurricane Visualization[END_REF], [START_REF] Healey | Large datasets at a glance: Combining textures and colors in scientific visualization[END_REF], [START_REF] Goffin | Exploring the effect of word-scale visualizations on reading behavior[END_REF], [START_REF] Frisson | Audiometro: Directing search for sound designers through content-based cues[END_REF]. 4.4 Overall, most studies used synthetically created multi-dimensional data (41/64, 64.06%). The majority (44/64, 68.75%) of studies used glyphs encoding less than 10 dimensions.

4.5

Most studies used a similarity search or a direct lookup task to measure the performance of glyph designs.

Study Outcomes

4.6.1

The influence of background and layout on reading data glyphs has so far received little research attention.

The limited evidence from this work suggests that the background and neighborhood of a glyph did not affect glyph readability. Nevertheless more work is needed to determine the perceptual difficulties of reading glyphs depending on their background and layout. 4.6.2

Increasing the number of data points negatively affects search within a set of data glyphs, indicating that theyeven face glyphs-cannot be read pre-attentively. 4.6.3

Increasing the number of dimensions negatively affects the performance of data glyphs [START_REF] Fuchs | Evaluation of alternative glyph designs for time series data in a small multiple setting[END_REF], [START_REF] Wilkinson | An Experimental Evaluation of Multivariate Graphical Point Representations[END_REF], [START_REF] Fuchs | The Influence of Contour on Similarity Perception of Star Glyphs[END_REF], [START_REF] Wu | Visualizing multivariate networks: A hybrid approach[END_REF]. 4.6.4

Study results differed based on individual factors like number of dimensions, task, number of data points, or slight variations to the designs. Our summary tables can be considered as a performance overview pinpointing to relevant literature. 4.6.5

A small number of previous studies suggest that metaphors may help to better understand the underlying data.

Open Research Gaps

Adding qualitative evaluations observing analysts working with different glyph designs, datasets, and tasks, would help to better understand the glyphs design space. In particular, information about subjective preferences and the applicability of specific glyph designs in practice would be useful. It would be interesting to capture which design analysts choose to solve which analysis task. 6

Running quantitative experiments, using both datasets from synthetic to real world and vice versa will enhance our knowledge on the behavior of data glyphs in different situations. [START_REF] Yang | Interactive Hierarchical Dimension Ordering, Spacing and Filtering for Exploration of High Dimensional Datasets[END_REF] Adding exploration tasks or extracting insights from an unknown dataset are realistic real-world analysis tasks. They should, therefore, be added to the repertoire of user study tasks in glyph evaluation to further reason about the practical applicability of data glyphs. 6

Since data glyphs cannot only be positioned in small multiple grids, evaluating different arrangements of more complex layouts (i. e., treemaps, etc.), would help to better understand the influence of specific data glyph designs on the context and vice versa. 6

Quantitative user studies should be conducted to compare data glyph designs which have not yet received much research attention (i. e., pie chart glyphs). The number of dimensions should be varied during the experiment and considered as a factor for analysis, to better understand glyph scalability.

studies (e. g., datasets, tasks, encodings). There are still several years of research possible to understand how humans perceive and use glyphs. We outlined many opportunities for design and evaluation and hope this work encourages researchers to pursue them.

Fig. 1 .

 1 Fig. 1. Data Glyphs: A selection of the different data glyph designs used in the quantitative experiments we analyzed.

Faces

Fig. 2 .

 2 Fig. 2. Statistics: Ratio of papers evaluating different visual encodings (distinguished by color). Low saturation indicates experiments evaluating design variations of this encoding, and high saturation other experiments (e.g. comparisons to other encodings).

Fig. 3 .

 3 Fig. 3. Glyph Design Table:Columns represent the different categories of glyph encodings, replicated in rows with glyph examples for each category. Additionally, color is used to visually separate the different categories. References refer to articles in our study bibliography that compare glyph variations from the respective row and column. Studies placed in the diagonal evaluate either variations of the same glyph type, or comparisons of the glyph with data tables (starred *). Note that papers can fall in multiple cells. Since PlanningLines[START_REF] Aigner | Planninglines: novel glyphs for representing temporal uncertainties and their evaluation[END_REF], weather vanes[START_REF] Martin | Results of a User Study on 2D Hurricane Visualization[END_REF],[START_REF] Pilar | Representing flow patterns by using streamlines with glyphs[END_REF], shapes[START_REF] Yost | Single complex glyphs versus multiple simple glyphs[END_REF],[START_REF] Healey | Large datasets at a glance: Combining textures and colors in scientific visualization[END_REF], roses[START_REF] Li | Metaphoric transfer effect in information visualization using glyphs[END_REF], themes[START_REF] Chan | Visualizing the semantic structure in classical music works[END_REF], arrows[START_REF] Wittenbrink | Glyphs for visualizing uncertainty in vector fields[END_REF], Motifs[START_REF] Dunne | Motif simplification: Improving network visualization readability with fan, connector, and clique glyphs[END_REF],[START_REF] Cayli | Glyphlink: An interactive visualization approach for semantic graphs[END_REF], flowers[START_REF] Chau | Visualizing Web Search Results Using Glyphs: Design and Evaluation of a Flower Metaphor[END_REF], and MILSTD2525 glyphs[START_REF] Siva | Evaluation of MILSTD 2525 Glyph Features in a Visual Search Paradigm[END_REF] use a unique encoding and are not compared to other glyphs, we positioned them slightly apart in the "One-to-One Mapping" category.

Fig. 4 .

 4 Fig. 4. Statistics: Ratio of papers evaluating different glyph layouts (distinguished by color). Low saturation indicates experiments evaluating a varying number of data glyphs, and high saturation a fixed number of data glyphs.

TABLE 1

 1 Overview of defining glyph characteristics mentioned in the literature.

TABLE 2 Datasets:

 2 This table illustrates detailed information about the real datasets used in the experiments.

TABLE 3 Presentation Setting :

 3Setting This table distinguishes between the number of data points shown to the participants during the studies and the used layout. Color is used to better distinguish between the different categories.

		Layout	References
	Single	Text	[48][49][50][54][64][91] [55]
		Grid	[17][21][24][33][39][45][47][51][52][53]
	Multiple Glyphs	Geo map Scatterplot Other	[57][58][59][61][66][67][71][72][73][74] [75][76][77][78][80][84][85][86][87][89] [36][37][38][40][60][65][68][69] [34][88][90] [32][35][56]
		Node-link	[41][70]
	Varying	Grid Node-link Geo map	[46][62][79][82][83][92] [43][44][63] [42][81]

TABLE 4 Number of Dimensions:

 4 This table illustrates the different data dimension densities used in the studies. Color is used to better distinguish between the different categories.

	Number of Dimensions

(14/44, 31

.82%), and trend detection tasks (7/44, 15.91%). An example of a similarity search task can be found in two studies by

Klippel et

al.: Using a visualization tool showing 81 glyphs each representing one car, participants had to group these glyphs into different categories based on their attributes

TABLE 5

 5 Data and Tasks: Most studies were conducted using a lookup or similarity search task with multi-dimensional data.

	Data Type		Task Description
	Elementary Task		Synoptic Task
	Lookup	3D Navigation	Trend Detection	Similarity Search	Visual Search
		(distance
		calculation)

TABLE 6

 6 Studies and their result rankings: faces vs. circular profiles.

TABLE 7

 7 Studies and their result rankings: faces vs. linear profiles. Conflicting results are marked with orange color.

TABLE 8

 8 Studies and their result rankings: linear vs. circular profiles. Conflicting results are marked with orange color.

	Elementary Task	Synoptic Task
	>	[48]	[52]
	>	[49], [51]
	>	[24], [50], [53]	[24]
	>	[24]

TABLE 9

 9 Studies and their result rankings: color saturation vs. profiles. Conflicting results are marked with orange color.

	Elementary Task	Synoptic Task
	[24]		[24]
	[24], [54]	>	[24], [54]
		>	[54]
	[65]		[65]
	[65]		[65]

TABLE 10 Overall Summary :

 10Summary This table is a collection of all design considerations, open research gaps, and findings proposed throughout the paper. We found similar study goals across many experiments, yet varied were factors like number of data points and dimensions, task, or glyph design. These variations make individual study outcomes hard to compare. 4.2

	Section	Summary
	4.1
	Study Settings

ACKNOWLEDGMENTS

This work has been supported by the Consensus project and has been partly funded by the European Commission's 7th Framework Programme through theme ICT-2013.5.4 ICT for Governance and Policy Modelling under contract no.611688.