Augustin Lefèvre
email: augustin.lefevre@inria.fr

Francis Bach
email: francis.bach@ens.fr

Cédric Févotte
email: fevotte@telecom-paristech.fr

SEMI-SUPERVISED NMF WITH TIME-FREQUENCY ANNOTATIONS FOR SINGLE-CHANNEL SOURCE SEPARATION

We formulate a novel extension of nonnegative matrix factorization (NMF) to take into account partial information on source-specific activity in the spectrogram. This information comes in the form of masking coefficients, such as those found in an ideal binary mask. We show that state-ofthe-art results in source separation may be achieved with only a limited amount of correct annotation, and furthermore our algorithm is robust to incorrect annotations. Since in practice ideal annotations are not observed, we propose several supervision scenarios to estimate the ideal masking coefficients. First, manual annotations by a trained user on a dedicated graphical user interface are shown to provide satisfactory performance although they are prone to errors. Second, we investigate simple learning strategies to predict the Wiener coefficients based on local information around a given time-frequency bin of the spectrogram. Results on single-channel source separation show that time-frequency annotations allow to disambiguate the source separation problem, and learned annotations open the way for a completely unsupervised learning procedure for source separation with no human intervention.

INTRODUCTION

During the past decade, nonnegative matrix factorization (NMF) has become the core algorithm in single-channel source separation. A rich literature has been developed to adapt NMF to difficult scenarios in which sources are highly synchronized, and little or no development data is available.

In the past years, intensive research on Bayesian modelling and parameterized methods have been conducted to improve the identifiability of basis elements by restricting the complexity of the estimated model. More recently, another category of contributions consider incorporating information that is directly relevant to the data at hand, and specified by the user. In [START_REF] Wang | Musical Audio Stream Separation[END_REF], time activation of the sources is used to specify direct constraints on the activation coefficients of the decomposition. Pitch estimates [START_REF] Durrieu | Musical audio source separation based on user-selected f0 track[END_REF] were used for lead voice extraction. In [START_REF] Hennequin | Score informed audio source separation using a parametric model of non-negative spectrogram[END_REF], detailed score information is provided so that each individual note can be separated. While these contributions may use different NMF models, a common trait is that user information is used to specify the support of decomposition coefficients at the coding stage. A quite different line of work is proposed in [START_REF] Smaragdis | Separation by "humming": User-guided sound extraction from monophonic mixtures[END_REF][START_REF] Fitzgerald | User Assisted Source Separation using Non-negative Matrix Factorisation[END_REF], where isolated signals are used as proxy for the source signals, so that information on both the basis functions and the activation coefficients can be used to constrain the factorization.

In this paper, we propose to annotate directly the timefrequency representation that is used to perform source separation. We assume that we are given recordings where a large fraction of time-frequency bins of the spectrogram may be assigned unambiguously to one dominant source. This hypothesis holds as long as there are not too many sources, and post-processing of the recording does not involve heavily non-linear effects. As illustrated in Figure 1, some patches in the spectrogram are cues for sourcespecific activity, which may be exploited as information on the optimal binary mask. Figure 1: Cues from computational audio source analysis may be used as information on the optimal masking coefficients

In this article we make three contributions : we propose in Section 2 a novel modification of NMF (semi-supervised NMF) to take into account time-frequency annotations of the spectrogram, that is robust to errors in the annotations. In Section 2.3, we present a graphical user interface to retrieve such time-frequency annotations. In Section 3, we propose supervised learning algorithms to automatize annotations, and explain how to combine them with semisupervised NMF. Finally, we illustrate our contributions on publicly available source separation databases in Section 4.

SEMI-SUPERVISED NMF

Model and interpretation

In this section we propose a novel modification of NMF to incorporate annotations in the spectrogram. Let us first briefly summarize our NMF model and introduce mathematical notations, before proceeding to the main part of the contribution.

Given the short time Fourier transform of a signal X ∈ C F ×N (in the following f indexes frequency and n time), we assume that X = g S (g) , where S (g) ∈ C F ×N is the spectrogram of each source signal for g ∈ {1, . . . , G}. Define the power spectrograms of the sources V (g)

f n = |S (g) | 2 .
They are assumed to follow a linear model :

V (g) f n = Kg k=1 W (g) f k H (g) kn , where W (g) ∈ R F ×Kg + , H (g) ∈ R Kg×N . Define K = g K g , W = (W (1) , . . . , W (G) ) ∈ R F ×K + and H ⊤ = ((H (1) ) ⊤ , . . . , (H (G) ) ⊤ ) ∈ R K×N + .
Then, depending on the assumed distribution of S (g) , estimation of W and H amounts to minimizing d(V, W H) where d is a measure of fit between data and the underlying model. In this article we will use the Itakura-Saito divergence, but actually any β-divergence may be used.

Given estimates V (g) f n of the power spectrogram of each source, time domain estimates of the sources are then computed by Wiener filtering, where the Wiener coefficients of the source in the time-frequency domain are given by :

M (g) f n = V (g) f n
Vfn .

The key idea in our contribution is the following : suppose we have at hand a set L of annotated time-frequency bins and a set of time-frequency masks M (g)

f n such that : M (g) f n ∈ [0, 1], and g M (g) f n = 1 if (f, n) ∈ L, g M (g) f n = 0 otherwise.
For annotated time-frequency bins, we define target values for each source spectrogram :

Ṽ (g) f n = M (g)
f n V f n . The remaining, un-annotated entries of V are then computed so as to fit the observed spectrogram. This idea translates into the following optimization problem :

min (f,n) d IS (V f n , Vfn ) + λ (f,n)∈L g=1,...,G µ f n d IS ( Ṽ (g) f n , V (g) f n ) , (1
) where d IS (x, y) = x y -log x y -1 is the Itakura-Saito divergence 1 , and optimization is subject to the constraints that W ≥ 0 (point-wise nonnegativity), H ≥ 0, and f W f k = 1 to avoid scaling ambiguity. We interpret the second term in Eq. ( 1) as a relaxed version of the constraints that V (g)

f n
be equal to their target value M (g)

f n V f n , for all annotated bins (f, n) ∈ L.
We may tune the relative importance of annotation by varying parameter λ, from λ = 0 (standard NMF), to λ → +∞ (in which case (W H) f n = V (g) f n is enforced exactly if there are any feasible solutions). Thus, robustness to uncertainty in the annotations is introduced by replacing hard constraints by penalty terms in the NMF optimization problem. Note that since annotations dictate the assignment of components to sources, there is no need to group components by hand. We will discuss the role of µ f n in the next section : in the case of user annotations, µ f n = 1. Let us discuss two cases : (a) M (g) fn ∈ {0, 1}: this is the case of user annotations, where time-frequency bins are labelled by hand. In this case, there can be only one active source at each timefrequency bin, since g M (g) f n = 1. This is a strong assumption, which is verified for a large fraction of the mixtures that are found in blind source separation. Discussing the algorithm is beyond the scope of this paper : we used a multiplicative updates algorithm with appropriate modifications to deal with the additional terms in Eq. ( 1) [START_REF] Févotte | Nonnegative matrix factorization with the Itakura-Saito divergence: With application to music analysis[END_REF].

Relation with previous work

As in [START_REF] Wang | Musical Audio Stream Separation[END_REF][START_REF] Hennequin | Score informed audio source separation using a parametric model of non-negative spectrogram[END_REF][START_REF] Durrieu | Musical audio source separation based on user-selected f0 track[END_REF], annotations are used to constraint some sources to be inactive. In fact, time annotations are a special case of our model, where annotations are such that M (g)

f n = M (g) f ′ n for all (f, f ′ ) (i.e.,

zeroes come in columns).

Our model deals with that case when there are two sources. The only difference between our model and [START_REF] Wang | Musical Audio Stream Separation[END_REF] is that instead of enforcing H kn = 0 as a hard constraint, we introduce a soft penalty to enforce W f k H kn = 0, with the added benefit that incorrect annotations are dealt with in a robust fashion. The case of more than two sources is dealt with a simple extension of Eq. ( 1), which we omit here for lack of space.

A graphical user interface for time-frequency annotation of spectrograms

In this section, we investigate manual annotation of the spectrogram. A GUI was designed in Matlab to annotate spectrograms (see Figure 2), with some extra sound functionalities to help the user. It takes sound files as input, applies some basic preprocessing (re-sampling at userspecified rate, down-mixing to mono), computes a timefrequency representation via user-specified parameters, and displays the spectrogram. Zooming and slide-rule navigation are enabled for better visualization. Annotation of sources is done with a simple rectangle drawing utility : one color for each source, as illustrated in Figure 2. Annotations are stored in an annotation mask of dimension F × N × G (where (F, N ) is the size of the spectrogram and G the number of sources). Several annotation masks may be loaded into memory and displayed alternatively, so the user can compare, for instance, manual annotations with the output of a blind source separation algorithm. Annotation masks may be exported to .mat format for further processing. Finally, we implemented playback functionalities to help the user annotate the spectrogram.

We designed the GUI to make the annotation process easier and faster : indeed, in our experience, while time annotations are easy and require only listening once or twice to the mix, time-frequency annotations are hard even for trained users : it takes up to one hour to annotate 20% of a twenty seconds track.

TOWARDS A SUPERVISED ALGORITHM FOR ANNOTATION

Research in computational audio scene analysis (CASA) has emphasized the role of frequency tracks in source identification : indeed by looking at a spectrogram, it is easy to assign a significant number of frequency tracks either to a voiced source or a musical source (see Figure 1). In previous works, such cues have been used to compute a similarity matrix that would then be used to perform clustering see [START_REF] Lagrange | Normalized cuts for predominant melodic source separation[END_REF][START_REF] Bach | Blind one-microphone speech separation: a spectral learning approach[END_REF]. We propose here a supervised learning procedure to predict annotations automatically. At train stage, we have at hand separate sources so that we observe not only the mix, but also the Wiener coefficients M (g)

f n computed on the ground truth, while at test stage we only observe V . Thus, the goal is to predict E(M (g) |V ). In order to alleviate the computational burden 2 , we make two restrictions on the learning procedure : each vector (M

(1) f n , . . . , M (G) f n )
for a given time-frequency bin (f, n) is predicted independently of the others, and based only on the values of patches centered at that time-frequency bin.

We now introduce the features and algorithms used to train our predictor.

Features

The basic input to our learning algorithms consists in rectangular time-frequency blocks extracted from the input power 2 indeed even for ten seconds' excerpts, there are more than 500 × 1000 time-frequency bins for standard STFT parameters spectrogram. The size of the rectangular blocks is fixed as a parameter of the algorithm. They are then normalized to have unit ℓ 1 -norm so the features are scale invariant. We also considered taking the log of patches, adding coordinates of the patch as additional information, and taking a Gabor transform of the patches. The Gabor transform in particular was introduced so that correlations between pixels in each time-frequency blocks is taken into account. Finally, we also tried averaging the ground truth Wiener coefficients before learning, so that predicted regression surfaces are smoother in time-frequency space.

Learning algorithms

Due to space limitation, we restrict ourselves to naming the algorithms we chose and highlighting the key parameters to tune. We refer the reader to standard textbooks on machine learning for more details (e.g., [START_REF] Hastie | The elements of statistical learning[END_REF]). K-nearest neighbors (knn): for each test point x We will refer to this supervised learning procedure as automatic annotations, no matter which algorithm is used.

Computation of µ f n for automatic annotations

While the learning algorithms presented above predict Wiener coefficients, output values near 0.5 reflect uncertainty in the Wiener coefficients rather than prediction of mixed volumes. For this reason we introduce an additional tuning parameter µ f n in Eq. ( 1), so that output values near 0.5 are less taken into account than values near {0, 1}. As a rule, we choose

µ f n = 1 -G G-1 g M (g) f n (1 -M (g) f n ), so that 0 ≤ µ f n ≤ 1 and µ f n = 0 if all M (g)
f n are equal. Moreover, when annotations are in {0, 1}, we always have µ f n = 1.

EXPERIMENTAL RESULTS

Description of music databases

We used two publicly available databases in our experiments: the QUASI database 3 and the SISEC database for Professionally Produced Music Recordings 4 . All source tracks were down-sampled from 44100 Hz to 16000 Hz, and down-mixed to mono by taking the average of left and right channels. A voice track and accompaniment track are then created by aggregating the various source files, and then a final mix is created by summing the two tracks. Sine-bell windows of size 1024 with 512 overlap were used to compute short time Fourier transforms. The QUASI database contains longer tracks that are amenable to time annotations. The SISEC database contains short tracks where only time-frequency annotations can be used. Although detailed instrumental tracks are provided for most of the mixtures, we work only on single-channel signals.

Since we are dealing with under-determined mixtures, we restrict ourselves to separating voice from accompaniment in each track, in order to alleviate the difficulty of the problem. 1 displays source separation results achieved by semi-supervised NMF on the SISEC database when fed with the actual Wiener coefficients computed from the ground truth sources. Source separation performance is measured by Source to Distortion Ratio (SDR), Source to Interference Ratio (SIR), and Source to Artefact Ratio (SAR). Higher values indicate better performance. As we can see, satisfactory results are obtained with as little as 10% of annotations. When 100% of annotations are given, NMF does nothing and the computed masks are simply the ideal Wiener coefficients computed from the sources.

Ideal performance of semi-supervised NMF and robustness to wrong annotations

SDR1 SDR2 SIR1 SIR2 SAR1 SAR2 0.
We study the robustness of our NMF routine by replacing part of the ideal annotations by noise to simulate human errors. Table 2 displays average SDRs obtained when fixing the annotation rate to 10% and varying either the rate 3 www.tsi.telecom-paristech.fr/aao/ 4 sisec.wiki.irisa.fr wrong annotations p or the optimization parameter λ. As expected, for fixed λ the average SDR drops as p increases. When p is fixed, there is an optimal value of λ that trades off the benefits and drawbacks of annotations. Fixing the target annotation rate to 10%, satisfactory results are obtained with up to 10% of wrong annotations (i.e.1% of the spectrogram). λ p = 0 p = 0.05 p = 0.1 p = 0.2 p = 0. Learning algorithms were trained by dividing the SISEC database in two sets of tracks. For each set, we train detectors and test them on the other set. Thus we may compute annotations and run semi-supervised NMF for all tracks without the risk of overfitting. We emphasize the fact that each track is annotated with a detector that has never seen the spectrogram before : our method is purely supervised with no adaptation to test data. Parameters of the learning algorithms were selected at train stage by cross-validation. Time-frequency patches of size in {4, 8}×{8, 16, 32} were extracted. Out of each track we extract 5 × 10 3 patches at train time, and 10 5 patches a test time, so approximately 10% of the track is annotated at test time when semi-supervised NMF is called.

We display in Table 3 the results of the best 5 detectors, in terms of mean prediction error (first column) and in terms of relative improvement over a purely random predictor. Detectors are named after the following rule : {patch size} {feature} {learning method} {averaging or identical}. For instance, the tag loggabor corresponds to taking log then Gabor transform of patches, and wcoords adding frequency coordinates of the patches as side information. Note that we used exact Wiener coefficients to compute errors, so that all detectors can be compared even when averaging was used at train stage. The improvement over a random predictor is consistent across the features and the algorithms that were used. Figure 4 compares annotations provided by the best detectors from Table 3 with ideal annotations at the same points were automatic annotations were made. Red time-frequency bins correspond to accompaniment, and green to voice. The most striking observation is that, while ideal annotations are in very bright colors (few Wiener coefficients are different from 0 or 1), automatic annotations, on the other hand, are generally biased towards 0.5. This is to be expected since predicting 0.5 incurs a risk of losing at most 0.25 (since we use a regression loss), while predicting 0 or 1 incurs a maximum loss of 1. The main asset of automatic annotations is that pitch tracks with varying frequency are successfully predicted as voice. Automatic annotations are biased towards predicting voice in the higher frequencies : however the learning algorithm in this example did not have the information of frequency. This might be because transients "look" a lot like patches of unvoiced speech. Finally, one may spot inconsistencies in the predictions in the sense that points belonging to the same pitch tracks are sometimes classified incoherently, which is not surprising since the learning algorithms we have proposed predict time-frequency bins independently.

To sum up, predictions of Wiener coefficients from local patches are not perfect but provide a good starting point for further modelling of the spectrogram. We expect that better performance could be obtained by using more advanced cues from CASA, such as pre-clustering the spectrogram into pitch tracks and transient tracks, before learning 5 .

Overall results

We now turn to results obtained by semi-supervised NMF combined with various annotation methods. On the SISEC database, manual time-frequency annotations were done with the GUI presented in Section 2.3. On the QUASI database, tracks were amenable to significant time anno-tations, so by comparing results on both databases we can compare the respective benefits of time-frequency annotations VS time annotations.

In both scenarios, we compare five methods : auto : Automatic annotations and semi-supervised NMF. The best detector from Table 3 was chosen. user : User annotations and semi-supervised NMF (timefrequency annotations for SISEC, manual annotations for QUASI). We tried K ∈ {5, 10, 20} for the SISEC database and {10, 20, 50} for the QUASI database, as well as λ ∈ {1, 10, 100}, and selected parameters yielding highest SDR for fair comparison with the baseline. baseline : Run NMF and permute factors to obtain optimal SDR. We set K = 8 because it already takes a 10 times as long to evaluate SDRs for all permutation on a single track as it takes to run semi-supervised NMF. self : set s (g) = 1

G x as estimates for the sources, it serves to estimate the difficulty of the source separation problem for a given database. oracle : results obtained with Wiener coefficients computed from the ground truth. In addition we display track by track annotation accuracy for user annotations, for comparison with Table 2. For each method, we ran NMF three times for 1000 iterations to avoid local minima, and kept the run with the lowest objective cost value.

Tables 5a and 5b display average evaluation metrics for each source (source 1 is always the accompaniment, and source 2 is always the voice), on two different databases : on the SISEC database, we experimented with time-frequency annotations since the tracks were too short for time annotations. Overall, results on the SISEC database are better than those on QUASI. Our interpretation is that since most of the time the accompaniment is active, the dictionaries tend to overfit the accompaniment and underfit the voice. Time-frequency annotations on SISEC yield SDRs that are a few points below that predicted by our benchmark from Table 2 : indeed human errors are not distributed randomly as was the case in our benchmark. Timefrequency annotations outperform the baseline by 1 point in SDR, which is significant because in semi-supervised NMF there is no manual grouping of the components. Time annotations loose to the baseline by -1 in SDR, but they are still significantly correlated with the true sources when compared with the baseline. On the SISEC database, automatic annotations are also below the baseline, however they are also significantly correlated with the true sources, when compared with the "self" column. Signal to Interference Ratios are even comparable with those of the baseline on the SISEC database. Automatic annotations do not perform as well on the QUASI database since we trained detectors only on tracks from SISEC, so that more supervision would significantly improve those figures.

To conclude, we have shown that time-frequency annotations can improve significantly over NMF with ideally grouped components. On longer tracks, time only annotations yield reasonable results, but even when 100% of the track is annotated, the estimated sources contain strong interferences. Automatic annotations yield similar results, but leave considerable room for improvement, since with time-frequency annotations there will always be a point where enough annotations with limited errors will provide audible estimates of the sources.

CONCLUSION

We have proposed a novel formulation of semi-supervised NMF that successfully takes into account annotations to enhance the discriminative power of NMF. Semi-supervised NMF is defined so that when a certain amount of annotations is reached, source separation quality is near that of ideal binary masks. Manual annotations retrieved with our graphical user interface yield satisfactory results. We are investigating ways to define annotations independently of the particular time-frequency representation that is used.

Finally, semi-supervised NMF opens the way for interaction with methods from computational audio scene analysis. As such, the simple features and textbook pattern matching algorithms we have presented show promising results.

  fn ∈ [0, 1]: this general case is relevant to the learning procedures we introduce in the next section, since they output decision values in [0, 1].

Figure 2 :

 2 Figure 2: Example of user annotations in a ten seconds' audio track: green regions are assigned to voice, and red regions to accompaniment (best seen in color).

Figure 3 :

 3 Figure 3: Samples of patches extracted from the SISEC database. Intensity reflects amplitude, patches which are labeled as accompaniment are in red, while patches which are labeled as voice are in green. Patches in brown have mixed Wiener coefficients (best seen in color).

  j , j ∈ {1, . . . , C} from the train set are used to predict M(g) i = 1/C j M (g) j .Quantized knn (km): We learn C clusters from the train set using K-means; for each cluster, we compute average prediction coefficients M (g) c . For each test point, we predict M (g) c from the nearest cluster c. Random Forests (rf): We learn C regression trees of depth d from the train set and average over the C predictions for each test point.

Figure 4 :

 4 Figure 4: Comparison of automatic annotations and correct annotations (at the same time-frequency bins). Gray-scale time-frequency bins are not annotated, red bins are annotated as accompaniment, green bins as voice(best seen in color).

Table 1 :

 1 Mean results on the SISEC database, as the proportion of annotation increases.

	1 %	-0.02	-0.60	5.15	5.16	3.62	2.33
	1 %	0.70	0.24	4.59	6.25	4.39	2.85
	10 %	6.71	6.68	13.57 16.53	7.95	7.40
	100 % 10.40 10.41 19.88 20.88 11.00 10.88
	Table

Table 2 :

 2 Mean SDR value as λ and the proportion of wrong annotations vary. The proportion of annotations is set to 0.1

	5

Table 3 :

 3 Mean error on Wiener coefficient predictions on the SISEC database (% improvement over random prediction), for various learning strategies .

Table 4 :

 4 Evaluation of user annotations on the SISEC database.

		% annotated % correct
	track 1	0.23	0.91
	track 2	0.10	0.89
	track 3	0.29	0.91
	track 4	0.17	0.81
	track 5	0.22	0.95

Table 5 :

 5 Results on the evaluated databases: (a) time-frequency annotations, (b) time annotations.

Given that some values are set to zero, we replace the IS divergence d IS (x, y) by d IS (ǫ + x, ǫ + y) (where ǫ = 10 -7 ) in our optimization problem, in order to deal with ill-conditioning of the objective function.

This is very similar to what is done in vision, where super-pixels help deal with consistency in prediction and alleviate the computational burden of predicting all pixel values.


Nadia Boukhelifa

Anastasia Bezerianos

Member, IEEE Tobias Isenberg

Member, IEEE Jean-Daniel Fekete

Evaluating Sketchiness as a Visual Variable for the Depiction of Qualitative Uncertainty

Keywords: Uncertainty visualization, qualitative evaluation, quantitative evaluation, perception

We report on results of a series of user studies on the perception of four visual variables that are commonly used in the literature to depict uncertainty. To the best of our knowledge, we provide the first formal evaluation of the use of these variables to facilitate an easier reading of uncertainty in visualizations that rely on line graphical primitives. In addition to blur, dashing and grayscale, we investigate the use of 'sketchiness' as a visual variable because it conveys visual impreciseness that may be associated with data quality. Inspired by work in non-photorealistic rendering and by the features of hand-drawn lines, we generate line trajectories that resemble hand-drawn strokes of various levels of proficiency-ranging from child to adult strokes-where the amount of perturbations in the line corresponds to the level of uncertainty in the data. Our results show that sketchiness is a viable alternative for the visualization of uncertainty in lines and is as intuitive as blur; although people subjectively prefer dashing style over blur, grayscale and sketchiness. We discuss advantages and limitations of each technique and conclude with design considerations on how to deploy these visual variables to effectively depict various levels of uncertainty for line marks.

INTRODUCTION

Information visualization can show not only what we know about the data but also the degree of our confidence in that data. This confidence could be considered as yet another data dimension. However, information on data quality in general-if at all available-rarely comes in numerical format. Qualitative measures of uncertainty are far more common and often come as ordinal meta-data. For instance, a utility company holds a positional confidence attribute for their assets where uncertainty is mapped to five categorical values from least certain to more certain: schematic, assumed, indicative, third party survey, and internal survey [START_REF] Boukhelifa | Uncertainty Visualization -Why Might it Fail[END_REF]. Such ordinal data can be visualized using Bertin's [START_REF] Bertin | Semiology of Graphics: Diagrams, Networks, Maps[END_REF] retinal variables texture, value, or size. When visualizing uncertainty, however, a number of visual variables are considered more 'intuitive' for this domain; examples include blur, sharpness of focus, and color saturation. These variables may bear direct perceptual resemblance to what the uncertainty indicates and, thus, may provide an easier reading of uncertainty [START_REF] Cedilnik | Procedural Annotation of Uncertainty Information[END_REF][START_REF] Maceachren | Visualizing Uncertain Information[END_REF][START_REF] Strothotte | Visualizing Knowledge about Virtual Reconstructions of Ancient Architecture[END_REF]. However, to our knowledge, no formal studies are reported to back up these observations.

In this paper, we investigate sketchiness as a visual variable to depict uncertainty information in line marks such as for graphs, hierarchies and route maps. The design of these sketchy lines is inspired by the field of Non-Photorealistic Rendering (NPR) and by our own observations from analyzing child and adult hand drawings. Our analogy is that the 'cleanness' of the hand-drawn lines corresponds to the quality of the data. We hypothesize that sketchiness is a good metaphor for the qualitative visualization of uncertainty information.

Our contribution is two-fold: [START_REF] Aerts | Testing Popular Visualization Techniques for Representing Model Uncertainty[END_REF] we provide an empirically-based method to generate line trajectories that resemble hand-drawn strokes of various levels of proficiency-ranging from child to adult strokeswhere the amount of deviations in the line corresponds to the level of uncertainty in the data; [START_REF] Osorio | Contouring with Uncertainty[END_REF] we present a qualitative and quantitative evaluation of four uncertainty visualization techniques using blur, dashing, grayscale, and sketchiness. In particular, we attempt to answer the following questions: (a) can people intuitively associate sketchiness to uncertainty, (b) is sketchiness as effective for depicting uncertain information as the other visualization techniques, and (c) which method do people subjectively prefer. The remainder of the paper is organized as follows. After discussing previous work, we first present our model for generating sketchy lines, and then present a series of studies that answer our questions on intuitiveness, accuracy of reading uncertain visualizations, and subjective viewer preference in regards to the different uncertainty visualizations.

RELATED WORK

The previous work that relates to our own can generally be thought of as threefold: (a) work in non-photorealistic rendering that studies the generation of primitives that appear sketchy or hand-drawn as well as their applications, (b) methods to depict uncertainty data, and (c) user studies that examine how people perceive and interpret uncertain data depictions. We discuss each of these three fields in turn.

Sketchy Lines in Non-Photorealistic Rendering

The field of non-photorealistic rendering (NPR) [START_REF] Gooch | Non-Photorealistic Rendering[END_REF][START_REF] Strothotte | Non-Photorealistic Computer Graphics. Modeling, Animation, and Rendering[END_REF] has been inspired by the many ways of traditional depiction that humans have employed over the last decades, centuries, and millennia. As such, computer-generated line drawings have been among the first goals for NPR. Early-on, for example, loose and sketchy line rendering and animation [START_REF] Curtis | Loose and Sketchy Animation[END_REF] have been simulated. Researchers have also developed ways to represent lines such that the line path can be separated from the (sketchy) line perturbation [START_REF] Finkelstein | Multiresolution Curves[END_REF][START_REF] Schlechtweg | Surfaces to Lines: Rendering Rich Line Drawings[END_REF]. These line models can, in turn, be applied to line renderers (e. g., [START_REF] Strothotte | How to Render Frames and Influence People[END_REF][START_REF] Winkenbach | Computer-Generated Pen-and-Ink Illustration[END_REF]) to generate images that resemble-to varying degrees-traditional hand-drawn ones. More recently, NPR line models have been extended to be more examplebased, e. g., by taking the human arm movement into account in their generation [START_REF] Almeraj | Automatically Mimicking Unique Hand-Drawn Pencil Lines[END_REF]-a goal that we share for generating our sketchy lines.

In addition to reproducing a generic hand-drawn look, sketchy nonphotorealistic rendering has also specifically been employed to portray uncertainty. For example, Strothotte et al. [START_REF] Strothotte | Visualizing Knowledge about Virtual Reconstructions of Ancient Architecture[END_REF] used a general level of sketchiness to indicate a general notion of uncertainly in the domain of archeology visualization. They also describe examples that continuously change the line thickness/line saturation or the degree of perturbation in order to visualize continuously changing degrees of uncertainty. In the same domain, Potter et al. [START_REF] Potter | Resolution Independent NPR-Style 3D Line Textures[END_REF] perturb the vertices of a line-based rendering style to control the degree of sketchiness of the depiction, which in turn is used to visually express the level of confidence in a reconstruction. A similar notion of general sketchiness to indicate uncertainty was used by Nienhaus and Döllner [START_REF] Nienhaus | Sketchy Illustrations for Presenting the Design of Interactive CSG[END_REF] for the visualization of 3D shape concepts in CAD-supported by earlier findings of Schumann et al. [START_REF] Schumann | Assessing the Effect of Non-photorealistic Rendered Images in CAD[END_REF]. Instead of these rather implicit notions of uncertainty visualization we are interested in a more precise analysis of how line sketchiness can be employed for qualitatively depicting a number of different levels of uncertainty, and thus in how sketchiness can be used intentionally as a dedicated visual variable.

Visual Variables to Depict Uncertainty

Many visual variables have been proposed for the depiction of uncertainty including Bertin's [START_REF] Bertin | Semiology of Graphics: Diagrams, Networks, Maps[END_REF] retinal variables position, size, and value. To help visualization designers choose a technique from a myriad of possibilities, various taxonomies were proposed [START_REF] Maceachren | Visualizing Geospatial Information Uncertainty: What We Know and What We Need to Know[END_REF][START_REF] Pang | Approaches to Uncertainty Visualization[END_REF][START_REF] Thomson | A Typology for Visualizing Uncertainty[END_REF]. Our aim here is not to replicate these efforts but to provide an overview of widely used methods that are applicable to line primitives. We group these methods into three main categories: (1) color-based techniques that manipulate hue, saturation, or brightness dimensions; (2) focusbased techniques that modify contour crispness, transparency, or resolution; and (3) geometry-based techniques that distort line marks by applying a rendering style as in sketchiness.

Color is repeatedly used to depict uncertainty in information visualization [START_REF] Aerts | Testing Popular Visualization Techniques for Representing Model Uncertainty[END_REF][START_REF] Diepenbrock | Interactive Visualization Techniques for Neurosurgery Planning[END_REF][START_REF] Strothotte | Non-Photorealistic Computer Graphics. Modeling, Animation, and Rendering[END_REF]. Since color scales are readily available in today's graphical packages and libraries, assigning a range of a color scale to a range in the data is straightforward. Amongst all color dimensions, saturation is often preferred because the reduction of color purity conveys more intuitively the notion of degrading data quality or confidence [START_REF] Maceachren | Visualizing Uncertain Information[END_REF][START_REF] Strothotte | Visualizing Knowledge about Virtual Reconstructions of Ancient Architecture[END_REF]. Hue is also used [START_REF] Rhodes | Uncertainty Visualization Methods in Isosurface Rendering[END_REF][START_REF] Xie | Exploratory Visualization of Multivariate Data with Variable Quality[END_REF][START_REF] Zehner | Visualization of Gridded Scalar Data with Uncertainty in Geosciences[END_REF] and rainbow scale is often chosen to map ordinal uncertainty data up to seven levels-despite of its lack of perceptual ordering [START_REF] Borland | Rainbow Color Map (Still) Considered Harmful[END_REF]. For value, darker lines suggest more certainty about an aspect of the underlying data. In this case, however, the line width may need to be adjusted to preserve perceptibility [START_REF] Cedilnik | Procedural Annotation of Uncertainty Information[END_REF]. More advanced color mapping techniques are also deployed including whitening where white pixels are randomly placed or actual hue and white are blended [START_REF] Hengl | Pixel and Colour Mixture: GIS Techniques for Visualisation of Fuzziness and Uncertainty of Natural Resource Inventories[END_REF][START_REF] Hengl | Maps Are Not What They Seem: Representing Uncertainty in Soil-Property Maps[END_REF][START_REF] Pfeiffer | Using Brightness and Saturation to Visualize Belief and Uncertainty[END_REF].

Amongst focus-based techniques, blur, which is defined as the removal of high-frequency spatial detail from the information [START_REF] Brown | Animated Visual Vibrations as an Uncertainty Visualisation Technique[END_REF] has widely been used to indicate fuzziness and ambiguity in the data [START_REF] Botchen | Texture-Based Visualization of Uncertainty in Flow Fields[END_REF][START_REF] Brecheisen | Illustrative Uncertainty Visualization of DTI Fiber Pathways[END_REF][START_REF] Gerharz | Usability of Interactive and Non-Interactive Visualisation of Uncertain Geospatial Information[END_REF][START_REF] Maceachren | Visualizing Uncertain Information[END_REF]. For example, Bisantz et al. [START_REF] Bisantz | Holistic Battlespace Visualization: Advanced Concepts in Information Visualization and Cognitive Studies[END_REF] applied blur to a set of airplane symbols to provide decision makers with a fast way to understand the level and uncertainty of a given threat. Gershon [START_REF] Gershon | Visualization of Fuzzy Data Using Generalized Animation[END_REF] used blurred versions of images in an animation loop from focused to blurry (or the inverse) to draw users' attention to uncertain objects. Kosara et al. [START_REF] Kosara | Useful Properties of Semantic Depth of Field for Better F+C Visualization[END_REF] described their 'semantic depth of field' technique that uses blur to de-emphasize objects of less importance in the scene (de-emphasis of objects that are less important can also be achieved using transparency; e. g., Correa et al. [START_REF] Correa | A Framework for Uncertainty-Aware Visual Analytics[END_REF] map uncertainty to both size and transparency). We use blur as one visual variable in our evaluation and compare it to others like sketchiness, dashing, and value.

Modifying the geometry of line marks can be a powerful way to convey uncertainty. For instance, Griethe and Schumann [START_REF] Griethe | The Visualization of Uncertain Data: Methods and Problems[END_REF] argue that "wavy or dotted lines could convey less trusted relationships." Similarly, Strothotte et al. [START_REF] Strothotte | Visualizing Uncertainty in Virtual Reconstructions[END_REF] show how sketch-like renditions (and the use of transparency) can express the uncertainty in archaeological reconstructions. The domain of NPR also proposed many geometry-based rendering techniques to convey uncertainty as described above. Using drawing primitives from NPR, Pang [START_REF] Pang | Visualizing Uncertainty in Geo-spatial Data[END_REF] used gaps in contour lines of geographic maps to encode uncertainty such that larger gaps in the contour line encode an increased uncertainty. In the related area of oceanography, Osorio et al. [START_REF] Osorio | Contouring with Uncertainty[END_REF] also augmented contour visualizations using uncertainty bands that indicate different possible locations of a contour line. Luboschik et al. [START_REF] Luboschik | Using NPR-Rendering Techniques for the Visualization of Uncertainty[END_REF] suggested the use of dashed and wavy lines to show uncertainty in parallel coordinates. In the context of maritime situational awareness, Matthews et al. [START_REF] Matthews | Improving the Maritime Surface Picture with a Visualization Aid to Provide Rapid Situation Awareness of Information Uncertainty[END_REF] depict the timeliness and quality of sensor information using icon borders-solid or broken. More relevant to our work, however, is the distorted annotation technique by Cedilnik and Rheingans [START_REF] Cedilnik | Procedural Annotation of Uncertainty Information[END_REF] who distort grid lines proportionally to the amount of uncertainty in the data. Similar to our approach for sketchiness, Cedilnik and Rheingans map the amplitude of line distortion to the amount of uncertainty in the data. Our method, however, does not separate the data from the uncertainty depiction be-cause the affected lines themselves are data carriers in our case.

Perceptual Studies and User Evaluations

Several perceptual studies and user evaluations have examined the general application of visual variables in visualizations. It is established that color is a powerful dimension to indicate data quality in general [START_REF] Leitner | Guidelines for the Display of Attribute Certainty[END_REF][START_REF] Maceachren | Visualizing Uncertain Information[END_REF]. Specifically for the context of multidimensional data visualization, Xie et al. [START_REF] Xie | Exploratory Visualization of Multivariate Data with Variable Quality[END_REF] found that hue has stronger capacity to convey quality information for parallel coordinates than brightness or line width-even for large datasets. Color, however, may suffer from the lack of an intuitive order. MacEachren [START_REF] Maceachren | Visualizing Uncertain Information[END_REF] showed, for example, that subjects cannot spontaneously order colors into a legend arrangement for bi-variate choropleth maps but that they can recognize order in that arrangement. The question of user preference is also pertinent to the problem of uncertainty visualization; in a user opinion survey conducted by Gerharz et al. [START_REF] Gerharz | Usability of Interactive and Non-Interactive Visualisation of Uncertain Geospatial Information[END_REF] in the context of geographical information systems, people disliked whitening [START_REF] Hengl | Maps Are Not What They Seem: Representing Uncertainty in Soil-Property Maps[END_REF] to convey uncertainty. The authors argue that the principle of whitening is easy to understand but getting detail information from it is difficult. Li et al. [START_REF] Li | Visualizing Large-Scale Uncertainty in Astrophysical Data[END_REF] investigated the issue of scale for uncertainty visualization for astrophysical data and used a unified color scheme to represent log-scale distances and percentage errors. They found that participants were able to determine the amount of uncertainty using colors with 96.7% success rate. Similarly to the findings by Gerharz et al. [START_REF] Gerharz | Usability of Interactive and Non-Interactive Visualisation of Uncertain Geospatial Information[END_REF], however, access to detail was difficult especially for neighboring color ranges.

Blur as an example of a focus-based uncertainty visualization technique is a well-studied visual variable in domains that go beyond information visualization (e. g., [START_REF] Hamerly | Detection and Discrimination of Blur in Edges and Lines[END_REF][START_REF] Watt | The Recognition and Representation of Edge Blur: Evidence for Spatial Primitives in Human Vision[END_REF]). Kosara et al.'s [START_REF] Kosara | Useful Properties of Semantic Depth of Field for Better F+C Visualization[END_REF] evaluation of the previously mentioned 'semantic depth of field' specifically examined people's ability to read absolute blur levels. Their results show that participants can distinguish between different blur levels with good accuracy but cannot quantify this difference nor identify objects of the same blurriness. Kosara et al. thus concluded that blur can guide attention but is hard to quantify and thus may not be recommended for showing quantitative uncertainty. Moreover, Kosara et al. found that participants disliked looking at blurred objects. These results motivate our work in which we examine whether blur is effective in communicating qualitative uncertainty information.

Few quantitative evaluations exist for geometry-based methods. One exception is the work by Matthews et al. [START_REF] Matthews | Improving the Maritime Surface Picture with a Visualization Aid to Provide Rapid Situation Awareness of Information Uncertainty[END_REF] who found that-in visual search tasks on maps where latency of information is indicated either by color hue (green or gold) or border style (solid or broken)search time was faster with border style than with the color format.

More generally, the choice of which uncertainty visualization method to pick may be task-dependent [START_REF] Sanyal | A User Study to Compare Four Uncertainty Visualization Methods for 1D and 2D Datasets[END_REF]. Evaluating uncertainty depiction techniques for information visualization applications in general is thus challenging. For example, MacEachren [START_REF] Maceachren | Visualizing Uncertain Information[END_REF] states that "for exploratory applications, where there is no predetermined message to communicate, we can not judge uncertainty depictions using communication effectiveness standards. We can only evaluate these depictions in terms of how they alter decision making, pattern recognition, hypothesis generation, or policy decisions." Zuk and Carpendale [START_REF] Zuk | Theoretical Analysis of Uncertainty Visualizations[END_REF] presented a set of heuristics for uncertainty visualization evaluations and stressed the need for more research in human factors and perception. We are inspired by this call for action and conduct a comparative study between four popular visual variables traditionally used to visualize uncertainty including methods from the three described categories. For color-based methods we use the visual variable value (i. e., grayscale), for focus-based techniques we examine blur, and for geometry-based techniques we selected dashing and sketchiness. To our knowledge, there has not been a formal comparative evaluation of the use of the mentioned visual variables to facilitate an easier reading of uncertainty in visualizations that rely on line primitives. Sanyal et al.'s [START_REF] Sanyal | A User Study to Compare Four Uncertainty Visualization Methods for 1D and 2D Datasets[END_REF] comparative study for 1D and 2D data comes close to ours, but they evaluated different visualization methods (error bars, size of glyphs, color-mapping on glyphs, and color-mapping on data surface), and tested for different user-study tasks (search and count). To enable our evaluation of the selected variables we first describe in the next section how to mimic hand-drawn trajectories to be able to create sketchy lines, before detailing our study.

MIMICKING HAND-DRAWN TRAJECTORIES

To to be able to synthesize lines that mimic hand-drawn ones we first studied characteristics of hand-drawn strokes collected using an Anoto pen. We asked 20 people whose ages ranged from five to 47 years (mean of 27.0 years) to trace six different shapes (see dashed blue lines in Fig. 1); each participant first drew with their dominant and later with their non-dominant hand to ensure variations in drawing proficiency. There was no time limit set for the task but participants were instructed to draw as they would normally but they should aim for good accuracy. To facilitate this task, we asked participants to stay within the enclosing red boxes in Fig. 1. In total, we collected 204 hand-drawn shapes.

With the exception of one person, none of our volunteers had a formal drawing training experience.

Our hypothesis was that drawing proficiency can be determined by examining the average deviations of the drawn path from the template path. A quick visual inspection showed that, indeed, the subjectively more proficient-looking drawings deviated less from the template shapes. Fig. 2 shows examples of unfolded hand-drawn shapes, ranked by their average deviation from the target shapes. Within the set of hand-drawn strokes we collected, the average deviations per stroke ranged from 0.55 mm to 6.04 mm (which corresponds to 2.08-22.82 pixels at 96 ppi screen resolution). An error distribution analysis of all stroke control point deviations (signed) showed a normal distribution with a mean of 0.36 mm and a standard deviation of 2.08 mm (min: -13 mm; max: 18 mm). We decided to use the average deviation from a straight line as our way to map levels of uncertainty in data.

Mimicking hand-drawn lines, of course, requires a generative model of hand movement. Flash and Hogan [START_REF] Flash | The Coordination of Arm Movements: An Experimentally Confirmed Mathematical Model[END_REF] described such a model for straight lines, based on minimizing jerk during a stroke from

p 0 = [x 0 , y 0 ] to p 1 = [x 1 , y 1 ],
jerk being the derivative of acceleration. At time t, the model (improved by AlMeraj et al. [START_REF] Almeraj | Automatically Mimicking Unique Hand-Drawn Pencil Lines[END_REF]) generates a point:

x(t) = x 0 + (x 0 -x 1 )(15τ 4 -6τ 5 -10τ 3 ) + D y(t) = y 0 + (y 0 -y 1 )(15τ 4 -6τ 5 -10τ 3 ) + D (1)
with τ = t t f , t f being the time of the end of the stroke, and D a random value in a specified (pixel) range-D adding squiggliness to the line. AlMeraj et al. [START_REF] Almeraj | Automatically Mimicking Unique Hand-Drawn Pencil Lines[END_REF] empirically defined the time sampling parameter δt according to the length of the desired line: δt = 0.5s for lines shorter than 200 pixels, δt = 0.3s for lines of 200-400 pixels, and δt = 0.2s for lines longer than 400 pixels. They also fixed t f = 2s and D within [-5, 5] pixels range based on empirical evidence. Using these parameters in Equation 1 yields multiple points that are connected using a smoothing spline.

Generating Levels for Sketchiness

Our method to produce sketchy lines takes a number of control points generated by the Flash and Hogan model and a value for D, our sketchiness parameter, to produce deviations within [-20, 20] pixels. This range corresponds roughly to the average stroke deviation range we observed from hand-drawn strokes (where the top range of deviation was 22.83 pixels). Since we are essentially adding noise to straight lines to produce sketchy lines that are consistently perceived by viewers as belonging to the same level, we make sure that the mean deviation stays faithful to the input D value. To achieve this correspondence we sample values from a normal distribution where the mean is set to D and the standard deviation is set empirically to D/6. For our studies, we produced sketchy lines where the difference between D value and actual mean is less than 0.1 pixels. This approach can be described as being mean-biased.

An alternative strategy to generate levels consistently, described as max-biased, is to use the maximum deviation. We sample values for the D parameter from a normal distribution where the mean is equal to 0 and the standard deviation is equal to D. In addition, we add a constraint whereby D values outside [-20, 20] pixels are not allowed.

Fig. 3 shows five major levels as generated by the two methods. As D grows, lines under the mean-biased condition get more undulated in order to preserve the overall mean value, whereas sketchy lines under the max-biased condition overall have a more flat appearance with the exception of a few peaks.

We conducted a between-subjects pilot study to compare the two variations. The pilot was identical in setup to the levels study in Section 6.2 and was carried out in two parts. First, participants were shown one line representing the maximum magnitude for max-biased sketchiness (100% our modulus) as well as a stimulus line that they had to express as a percentage of the modulus. Second, a similar prompt was presented for the mean-biased variation. Results from 20 participants (10 per variant) were analyzed and a mean error comparison of the two variations showed that the max-biased variation resulted in smaller errors in the perception of the sketchiness level than the mean-biased variation, and thus we used the former in our studies.

Assessing Generated Sketchiness Quality

We conducted an online study with 40 participants to determine if people are able to tell the generated lines (using the max-biased approach) apart from hand-drawn ones. The details of our general study setup are reported in Section 4.2.

We selected 45 samples from our library of hand-drawn strokes of three shapes (hexagons, rounded rectangles, triangles) from a deviation range between 0.9 mm and 4.9 mm (within the range we observed in real strokes), and generated the same shapes using the generative model (Fig. 4). For the hand-drawn condition we selected shapes with complete outlines and no noticeable variations in width or overshoots in corners (as much as possible). For the sketchy shapes, we varied D within [0.4, 10] pixels to obtain different sketchiness levels. Our hypothesis was that people cannot tell computer-from human-generated shapes apart for D values of up to 6 pixels . This hypothesis is based on our own subjective observations that shapes start to look synthetic at D ≈ 6: for larger D values, corners start to loose their shape and knots start to appear, something that is not associated with hand-drawn strokes. The null hypothesis was, thus, for a given sketchiness D, participants will be unable to distinguish machine-generated strokes from human-made ones (i. e., selecting the right answer 50% of the time).

The results of the online study show that, overall, 60% of the computer-generated shapes were identified as hand-drawn ones, and 71% of the real hand-drawn shapes were correctly identified as such. A one-tailed non-parametric Bionomial test rejected the null hypothesis (p < .0001), showing that participants were significantly more likely to think the computer-generated lines were human-made (thus the effect was not due to chance). Nevertheless, we did not find an effect of sketchiness level, indicating that regardless of D (i. e., even with the highest level of sketchiness tested) participants were still more likely to perceive the shapes generated by our algorithm as hand-drawn.

USER STUDIES: DESIGN GOALS AND REQUIREMENTS

Using our results from mimicking sketchy lines based on empirically determined parameters, we can now pursue our main objective: investigate whether sketchiness can be used as a visual variable to encode qualitative uncertainty data in information visualization. For this purpose we compared sketchiness to popular uncertainty visualization techniques with respect to three criteria: intuitiveness, accuracy, and subjective user preference. For intuitiveness, we wanted to investigate if and to what extent people associate sketchiness (and the other variables) to uncertainty. For accuracy, we wanted to investigate how accurately people can read values from uncertain line marks (characterized by the use of blur, dashing, grayscale, and blur) and the number of distinctly perceivable levels for each technique. Finally, we wanted to study which technique would be preferred by participants to encode uncertainty.

Generating Levels for Blur, Dashing, and Grayscale

We generated all blurred, dashed, and grayscale lines using the GNU Image Manipulation Program (GIMP). We generated 20 levels per visual variable; each line was 400 pixels long, three pixels wide, and used black except for the grayscale case. Fig. 5 shows the minimum and maximum levels for each visual variable (excluding minima).

For blur, we used a Gaussian filter for which we varied the horizontal and vertical blur radius in equal amounts from 1 to 20 pixels. For dashing, we selected a style where the dash/gap length grows proportionally with each consecutive level. The minimum dash/gap size was set to 3 pixels and the maximum to 60 pixels. For grayscale, we partitioned a linear grayscale from 4% up to value 90% into 20 discrete levels from black to white since an increase in whiteness may be seen as an increase in uncertainty in the data [START_REF] Hengl | Maps Are Not What They Seem: Representing Uncertainty in Soil-Property Maps[END_REF]. As described in Section 3, the 20 levels for sketchiness were produced using max-biased generation, with D ∈ [-20, 20].

Setup of the Studies

Our studies were conducted on Amazon Mechanical Turk (AMT), inspired by previous graphical perception experiments [START_REF] Heer | Crowdsourcing Graphical Perception: Using Mechanical Turk to Assess Visualization Design[END_REF]. In total, we deployed seven experiments and participants in one study could not take part in any of the other ones. We had 1176 participants in total, and subjects were paid on average $ 0.34 per Human Intelligence Task (HIT). To ensure that our sample participant size was representative of the general participant pool, we conducted our studies in increments (three to five blocks). For each block we calculated the mean of what we are measuring (e. g., user subjective preference per technique) and stopped the experiment after we noted stabilization (i. e., discrepancies in mean values consistently inferior to 10%). This sampling procedure was followed throughout our experiments. Given the simple nature of the perceptual studies we were carrying out, no qualification tests were required to complete our HITs. In accordance with AMT guidelines, however, only workers with 95% or more HIT approval rate were allowed to participate. Furthermore, we dismissed work were the participant's written language was poor, indicating a possible language barrier affecting comprehension. Finally, for verification purposes, we included an unrelated answer when presenting workers with a list of options (for studies in Section 5.2 and Section 5.3). Workers were not paid for HITs where they picked the control answer and their data was not included in the analysis.

ON THE QUESTION OF INTUITIVENESS

The question of whether people intuitively associate sketchiness with uncertainty is pertinent to our evaluation of sketchiness as a visual variable. We define intuitiveness as the spontaneous association between signifier and signified: signifier being sketchiness and signified being uncertainty. If we find that there exists a spontaneous association between the two we could hypothesize that people can intuitively associate sketchiness with uncertainty. In that case they do not need to consult a legend to identify the meaning of the visual variable. Note that at this stage we are interested in the meaning associated with the representation-that it is uncertain data-and not its perceived magnitude, i. e., how uncertain it is. We, thus, conduced a study to examine the intuitiveness of sketchiness as a visual variable for uncertainty. This study was conducted in three parts. Each participant only completed one of the three parts, and each part was conducted as a between-subjects experiment.

In Part I, participants were shown visualization scenarios that included sketchiness and were asked in an open-ended question to explain the meaning of sketchiness. Their answers were then used to establish the main categories of interpretations that people spontaneously associate to sketchiness and allowed us to examine whether uncertainty is prominent among them. In Part II we ran a similar study, but participants were presented with a closed list of possible interpretations for sketchiness (multiple choice), established based on the results of Part I. The goal of Part II was to check if the extent of this association changes with the introduction of limited alternatives. In Part III we compared sketchiness to blur-a visual variable that is highly regarded in the literature as being congruent with uncertainty depiction [START_REF] Cedilnik | Procedural Annotation of Uncertainty Information[END_REF][START_REF] Maceachren | Visualizing Uncertain Information[END_REF][START_REF] Strothotte | Visualizing Knowledge about Virtual Reconstructions of Ancient Architecture[END_REF]-using the same closed list of categories as in Part II. This list was general enough to apply to both visual variables.

For this series of studies, and based on our own experience and observations from related work, we hypothesized that: H1 People are likely to spontaneously associate sketchiness to uncertainty in the data.

H2 People are more likely to associate sketchiness to uncertainty for abstract contexts (e. g., hierarchy) than for non-abstract contexts (e. g., map) where geometry already has an inherent meaning.

H3 Participants are more likely to associate blur with uncertainty than they are to associate sketchiness with uncertainty.

Part I: Sketchiness

Scenarios. We asked people to look at different visualizations with lines as a major visual feature and to interpret what sketchy lines mean to them. We designed six different scenarios, the first four being abstract contexts (Fig. 6) and the last two being non-abstract contexts (Fig. 7): (S1) a bar chart where sketchy lines were applied to the contours of some of the bars, (S2) a family tree where sketchiness was applied to some connections between parents and children, (S3) a social network graph where sketchiness was applied to certain connections between nodes (i. e., to relationships between two connected people), (S4) a Venn diagram where sketchy outlines were applied to (parts of) the outlines of some ellipsoids that indicate set memberships, (S5) a rail network where sketchiness was applied to some of the links between two train stations, and (S6) a utility map where sketchiness was applied to line representations of some buried assets.

For each scenario we generated five image variations, changing the percentage of lines in the visualization that were represented as sketchy (between 10% to 50% of all lines in the visualizations). The sketchy lines themselves were hand-drawn with an Anoto pen. Participants only saw one variation of each scenario.

Participants, Study Design, and Procedure. 210 participants took part in this study. Participants were split into six groups of 35 people. Each group was exposed to one of the different scenarios described above and each participant in the group saw a single variation of the scenario. Participants were first introduced to the task, provided with a short scenario description, and then presented with an image as described above. We asked participants to type two different interpretations into a text box stating what these lines convey to them. We highlighted each of the sketchy lines in question using a red arrow. Overall, our experiment consisted of: 6 scenarios × 35 participants per scenario = 210 trials in total After discarding entries that obviously did not represent an interpretation (e. g., jokes, unrelated answers) [START_REF] Willett | Scented Widgets: Improving Navigation Cues with Embedded Visualizations[END_REF], we could analyze a total of 180 trials, 30 for each scenario.

Results. We carried a qualitative evaluation of the results; one of the authors coded all interpretations; then a second author independently encoded 40% of the total number of interpretations. The concordance rate between the two encoders was around 74%. Coding conflicts were resolved and we were able to identify six major categories of interpretations for sketchiness as given below with their association rate (derived using the mean for both interpretations combined). In contrast to what we stated in our first hypothesis H1, we found that only 11.7% of people associate sketchiness to uncertainty, less than we expected. However, we found no effect between the amount of sketchy lines in an image and the number of interpretations that associated it with uncertainty. The results for all categories are listed below.

• Alternative (36.7%): a different relation from what is conveyed by a straight line, which includes 'geometry' where sketchiness is attributed to the actual shape of the displayed feature.

• Qualitative (23.3%): the same relation as for a straight line but with emphasis on a particular quality other than uncertainty. Often, this was a negative quality.

• Ignore (13.9%): the exact same relation as for straight lines (no added information).

• Uncertainty (11.7%): all data quality descriptors that are related linguistically to the term uncertainty such as ambiguity, vagueness, impreciseness, doubt, or unreliability.

• Style (10.0%): intended drawing style; e. g., to draw attention to a particular part of the image.

• Glitch (4.4%): unintended style; i. e., human or computer error.

It was interesting that some participants did not spontaneously associate any semantic meaning with sketchiness (i. e., the 'ignore' category above). Because we were interested in meaningful interpretations, however, we decided to exclude this category from the list for the next two primed studies and thus to only use the remaining five categories.

Part II: Primed Sketchiness

Similar to Part I of the study in Section 5.1, we asked people to look at a line drawing and interpret what the sketchy line might mean. We provided participants with the five-category list established above, further adapted for sketchiness abstract (SA) and non-abstract contexts (SNA) as described below.

Participants, Study Design, and Procedure. 168 participants took part in this study, adopting a between-subjects design. The design of the study was identical to the study in Part I, with the exceptions that we asked participants to pick an interpretation from the provided list of options that we adapted to include the 'geometry' category for SNA contexts S5 and S6. Note that, rather than the general category names, we provided specific example interpretations that were repeatedly given by participants in Part I. For instance, for the 'geometry' category in scenario S5 we gave 'crooked or un-smooth train tracks' as the corresponding possible interpretation. Thus overall we had: 6 scenarios × 28 participants per scenario = 168 trials in total (1/3 for SNA scenarios) Results. Overall, people attributed sketchiness first to style (36.3%) and then to uncertainty (22.0%), showing an increase for uncertainty under the closed list condition. The next category scores, in decreasing order, were qualitative (13.69%), alternative (12.50%), geometry (10.12%), and glitch (5.36%). Separating SA and SNA scenarios, we found an important difference: in agreement with our hypothesis H2, 28.6% of participants under the SA condition attributed sketchiness to uncertainty in comparison to only 8.9% for the SNA condition. This may have implications on the type of visualization context for which sketchiness is more intuitive as a visual indicator of uncertainty.

Part III: Primed Blur

We replicated the previous study (Part II) for the blur visual variable, therefore the tasks are the same as in Section 5.2.

Participants, Study Design, and Procedure. 168 participants took part in this study, adopting a between-subjects design. Participants were split into six groups with 28 people per group. Each group was exposed to one of the different scenarios S1-S6. Participants were first introduced to the task, were given a short description of the scenario, and were then presented with an image in which we had applied a Gaussian blur to one line (as outlined in Section 4). We asked participants to pick a single interpretation from the five-category list in Section 5.1. We highlighted the blurred line in question using a red arrow each time. Thus overall our experiment consisted of: 6 scenarios × 28 participants per scenario = 168 trials in total (1/3 for non-abstract scenarios) Results. Overall, people attributed blur mostly to style (26.2%), similar to the results for sketchiness. The second-most frequent attribution was a qualitative measure (23.8%) and only the third-most frequent one was uncertainty (22.6%). The least frequent attributions for blur were alternative (17.26 %) and glitch (10.12 %). Therefore, contrary to our hypothesis H3, sketchiness appears to be as intuitive as blur for the tested categories.

ON THE QUESTION OF PRACTICAL CONSIDERATIONS

After having established that sketchiness is just as associated with uncertainty as blur, we can now investigate the practical aspects of using sketchiness for this purpose, or as a visual variable in information visualizations in general. More specifically, we are interested in determining if we can express a scale using sketchiness as accurately as when using one of the other three visual variables traditionally used to represent uncertainty (blur, grayscale, and dashing). We thus compare the four techniques with respect to: (i) how close their perceived values are to the actual visual variable value; (ii) how many distinct levels can be identified by participants for each technique to use them in ordinal scales; and in (iii) whether the perception of these techniques follows previous models that express this perception mathematically.

Hypotheses

Based on our experience and observations from related work, we hypothesized that a few discrete levels of line sketchiness can be perceived but that people cannot accurately quantify the amount of sketchiness applied to a line. This led to the following hypotheses: H4 Overall, people cannot accurately estimate the exact level for all techniques (sketchiness, blur, grayscale, and dashing), but this perception error will be different between techniques.

H5 People can discriminate between at least three levels of sketchiness but more levels for the remaining techniques.

H6

We can express the relationship between reported and actual levels for all visual encoding techniques (including sketchiness) using a mathematical model.

Task

To assess the visual perception of the techniques we conducted a study with tasks from psychophysics, the domain that focuses on measuring relationships between perceived and actual properties of visual objects [START_REF] Goldstein | Sensation and Perception[END_REF][START_REF] Wagner | The Geometries of Visual Space[END_REF]. Of the methods that help assess a viewer's visual perception of an object compared to its subjectively experienced magnitude, numeric estimation methods are most relevant to our research and have been frequently used in in the past (e. g., [START_REF] Goldstein | Sensation and Perception[END_REF][START_REF] Wagner | The Geometries of Visual Space[END_REF]). Participants are shown a standard modulus object with an assigned value (e. g., 100% sketchiness) and are then asked to assess a second object (the stimulus) and assign it a value based on the modulus (e. g., a percentage).

In our study, participants were shown one line representing the maximum magnitude for the specific visual variable technique (100% our modulus) as well as a stimulus line that they had to express as a percentage of the modulus. We used the previously generated 20 magnitude levels for stimulus lines for all our visual variables. These 20 stimulus levels correspond to 5-100% of the modulus object. Participants were also shown a line representing the minimum (0%) for the technique which, for all techniques, was a three-pixel-wide straight black line.

Participants, Study Design and Procedure

160 participants took part in this study. A mixed factorial design was used: the visual variable was treated as a between-subjects factor and the level was treated as a within-subjects factor. Participants were split into four groups of equal sizes. Each group was exposed to a different visual variable and each participant conducted magnitude estimation tasks for all levels using that technique.

Participants were first introduced to the task. They were then presented with lines of different levels of their respective visual variable and were asked to mark on a scale from 0% to 100% how sketchy, for instance, each line was with respect to the maximum modulus line which was always visible (Fig. 8 for an example HIT). The order of presentation of the different levels was randomized across participants and techniques. We tested a total of 21 levels per technique (including level zero). Our experiment thus consisted of: To balance our data, we conducted our level experiments in four steps; we ensured we had 40 completed and valid HITs for one visual variable before progressing to testing the next visual variable.

Results

The metrics used in our analysis were the absolute perception error AbsErr and the perceived magnitude level PerMag of a line. As done by similar magnitude estimation studies (e. g., [START_REF] Cleveland | Graphical Perception and Graphical Methods for Analyzing Scientific Data[END_REF]) we define AbsErr as the absolute difference between the true percent of the stimulus compared to the modulus and the reported estimation percent |reported leveltrue level|. Absolute magnitude errors have a skewed distribution and, as suggested by Cleveland and McGill [12], we normalize it by using the log variation of this metric for our analysis log 2 ( 1 8 + AbsErr). The means reported here are before normalization. Trials were marked as outliers when metrics were beyond two standard deviations from the mean for a given technique and level. 161 trials (5% of all trials) were identified as outliers and removed from further analysis. We performed an ANOVA and post-hoc pair-wise mean comparison p-values are adjusted using the Bonferroni criterion.

Perception Error

We first examined how closely participants came to predicting the real level value for each technique, conducting an analysis on AbsErr. The overall AbsErr was higher for sketchiness (16.6%), followed by dashing (13.2%), blur (12.7%), and grayscale (12.3%).

AbsErr increased steadily from 5% to 20% overall as we increased the level up to 11 (half of maximum magnitude). Then we observed a steady decrease in error for all techniques from 20% to 15% for level 19, and almost 5% in level 20. The ANOVA showed that the main effect of technique on AbsErr was statistically significant (F 3,156 = 4.890, p < .01). A significant interaction effect between technique and level was also present (F 60,3120 = 6.364, p < .001). Post-hoc comparisons showed that level predictions with sketchiness were significantly further from the true values than for the other techniques, but this was true only after level ten, with no difference between techniques for levels up to ten (all p < .05). This evidence supports H4.

Level Groupings per Technique

We then tried to determine for each technique how many levels participants can distinguish comfortably and which level values are good candidates for an ordinal scale, assuming that our data are an accurate representation of real perception effects for the different visual variables. We thus conducted the following exploratory analysis:

We first plotted the mean and standard deviation of the PerMag of each level per technique (Fig. 9) and identified possible level values that are distinguishable between them. We chose as candidates levels that have similar standard deviations to their neighbors (to ensure we select representative levels for their neighborhood). Starting from the lower chosen level, we chose the next one such that the mean of the first did not overlap with the mean and the standard deviation of the next selected level, and so on. This process gave us initial estimates for the number of levels that can be distinguished per visual variable and the level value that corresponds to them (see Fig. 9(a)-(d) with the distinct levels being highlighted).

We then ran an ANOVA to see what levels were significantly different with respect to their perceived magnitude PerMag. We plotted the levels that are not significantly different on the matrices in Fig. 10 where gray squares indicate pairs of levels that were not significantly different and risk being perceived as similar (and white pairs that are significant). By inspecting these images of statistical results, we can see visual clusters of levels whose mean values are similar and thus should not all be used to represent two distinct levels. We ensured that our chosen levels (in black in Fig. 10) fall under different such visual clusters, to further supporting our level choices. In comparing the four matrices for the different techniques, sketchiness appears to have similar size groups but different structure especially towards the mid-scale. This may be attributed to the slightly higher perception error for sketchiness after level ten (as previously explained in Section 6.4.1). Our ANOVA was conducted with visual variable treated as a between-subjects variable and level as repeated measures. There was a main effect of technique on AbsErr (F 3,156 = 24.27, p < .0001) and, more importantly for us, its interaction effect with level was also significant (F 60,3120 = 6.047, p < .0001), indicating that level perception was different between variables. A post-hoc comparison showed levels that were clearly different in their perception from each visual variable technique (all p < .05) and are seen in white in the matrices.

We then ran a hierarchical clustering algorithm across level means for each visual variable to ensure that the levels we chose fall under different clusters found by the algorithm (Table 1). This was indeed the case, further supporting our level choices. Based on our generation process, in Table 1 we propose levels that are clearly distinguishable by technique. These levels are illustrated in Fig. 11. Indeed, at least three (four counting the maximum sketchiness level) levels were identified for sketchiness with four levels for blur and grayscale (supporting H5).

Finally, to further support our findings regarding the established levels per technique, we ran a within-subjects study on AMT with 40 participants. The task was to ascendingly order lines by visual variable level by assigning a suitable rank to each line (e. g., rank one for the least blurry line and rank four for the most blurry). The order of lines and techniques was randomized. Participants took on average Table 1. Distinct levels, representative values and their clusters. 126.3 seconds to complete all four parts of the study, where each part corresponds to one visual variable. Our results show that participants were able to recognize the correct order of levels (as established in Table 1) for all techniques with great accuracy (100% for both blur and grayscale, 95% for dashing and 90% for sketchiness). These results provide further validation for our established levels per technique.

Mathematical Description of Perceived Techniques

Previous work has attempted to mathematically describe the differences between physical and perceived magnitude of objects as collected from user studies. One popular function describing this difference is Stevens' [START_REF] Stevens | Psychophysics[END_REF] power law: J = λ D α , with J = judged magnitude, D = actual magnitude, α = exponent, λ = scaling constant. Wagner [START_REF] Wagner | The Geometries of Visual Space[END_REF] provides a meta-analysis of articles reporting values for α collected under different conditions. To the best of our knowledge, no conditions matched dashing, blur, or sketchiness; nevertheless there are values for grayscale. Stevens and Galanter [START_REF] Stevens | Ratio Scales and Category Scales for a Dozen Perceptual Continua[END_REF] found an exponent of 1.2 for black-gray-white series. Given the previous discrepancies between the α varying across experiment setups [START_REF] Wagner | The Geometries of Visual Space[END_REF], we decided to mathematically describe the perceived magnitude of all four different visual variables. An initial curve fitting for the four techniques indicated that, indeed, a power model best fits our data (for all visual variables their respective fits had R 2 > .9, p < .0001). Using the parameter estimates from the fit we conducted a detailed regression analysis on our data to verify the mathematical relationship. Regression analysis for each of the techniques showed a very good fit (all R 2 > .9 and all adjusted R 2 > .9, p < .0001), as hypothesized (H6). Our results and the used coefficients are summarized in Table 2.

J = λ D α Regression λ α R 2 Adjusted

Discussion on Levels

We found that the perceived sketchiness is farther from the real value of a level than for the other techniques, but not by much (4%). This prediction error varied across techniques and was always > 10%. This value is somewhat large and can be too prohibiting if participants attempt to accurately retrieve values from the visual representations of these techniques in real life applications (H4). We thus believe that all techniques should be used for ordinal scales but not for value retrieval.

As in previous studies, our perceived variables can be mathematically modeled using a power law (H6). The fact that our coefficients do not match others on grayscale as reported by Wagner [START_REF] Wagner | The Geometries of Visual Space[END_REF] can be attributed to differences in study setups (as is often the case for perception experiments). An interesting observation is that the perception of sketchiness is fairly close to linear. This could be explained by our generation model that defines a sketchiness level based on the maximum distortion, as opposed to an average distortion across a line.

Using exploratory methods we were able to identify distinct levels for each of the visual variables (at least three for sketchiness-H5). We note that our selection is somewhat conservative. We base our selection on data that assume that each of the selected levels will be compared independently with the min/max values (e. g., when seeing a sketchy line in a visualization the user will immediately be able to determine if it indicates low, medium, or high uncertainty). Nevertheless, if viewers are presented with visual representations of other levels (e. g., comparing the sketchiness between two lines in a visualization, or between a line and a legend), it is possible they will be able to distinguish even more levels. This requires further investigation.

ON THE QUESTION OF PREFERENCE

One issue that still needs to be investigated is which techniques are preferred by people for visualizing uncertainty. Based on our own results from the intuitiveness study in Section 5 and also based on related work (notably on blur [START_REF] Kosara | Useful Properties of Semantic Depth of Field for Better F+C Visualization[END_REF]) we had the following hypothesis: H7 People prefer sketchiness to blur; but dashing and grayscale may be preferred overall due to people's familiarity with these styles.

Task

To judge people's preference about uncertainty encoding we asked participants to select one of four visual styles-blur, dashing, grayscale, or sketchiness-in a visualization scenario. To motivate participants into thinking about their choice and to avoid random answers we asked participants to justify their choice.

Participants, Study Design and Procedure

129 people participated in this study, using a between-subjects design.

Participants were split into six groups of equal size. Each group was exposed to one of the six different scenarios from our first study (Section 5). Participants were first introduced to the task, and a short description of the scenario was given. They were then presented with four side-by-side versions of the same image which only differed in the rendering of a single line (depicted using blur, dashing, grayscale, or sketchiness). All values of the visual variables were selected from the middle ranges for consistency. Participants were informed that the study is about comparing different styles to show uncertainty using the aforementioned visual variables. They were asked to choose their preferred style and were requested to justify this choice. We highlighted the uncertain line in question with a red arrow in each image.

Overall our experiment consisted of:

6 scenarios × 32 participants per scenario = 192 trials in total

Results

We ran our study in batches of 48 trials. For each batch we calculated the mean for each visual variable. After the third batch the means stabilized. Our results show that dashing is the preferred style for the participants (chosen by 68.3%), in agreement with our hypothesis (H7). However, blur (chosen by 15.10%) did better than both grayscale (chosen by 12.5%) and sketchiness (chosen by 3.12%).

We were able to process comments from 155 participants (80.7% of the responses). A closer look at their comments shows that the primary reason for preferring dashing over the other visual variables was 'noticeability' as participants valued the ability to easily distinguish between data and uncertainty (26.5% of our participants), whilst blur was chosen because it was regarded as congruent to what uncertainty conveys in terms of vagueness and reduced precision (4.5% of our participants). Those who preferred grayscale did so because they deemed it easy to understand (2% of our participants).

We only had four comments on the use of sketchiness, two of which argued that it is an intuitive way to represent instability in the data. More interesting, and which may explain the low preference score for sketchiness, are comments by participants who argued against the use of sketchiness. The primary reason for this seems to be that sketchiness tends to imply a disliked notion of 'unprofessionalism' or informality. Some participants thus disregarded it as a standard visual variable (4.5%). This highlights the need for paying attention to the context in which sketchiness is applied as a visual variable to denote uncertainty. We note that abstract and non-abstract contexts did not show different results in terms of user preference. Application contexts however may be a more influential factor in terms of impact on user subjective influence as will be discussed in the next section.

DISCUSSION AND CONCLUSION

In this paper, we have reported on seven user studies to investigate the appropriateness of using sketchiness as a dedicated visual variable for depicting uncertainty. We started by studying characteristics of collected hand-drawn strokes. Findings from this evaluation and existing work on a generative model of hand movement fed into our method for synthesizing sketchy lines that mimic stroke characteristics of handdrawn lines. Our mapping for uncertainty depiction consists of varying the amount of pixel deviations from a straight line in accordance to the amount of uncertainty in the data. An online study using Amazon Mechanical Turk provided evidence that lines generated by our model were significantly more often thought to be handwritten than not.

Sketchiness as a Visual Variable

We then investigated whether sketchiness is an appropriate visualization technique for representing uncertainty. The three questions we attempted to answer were:

(a) Can people intuitively understand sketchiness as an indicator of uncertainty? In a series of studies we found that if users are asked what a sketchy line represents, their most common reaction is that it is associated with a different semantic relation from what is conveyed by a straight line (other than uncertainty). When given a multiple choice of possible explanations for sketchiness, the number of people who associated sketchiness with uncertainty almost doubled. This was lower than expected; nevertheless, a similar study showed a very similar effect for blur, a visual variable which is traditionally associated with uncertainty in visualizations. Thus, although for the majority of people sketchiness was not associated with uncertainty, we feel that given a legend, sketchiness is a viable alternative to other visualization techniques such as blur, dashing, or grayscale, which can now be used to represent other types of information in the data.

(b) Is sketchiness as effective for depicting uncertain information as the other visualization techniques? Our results were encouraging for practically applying sketchiness as a visual variable to information visualizations. We compared sketchiness with other visual variables in a magnitude estimation task (it is very common in information visualization, e. g., to compare a given object to a legend). Our studies showed that sketchiness was only slightly more error-prone than other techniques. In fact, none of the techniques was very accurately perceived. This indicates that none of them is particularly well suited for true value retrieval, and the common practice of using them in small ordinal scales of uncertainty is indeed a sound one. Our mathematical model for predicting all visual variables indicates that this error is predictable for all techniques (including sketchiness). To help uncertainty visualization designers we proposed, based on our findings, a number of levels for ordinal scales for each visual variable (three or four excluding maxima). The number of levels we identified and proposed are close in number between sketchiness (three) and other techniques. Our process for selecting levels is based on the assumption that estimates are made when comparing to a maximal value object.

(c) Which method do people subjectively prefer? Sketchiness was not the preferred visual variable to encode uncertainty (dashing was preferred). Moreover, and in the context of our provided scenarios, some participants commented negatively on the 'informal' and 'unprofessional' look of sketchy lines. There are, however, some contexts where informality is advantageous and thus where sketchiness may be a more preferable choice such as for conveying initial design ideas [START_REF] Schumann | Assessing the Effect of Non-photorealistic Rendered Images in CAD[END_REF]. We thus conclude that sketchiness as a visual variable is a viable additional choice for depicting uncertainty in ordinal data, but further work is needed to investigate the appropriate visualization contexts where it can be deployed.

Contexts and Design Guidelines

Translating our findings into guidelines, we recommend sketchiness for conveying qualitative uncertainty on lines with certain caveats:

• for uncertainty depiction, sketchiness-as well as the other variables we studied-require a legend; • sketchiness is as intuitive as blur; • sketchiness is not suitable for spatial contexts such as maps to indicate uncertainty, because changes in the geometry tend to be perceived as related to the actual underlying spatial features; • conversely, sketchiness is more appropriate in abstract contexts such as hierarchies and diagrams where accurate readings of intermediate line points is not required, but not for line graphs for instance where geometry is also perceived as related to values change; • sketchy lines with high perturbations occupy more space and may increase clutter; therefore, sketchiness with many levels of encoding may not be suitable for dense displays (the same applies for dashing); • sketchiness-as well as the other variables we studied-can only convey 3-4 levels reliably; • sketchiness is perceived as informal; this may be desirable in some contexts (e. g., when showcasing preliminary designs) and less in others; and • sketchiness is not the preferred encoding for uncertainty but is acceptable unless the context requires formality.

Limitations and Future Work

We do not make the claim that our results are generalizable to all kinds of sketchy lines; our notion of sketchiness is specifically defined by a range of pixel deviations from a straight-line and implemented using a sampling model that is max-biased. Other methods to generate sketchy lines-for instance the mean-biased method-may yield different results notably for the number of levels that can be distinguished by the viewer. Sketchiness is a promising technique and we hypothesize that it can be extended beyond the tested deviation range to potentially support more levels.

To determine the number of levels that can be visually distinguishable we asked participants to carry out a low-level perceptual task. It would be interesting to test for higher-level tasks such as search tasks in various information visualization contexts; and to study the effect of shape and area on the perception of the sketchiness level. Another extension to this study would be to control for line width, especially for dashing, where it is likely that both the length and width of the line play a role in the perceptibility of its connectedness; and to add repeated trials in order to show the stability of the perceived levels.

Finally, we focused in this paper on sketchiness as a visual variable to encode uncertainty information; it would be interesting to investigate the link between sketchiness and engagement, in particular in cases where informality is advantageous, such as for conveying initial design ideas. Such a more general look at sketchy rendering in the context of information visualization has recently been carried out [START_REF] Wood | Sketchy Rendering for Information Visualization[END_REF] which looks beyond our focus on uncertainty and also investigates higher-level tasks, indicating that indeed engagement is affected.

Fig. 1 .

 1 Fig. 1. An eight-year-old drawing data sheet (traced with her left hand).

Fig. 2 .

 2 Fig. 2. Examples of unfolded hand-drawn strokes which we ranked by their average deviation (in mm) from their target shapes.

Fig. 3 .

 3 Fig. 3. Ranges (in pixel) of sketchiness used in the study for the meanbiased [A] and the max-biased [B] variants.

Fig. 4 .

 4 Fig. 4. Examples of hand-drawn (odd-numbered) and generated (evennumbered) shapes: each two variants have similar average deviation.

Fig. 5 .

 5 Fig. 5. Ranges of levels for the four visual variables used in our study (excluding minima).

  (a) (S1) Bar chart. (b) (S2) Family tree. (c) (S3) Social network. (d) (S4) Venn diagram.

Fig. 6 .

 6 Fig. 6. The four abstract scenarios used in the study.

Fig. 7 .

 7 Fig. 7. The two non-abstract scenarios used in the study.

Fig. 8 .

 8 Fig. 8. The user interface for the sketchiness levels study showing the instructions and legend (left) and a magnitude estimation task (right).

4

  techniques (dashing, blur, grayscale, and sketchiness) × 21 magnitude levels (including level zero) × 40 participants per technique = 3360 trials in total

  Real vs. estimated sketchiness.

Fig. 9 .

 9 Fig. 9. Real vs. estimated visual variables (red cirles for distinct levels; error bars represent one standard deviation).

Fig. 10 .

 10 Fig. 10. Black squares show our chosen representative levels, and gray ones indicate levels that are not significantly different.

Fig. 11 .

 11 Fig. 11. Visual illustration of the established levels per technique.

Table 2 .

 2 Relationships and regression results. All shown results are statistically significant at the 99% level.

	R 2