Evanthia Dimara
email: evanthia.dimara@gmail.com

Steven Franconeri
email: franconeri@northwestern.edu

Catherine Plaisant
email: plaisant@cs.umd.edu

Anastasia Bezerianos
email: anastasia.bezerianos@lri.fr

Pierre Dragicevic
email: pierre.dragicevic@inria.fr

A Task-based Taxonomy of Cognitive Biases for Information Visualization

Keywords: cognitive bias, visualization, taxonomy, classification, decision making. !

Information visualization designers strive to design data displays that allow for efficient exploration, analysis, and communication of patterns in data, leading to informed decisions. Unfortunately, human judgment and decision making are imperfect and often plagued by cognitive biases. There is limited empirical research documenting how these biases affect visual data analysis activities. Existing taxonomies are organized by cognitive theories that are hard to associate with visualization tasks. Based on a survey of the literature we propose a task-based taxonomy of 154 cognitive biases organized in 7 main categories. We hope the taxonomy will help visualization researchers relate their design to the corresponding possible biases, and lead to new research that detects and addresses biased judgment and decision making in data visualization.

INTRODUCTION

V ISUALIZATION designers must consider three kinds of limitations: those of computers, of displays, and of humans [START_REF] Munzner | Visualization Analysis and Design[END_REF]. For humans, the designer must consider the limitations of human vision along with the limitations of human reasoning. We focus on the latter, highlighting pitfalls of human judgment and decision making.

Our judgments and decisions routinely rely on approximations, heuristics, and rules of thumb, even when we are not consciously aware of these strategies. The imperfections of these strategies manifest themselves as cognitive biases [2]. While visualization tools are meant to support judgments and decisions, little is known about how cognitive biases affect how people use these tools. To understand how visualizations can support judgment and decision making, we first need to understand how the limitations of human reasoning can affect visual data analysis.

Within the information visualization community, there has been a growing interest in decision making [3], [4] and cognitive biases [5], [START_REF] Dimara | The attraction effect in information visualization[END_REF], [START_REF] Cho | The anchoring effect in decision-making with visual analytics[END_REF]. The IEEE VIS conference has held two workshops [START_REF]Decisive 2017 dealing with cognitive biases in visualisations : a vis 2017 workshop[END_REF] on cognitive biases in information visualization. Other papers have acknowledged the importance of studying cognitive biases in visual data analysis [5], [9], [10]. Despite this growing interest, empirical work remains limited. Most evaluations of visualizations do not include decision tasks [4], and those that do typically assume that human decision making is a rational process. Furthermore, studies that either confirm or disprove the existence of a particular cognitive bias in information visualization are rare [START_REF] Dimara | The attraction effect in information visualization[END_REF]. Meanwhile, most experimental tasks studied in the cognitive bias research use textual representations, and the information given consists of very small datasets -or no data at all. Therefore, the interplay between cognitive biases and visual data analysis remains largely unexplored.

We aim to help bridge the gap between cognitive psychology and visualization research by providing a broad review of cognitive biases, targeted to information visualization researchers. We define a taxonomy of cognitive biases classified by user task, instead of by proposals for psychological explanations of why biases occur. The goal of the paper is to lay out the problem space, facilitate hypothesis generation, and guide future studies that will ultimately help visualization designers anticipate -and possibly alleviate -limitations in human judgment.

The paper is organized as follow: Section 2 provides background information with definitions and related work. Section 3 describes the process we followed to generate the taxonomy. Section 4 reviews each category of bias, discuss related work in visualization, and highlights potential research opportunities. Finally, limitations are discussed before the conclusion section.

BACKGROUND

We first describe what cognitive biases are and why they are challenging to study. We then review existing taxonomies of cognitive biases and their limitations. More detailed descriptions and references for individual biases are included in the next section where we describe our taxonomy.

What is a cognitive bias?

Normative models of judgment and decision making assume that people follow axioms of rationality, have a fixed set of preferences, and make decisions that maximize their benefit. Expected utility theory, introduced by John Von Neumann and Oskar Morgenstern in 1947, is one classic example [START_REF] Neumann | Theory of games and economic behavior[END_REF]. This theory laid down a set of normative rules, which were later extended [START_REF] Leonard | The foundations of statistics[END_REF], [13], [START_REF] Karmarkar | Subjectively weighted utility: A descriptive extension of the expected utility model[END_REF], [START_REF] Payne | Alternative approaches to decision making under risk[END_REF], [START_REF] Coombs | Portfolio theory and the measurement of risk[END_REF]. Any decision that violated these rules was considered irrational.

In contrast, evidence for systematic violations of such rules were identified by Kahneman and Tversky [2], [START_REF] Tversky | Judgment under uncertainty: Heuristics and biases[END_REF], and named "cognitive biases". For example, in an experiment where people were asked to choose a program to combat an unusual disease, the program framed as having a "33% chance of saving a life" was preferred over the program described as having a "66% chance of death"despite the two programs being the same [START_REF]The framing of decisions and the psychology of choice[END_REF]. This bias was named the framing effect. Another well-known example is the confirmation bias, according to which people seek and favor information that confirms their beliefs [START_REF] Mahoney | Publication prejudices: An experimental study of confirmatory bias in the peer review system[END_REF].

Pohl [START_REF] Pohl | Cognitive illusions: Intriguing Phenomena in thinking, judgement and memory[END_REF] defines a cognitive bias as a cognitive phenomenon which:

1) reliably deviates from reality, 2) occurs systematically, 3) occurs involuntarily, 4) is difficult or impossible to avoid, and 5) appears rather distinct from the normal course of information processing.

Thus, a cognitive bias is a cognitive phenomenon which involves a deviation from reality that is predictable and relatively consistent across people. A person who is subject to a cognitive bias is unaware of it, and believes that their decision, judgment, or memory is unbiased. Biases often persist even when people are informed and trained on how to overcome them [21], [START_REF] Fischhoff | Debiasing/kahneman, d., slovic, p. and tversky, a[END_REF]. The last criterion from Pohl further contrasts cognitive biases with more mundane forms of human error such as misunderstanding or misremembering: cognitive biases differ from regular thought processes and as such, they "stick out" and "pique our curiosity" [START_REF] Pohl | Cognitive illusions: Intriguing Phenomena in thinking, judgement and memory[END_REF].

Difficulties with the cognitive bias concept

A major difficulty with the concept of cognitive bias lies in deciding what constitutes a deviation from "reality". This stands in contrast with the study of perceptual biases. In a visual illusion, reality is physically defined, and one can show that perception objectively diverges from that reality. However, for cognitive biases, reality is often difficult to operationalize. Reality is typically defined based on normative models of judgment and of decision making, but such models are not universally accepted and new normative models can emerge in the future, thereby changing what constitutes "reality". Furthermore, the quality of a decision or a judgment is difficult to assess without full information about the cost of 'incorrect' decisions, compared to the costs of following normative principles [23]. Thus, the concept of cognitive bias has fueled long controversies within the field of decision making. While some researchers maintain that cognitive biases are real and have important implications [START_REF] Kahneman | On the reality of cognitive illusions[END_REF], others argue that cognitive heuristics that yield errors in experimental settings can actually be effective strategies for solving complex problems in the real world [START_REF] Gigerenzer | Homo heuristicus: Why biased minds make better inferences[END_REF].

The answer likely lies in-between: in many cases, heuristics and rules of thumb can simplify complex problems and yield effective decisions, given that humans have limited time and cognitive resources. Yet we know that heuristics do not consistently lead to optimal decisions [2]. Some heuristics can routinely lead to decisions that deviate from optimality in a systematic and predictable fashion, possibly with real-life consequences. Thus, despite the difficulties in defining and studying them, cognitive biases are important and visualization researchers need to be aware of them.

Taxonomies of cognitive biases

We review existing taxonomies from different domains (psychology, decision systems, intelligence analysis, visualization). We found that the majority of existing taxonomies are explanatory: they organize biases according to why they occur, by considering cognitive mechanisms and explanatory theories. In contrast, our taxonomy is task-based. It organizes biases based on the experimental tasks they have been observed in, in order to help visualization researchers identify biases that may affect visualization tasks.

Explanatory taxonomies

Tversky and Kahneman classify biases according to which strategy (heuristic) is hypothesized that people follow to make a decision or judgment [START_REF] Tversky | Judgment under uncertainty: Heuristics and biases[END_REF]. For example, some biases are classified as outcomes of the "representative heuristic" where people estimate probabilities by the degree to which one event is similar to another event. Imagine we are given a salient description of an imaginary person named Linda with adjectives such as "bright", "outspoken", "deeply concerned with discrimination issues and social justice". If asked to chose the most likely alternative between "Linda is a bank teller" or "Linda is a bank teller and is active in the feminist movement", people tend to choose the second even though the conjunction of the two events cannot be more likely (in terms of probabilities) than either event alone [23]. Another class of biases includes the ones that are considered as outcomes of the "availability heuristic", in which people estimate an event as frequent or imaginable if they can recall it more easily in their minds, neglecting to apply a rational probability rule [START_REF] Tversky | Judgment under uncertainty: Heuristics and biases[END_REF]. For example, hearing news about a plane crash may temporarily alter people's feelings on flight safety [START_REF] Dimara | Accounting for availability biases in information visualization[END_REF]. Similarly, Baron [27] classifies 53 cognitive biases based on both the normative models they violate (e.g., Bayes theorem, regression to the mean) and their explanations (e.g., availability). This strategy-based classification raised several criticisms by Gigerenzer [START_REF] Pohl | Cognitive illusions: Intriguing Phenomena in thinking, judgement and memory[END_REF], who considers these strategies to be conceptually vague, imprecise and difficult to falsify, while other scientists give alternative explanations for why most of these biases occur [START_REF] Pohl | Cognitive illusions: Intriguing Phenomena in thinking, judgement and memory[END_REF]. In contrast, our taxonomy groups both these biases under the class "Estimation", to indicate they occurred in experimental tasks where decision makers attempt to estimate probability outcomes.

Other taxonomies from the psychology literature (e.g., [START_REF] Evans | Hypothetical thinking: Dual processes in reasoning and judgement[END_REF], [START_REF] Stanovich | Rationality and the Reflective Mind[END_REF]) also consider possible cognitive mechanisms that may lead to bias. One common approach is to consider the dual-process model of human reasoning [START_REF] Gilovich | Heuristics and Biases: The Psychology of Intuitive Judgment[END_REF], assigning biases either to system 1 of the model (heuristic/intuitive reasoning) or to system 2 (analytic/reflective reasoning). Padilla et al. [START_REF] Padilla | Decision making with visualizations: a cognitive framework across disciplines[END_REF] expanded the dual-process model for visualizations, emphasizing that the system 1 may result in faulty decision making, if salient aspects of the encodings focus on non-critical information. While these groupings explain when biases can occur in the reasoning process, in contrast to our taxonomy they do not explicitly group biases based on the tasks involved. Moreover, recent work indicates that the relation between the two systems in the dual-process model, and the heuristics (strategies) discussed by Tversky and Kahneman [START_REF] Tversky | Judgment under uncertainty: Heuristics and biases[END_REF], is not so clear-cut [START_REF] Evans | Questions and challenges for the new psychology of reasoning[END_REF], [START_REF] Kahneman | Representativeness revisited: Attribute substitution in intuitive judgment[END_REF].

These taxonomies that focus on the mechanisms behind the origin of biases (e.g., heuristics) are often not exhaustive when it comes to including all biases, but rather give examples of biases originating from specific reasoning processes.

Taxonomies considering tasks

Classifications developed in the domain of decision-support information systems also tend to be mostly explanatory, but some consider high-level tasks when grouping biases. In 1986, Remus and Kottemann [34] divided about 20 biases into two categories, data presentation and information processing (a high-level task of when biases occur) and later subdivided these categories based on the reasons why these biases occur (e.g., use of a certain heuristic, not understanding statistics, etc.). Similarly, Arnott in 2006 [START_REF] Arnott | Cognitive biases and decision support systems development: a design science approach[END_REF] considered the nature of the cognitive bias and classified 37 cognitive biases into categories, examples of which are: situation, for biases related to how a person responds to the general decision situation, or confidence, for biases that are believed to occur in order to increase the confidence of a person. Arnott [START_REF] Arnott | Cognitive biases and decision support systems development: a design science approach[END_REF] did not group biases by task, but he did map each bias category with components of a decision-support system schema, e.g., data acquisition, processing, or output. These components can be seen as high-level tasks.

Both of these taxonomies associate biases with complex data processing, though these associations are not supported by strong empirical evidence. Most of the biases have been only verified with small puzzles using static textual representations and not in the context of using a decisionsupport computer system dealing with data.

While not proposing a cognitive bias taxonomy per se, Heuer [START_REF] Heuer | Psychology of intelligence analysis[END_REF] discusses biases which are likely to affect highlevel tasks of intelligence analysis, e.g., hindsight bias in the evaluation of reports.

More recently in psychology, Pohl [START_REF] Pohl | Cognitive illusions: Intriguing Phenomena in thinking, judgement and memory[END_REF] classified cognitive biases into "memory", "thinking", and "judgment" biases. The memory class involves systematic errors in recalling or recognizing events [START_REF] Pohl | Cognitive illusions: Intriguing Phenomena in thinking, judgement and memory[END_REF]. The thinking class involves systematic errors in applying a certain rule (e.g., Bayes' theorem, hypothesis testing, syllogistic reasoning) [START_REF] Pohl | Cognitive illusions: Intriguing Phenomena in thinking, judgement and memory[END_REF]. These rules come from several norms, e.g., probability theory, expected utility, or the falsification principle, which determine the actions that deviate from "reality". The judgment class involves systematic errors when subjectively rating a stimulus (e.g., pleasantness, frequency, or veracity) [START_REF] Pohl | Cognitive illusions: Intriguing Phenomena in thinking, judgement and memory[END_REF]. In judgment biases, people can be affected by feelings of familiarity or confidence. As Pohl [START_REF] Pohl | Cognitive illusions: Intriguing Phenomena in thinking, judgement and memory[END_REF] himself mentions, this taxonomy has several limitations. Most biases in judgment and thinking also involve memory processes such as encoding, storage, and retrieval [START_REF] Pohl | Cognitive illusions: Intriguing Phenomena in thinking, judgement and memory[END_REF]. Also, when the material to memorize is outside of the laboratory, memory and subjective judgment biases cannot be distinguished because a faulty recall can be the reason for a faulty judgment (or not) [START_REF] Pohl | Cognitive illusions: Intriguing Phenomena in thinking, judgement and memory[END_REF]. Judgment and thinking classes also often overlap, e.g., people may not know that they are supposed to apply a Bayesian rule to estimate a probability and instead perform a subjective judgment of frequency. Our taxonomy goes beyond this grouping, considering a larger number of tasks to provide a more detailed classification of when biases occur.

In summary, while high-level tasks have been used to name a few categories in existing taxonomies, not all biases were grouped by task. Our taxonomy provides a grouping based on the lower level tasks where these biases have been observed and measured.

Reviews of biases in visualization

We found no comprehensive review of cognitive biases in the visualization literature. Some studies have begun to organize biases relevant to specific aspects of visual analysis. Zuk and Carpendale [9] categorize biases and heuristics relevant to uncertainty, discussing how improved design could mitigate them. For example, when making decisions individuals may be unable to retrieve from memory all relevant instances to a problem (availability bias) and rely on recent information (recency effect), but a visualization can display all instances [9]. Nevertheless, the focus is not on grouping all known biases based on the tasks where they have been observed, but rather on a subset of biases related to uncertainty in reasoning, grouped by how we could mitigate them using visual analysis. Ellis and Dix [10] briefly discuss seven biases that could affect visual analysis (such as anchoring and availability biases) and emphasize the lack of studies investigating whether or not visualizations elicit cognitive biases.

Need for a new taxonomy

The majority of existing bias classifications are explanatory (Sec. 2.3.1) based on generic explanations of their nature, such as why the bias occurs or which heuristic people use when it appears. This evolving body of work indicates that there is still no agreement among scientists about the cause of cognitive biases, as observed by Pohl [START_REF] Pohl | Cognitive illusions: Intriguing Phenomena in thinking, judgement and memory[END_REF]. Moreover, in some cases the association prepossessed between biases and complex data processing is not always strongly supported [34], [START_REF] Arnott | Cognitive biases and decision support systems development: a design science approach[END_REF], or is fairly high-level [START_REF] Pohl | Cognitive illusions: Intriguing Phenomena in thinking, judgement and memory[END_REF]. While understanding the nature and provenance of biases is important, we take a more practical approach. Based on papers that have experimentally studied the biases, we classify them based on the tasks they have observed and measured in.

Finally, most classifications include only a small subset of biases, usually 10-40, whereas a collaboratively-edited Wikipedia page lists 176 known cognitive biases [START_REF] Wikipedia | List of cognitive biases -wikipedia, the free encyclopedia[END_REF] as of today. Some classifications only gave examples [34], others listed a small sample of these biases [START_REF] Arnott | Cognitive biases and decision support systems development: a design science approach[END_REF]. One reason behind this limitation may be that not all biases are as relevant to all scientific domains. For example, the in-group favoritism bias, where people tend to make judgments that favor their own group, is more important in social psychology. The attraction effect, where people's choices are affected by inferior alternatives, is primarily studied in marketing research. Therefore, most taxonomies tend to account for the biases that are established in the respective domain of the authors. Another explanation, especially for explanatory taxonomies, could be that the objective of the taxonomy is to offer an abstract, unified theory that explains multiple biases (giving a few examples), rather than to organize and extensively review all biases covered in previous works. Data visualization can be used in a variety of domains, so researchers need to be aware of a larger set of biases.

METHODOLOGY

After a standard bibliographic search we gathered an initial list of biases. The second step was to search for the most representative paper that empirically tested each of the biases in our list. Third, we categorized the cognitive biases, using a bottom-up grouping method similar to card sorting. The categories and their labels were refined iteratively in an attempt to make them more useful to visualization researchers. We categorised all biases and reviewed each one from a visualization perspective by: 1) searching for existing relevant visualization work, if any (reported in Table 2); and 2) brainstorming future opportunities for visualization research (reported in their respective category description).

The new cognitive bias taxonomy proposed in this paper is organized by the tasks users are performing when the bias is observed. For example, estimating the likelihood of a heart attack or breast cancer is considered an estimation task. Choosing between different health insurance policies is a decision task. The causes for the bias may have been studied (e.g., false probability estimations may lead to a bad insurance choice), but this is ignored in our classification of the biases. Our taxonomy focuses on the tasks where biases occur, instead of why they occur -as previously done.

Initial list of biases

We decided to start with the list of cognitive biases (and their synonym names) we had found on the Wikipedia page "List of Cognitive Biases" [START_REF]List of cognitive biases -wikipedia, the free encyclopedia[END_REF] -retrieved on 20 November 2017. With a total of 176 cognitive biases it was by far the longest list of biases we could find. We refer to this page as the Wikipedia list page. Each entry of this list points to a separate individual Wikipedia page describing the bias. Later on, a few missing biases were added (see below). Although Wikipedia was the largest list of biases we found, it is not curated by researchers. Therefore, the next section describes our method to verify which of these cognitive biases are detected through reliable experimental protocols.

Selection of sources

Because of the large number of biases, we kept only one representative paper per bias in order to keep the number of references manageable. For each of the 176 biases, we used the following process:

Step 1: We searched whether the bias has been mentioned in InfoVis literature by typing the search term "bias name" + "information visualization" in Google Scholar. We collected all InfoVis papers mentioning the bias (See Table 2, column "Relevance to InfoVis"). In the visualization papers mentioning a bias, we collected the source reference used to describe the bias and determined if it was an eligible source (see below). We kept only one source paper.

Step 2: When we could not find an InfoVis paper mentioning the bias, or if these papers did not cite an eligible source, we searched for eligible sources in the Wikipedia list page, or in the individual Wikipedia page. If we did not find an eligible source in Wikipedia, we searched for a source by typing the search term "bias name" + "experiment" in Google Scholar. We only considered the first page of results and examined the papers by decreasing order of citations. We picked the first eligible paper. If no eligible paper was found, we repeated the process using a synonym for the cognitive bias. Synonyms had been collected on the Wikipedia pages and in academic sources (see Table 3).

Step 3: When no source could be found at all, the bias was removed from the list. This occurred for 21 of the biases on the Wikipedia list.

Source eligibility:

A source was considered eligible if:

1) It was a peer-reviewed paper.

2) We were able to access the document.

3) AND the paper either: a) reported on a human subject study testing for the existence of the bias (we did fulltext searches for the terms "experiment" and "study"), or b) cited another paper reporting on such study, and described the paper's experimental task in detail.

Method 3.b was used when the original paper was too old for the document to be accessible, or when a peerreviewed survey existed that described experimental tasks in enough detail. In general, we favored literature-reviews as references when they provided a good overview of the different studies conducted on a particular cognitive bias. We applied the accessibility rule (2) only to help us select one source over another -no bias was eliminated because of this rule. The reliability of the experiment (e.g., experiment design, validity of statistical methods, effect size, etc.) was not examined.

Final list of biases

At the end of the source selection process we were left with 151 cognitive biases out of the 176 in the initial list. The Wikipedia list contained 13 duplicates biases, that either pointed to the same individual Wikipedia pages, or had different individual Wikipedia page but with the same referenced work. We also added 3 additional cognitive biases we had found in literature: the ballot names bias (a bias identified in InfoVis [START_REF] Wood | Ballotmaps: Detecting name bias in alphabetically ordered ballot papers[END_REF] but which was not given a specific name); the phantom effect [40]; and the compromise effect [START_REF] Simonson | Choice based on reasons: The case of attraction and compromise effects[END_REF] -which were mentioned in the attraction effect literature but not listed in the Wikipedia page.

Two biases on the list were included even though they blur the line between a cognitive bias and a perceptual illusion. The Weber-Fechner law occurs when differences between quantities seem smaller (as absolute values) when the baseline quantity increases [START_REF] Choo | Enumeration of small collections violates webers law[END_REF]. We retained this bias because it has cognitive analogues -a fifty dollar upgrade offer does not seem expensive when buying a 20,000 dollar car, but seems large when buying a 300 dollar phone. We also kept Pareidolia, the propensity to see faces where none exist (within clouds, or on toasted bread). It is a perceptual bias, but analogous to cognitive biases such as the confirmation bias, as both show the existence of a heavily weighted prior probability toward a particular state of the world.

To non experts, some of the biases in the table may appear similar to each other, but we kept them separated in cases where bias researchers considered them as different. In addition, if two biases are known under different names and reported in different research papers, we kept both biases, even if they appeared to be similar.

Establishing categories

To produce a task-based taxonomy we first had to identify the type of task used in the study of the bias. We then used open card-sorting analysis to generate categories [START_REF] Wood | Card sorting: current practices and beyond[END_REF].

Task Identification: We went back to the original experiment protocol described in the representative paper and identified the task participants had performed when the bias was measured. Our assumption was that tasks should have many similarities across biases and that the number of tasks would be of manageable size. After identifying the experiment task used in the study, we proposed a short label to describe the task. When appropriate we reused previously assigned labels if they adequately described the task. If not, we proposed a new short label for the task. One coder performed the initial labeling of all tasks, then another two coders reviewed and proposed revisions to the labels until all three coders were satisfied.

Task Grouping: Similar tasks/labels were then grouped in seven categories, to form the tasks in our taxonomy. An initial grouping and task group names were proposed by a single person, and iteratively revised by four others (the coauthors). Tasks that could not be assigned to large groupings were placed under the category "Other".

The task-categories are (with the color used in Figure 1):

1) ESTIMATION 2) DECISION 3) HYPOTHESIS ASSESSMENT 4) CAUSAL ATTRIBUTION 5) RECALL 6) OPINION REPORTING 7) OTHER
These categories were created by the open card-sorting method described above, relying on only the papers from the bias literature. To avoid constraining these categories to the context of data visualization, we purposely did not base these categories on the numerous task taxonomies (e.g., [START_REF] Brehmer | A multi-level typology of abstract visualization tasks[END_REF], [START_REF] Amar | Knowledge precepts for design and evaluation of information visualizations[END_REF]) that have been proposed in the data visualization literature. We contrast our resulting taxonomy with these alternatives within the 'Visualization Research' subsection for each bias, and more globally in Section 5.2.

Because each category includes a fairly large number of biases, we added subcategories. Since there was not a clear set of subtasks to use for these set of subcategories, we instead chose a set of sub-categories (which we call flavors) that reflect other types of similarities among biases. We do not view these flavors as a primary contribution of this work, because they were developed in an intuitive way, instead of a more rigorous division into tasks, which can be traced directly to a user study protocol. We hope that the flavors will help readers see connections between the biases, both within and between categories.

The flavors we identified are:

1) Association, where cognition is biased by associative connections between information items 2) Baseline, where cognition is biased by a comparison with (what is perceived as) a baseline 3) Inertia, where cognition is biased by the prospect of changing the current state 4) Outcome, where cognition is biased by how well something fits an expected or desired outcome 5) Self perspective, where cognition is biased by a self-oriented view point.

Figure 2 illustrates how the task categories are distributed among flavors. In the next section we will describe the taxonomy table, and then discuss each category in detail.

TASK-BASED TAXONOMY OF COGNITIVE BIASES

The complete taxonomy is summarized in Table 2. The first column shows the task category color of the bias. The column Flavor tries to capture the general phenomenon behind the bias. The column Cognitive bias shows the name of each bias (synonym names for some biases can be found in Table 3). The column Ref shows the selected representative paper in which the bias was experimentally detected. The column Relevance to InfoVis shows how the bias has been studied in visualization research. The last column provides a very short description of the bias. In order to reveal the scarcity of research about cognitive bias in visualization we color-coded the InfoVis column: various shades of red indicate that the bias has been empirically studied. Black indicates that the bias has been discussed in a visualization paper but not yet studied. Shades of gray represent our estimate on how relevant the bias may be for visualization research (dark gray for biases more likely to be important, light gray for those less likely). This is a subjective rating only meant to help the reader get started when using the table.

We will now review each category by providing examples of tasks, describing a subset of the biases using examples from psychology research, discuss related work in visualization, and highlight potential research opportunities for visualization research.

Biases in estimation tasks

In estimation tasks, people are asked to assess the value of a quantity. For example, in real-life decision making tasks, a person may need to estimate the likelihood of theft to decide whether or not to insure their car, or to estimate their future retirement needs in order to choose a retirement plan.

The ESTIMATION category includes all systematic biases that have been experimentally observed when participants were asked to make an estimation. We identified 33 estimation biases, listed in Table 2.

Psychology research

In cognitive bias research, many estimation tasks require assessing the likelihood that an event will occur in a hypothetical situation or in the future (that is, prediction tasks). Thus, much of our discussion in this section focuses on probability estimation tasks.

Several psychology experiments involve a probability estimation task where the correct answer can be derived by calculation (e.g., by applying Bayes' theorem). Systematic deviations from the true answer are taken to be suggestive of a cognitive bias. For example, research on the base rate fallacy suggests that people can grossly overestimate the likelihood of an event (e.g., having breast cancer after a positive mammography) because they tend to focus on the specific event instance while ignoring probabilities that apply to the general population (e.g., the number of women with breast cancer) [START_REF] Barbey | Base-rate respect: From ecological rationality to dual processes[END_REF]. Probability estimation tasks with a well-defined ground truth have helped uncover other cognitive biases such as the conjunction fallacy [START_REF] Tversky | Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment[END_REF], where people believe that specific events are more probable than general ones. Moreover, according to studies on the conservatism bias, people typically do not sufficiently revise their probability estimations in the light of new information [START_REF] Phillips | Conservatism in a simple probability inference task[END_REF], [START_REF] Johnson | Sources of the continued influence effect: When misinformation in memory affects later inferences[END_REF], [START_REF] Furnham | A literature review of the anchoring effect[END_REF].

Some experiments involve probability estimation tasks without ground truth. Here, responses are not evaluated based on how much they agree with a true answer, but on how consistent they are with basic normative principles of rationality. For example, according to experiments on the optimism bias, when people are asked to make predictions about future events (e.g., finding a dream job, getting divorced, or getting lung cancer), they tend to make more optimistic predictions for themselves than for others [START_REF] Weinstein | Unrealistic optimism about future life events[END_REF].

A number of experiments use estimation tasks that do not involve explicit probabilities, but have a probabilistic component. Frequency estimation is such an example. As an example of bias, people tend to think that words starting with the letter "R" are more frequent than words having the letter "R" in third position [START_REF] Tversky | Availability: A heuristic for judging frequency and probability[END_REF]. This is thought to occur because people employ the availability heuristic, whereby words starting with "R" are easier to retrieve from memory and, therefore, are perceived to be more frequent [START_REF] Tversky | Availability: A heuristic for judging frequency and probability[END_REF]. Time prediction is another example of a "quasi-probabilistic" estimation task, since estimating the time or duration of a future event is related to estimating the probability that the event will fall before or after a certain moment in time. Several studies have been conducted where participants were asked to predict the time it will take them to complete a task (e.g., an academic project or an assignment), and where predictions were compared with actual outcomes [START_REF] Buehler | The planning fallacy: cognitive, motivational, and social origins[END_REF]. These studies consistently show that people tend to be overly optimistic in their predictions irrespective of their past experience, a bias called the planning fallacy [START_REF] Buehler | The planning fallacy: cognitive, motivational, and social origins[END_REF].

Finally, some experiments involve clearly nonprobabilistic estimation tasks, such as estimating country populations. In most studies, responses are again not evaluated based on a ground truth, but based on how consistent they are with basic principles of rationality. For example, Tversky and Kahneman asked participants to spin a fortune wheel, and then to estimate the number of African countries in the UN [START_REF] Tversky | Judgment under uncertainty: Heuristics and biases[END_REF]. People's responses tended to be close to the number the fortune wheel landed on [START_REF] Tversky | Judgment under uncertainty: Heuristics and biases[END_REF]. Since this number bears no relationship with the question, its influence on the answers is strongly suggestive of irrationality. That tendency for people's quantitative estimations to be biased toward a value they were initially exposed to is named the anchoring effect [START_REF] Tversky | Judgment under uncertainty: Heuristics and biases[END_REF]. On the other hand, Goldstein and Gigerenzer [START_REF] Gigerenzer | Reasoning the fast and frugal way: models of bounded rationality[END_REF] showed that when students are asked to compare cities by their population, they perform better with cities that are not from their home country thanks to their use of the recognition heuristic (i.e., if I never heard of a city, then it must be small) [START_REF] Gigerenzer | Reasoning the fast and frugal way: models of bounded rationality[END_REF]. This experiment should serve as a warning that heuristics do not necessarily lead to cognitive biases, and can sometimes even yield more accurate judgments.

Other examples of non-probabilistic estimation tasks are experiments in which participants have to estimate their performance after solving a given problem. Most often participants exhibit overconfidence, i.e., their self-rating is higher than their accuracy [START_REF] Klayman | Overconfidence: It depends on how, what, and whom you ask[END_REF]. In a smaller number of biases, people exhibit low confidence. Confidence can change according to the difficulty of the task (overconfidence for hard tasks, conservatism for easy ones [START_REF] Lichtenstein | Do those who know more also know more about how much they know?[END_REF]), or the expertise of the participant (overconfidence in non-specialists, conservatism in experts [START_REF] Kruger | Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments[END_REF]).

Visualization research

Although estimation tasks are not explicitly listed in visualization task taxonomies, they are omnipresent in visual data analysis. An analytic task may involve estimation whenever an exact answer is not possible or is not required.

In information visualization, there has been very little research on estimation biases that occur at the cognitive level (as opposed to the perceptual level), with the notable exception of research on Bayesian reasoning [START_REF] Ottley | Improving bayesian reasoning: the effects of phrasing, visualization, and spatial ability[END_REF], [START_REF] Micallef | Assessing the effect of visualizations on bayesian reasoning through crowdsourcing[END_REF], [START_REF] Khan | Benefits of visualization in the mammography problem[END_REF], [START_REF] Khan | Interactive instruction in bayesian inference[END_REF]. Researchers have studied whether visualizations such as Euler diagrams and frequency grids can reduce the base rate fallacy [START_REF] Micallef | Assessing the effect of visualizations on bayesian reasoning through crowdsourcing[END_REF], [START_REF] Khan | Benefits of visualization in the mammography problem[END_REF]. Even though the studies did not observe a systematic bias, people were often highly inaccurate in their responses, and visualizations did not seem to provide clear benefits [START_REF] Micallef | Assessing the effect of visualizations on bayesian reasoning through crowdsourcing[END_REF], [START_REF] Khan | Benefits of visualization in the mammography problem[END_REF]. Micaleff et al. [START_REF] Micallef | Assessing the effect of visualizations on bayesian reasoning through crowdsourcing[END_REF] conjectured that many participants may have ignored the visualizations and attempted calculations using the numbers provided in the textual narrative (also shown in the visualization condition). Their last experiment indeed suggests that visualizations can have a facilitating effect if numerals are not provided in the text, leading the authors to conclude that presentation formats that encourage approximate estimation and discourage precise calculation need to be further investigated [START_REF] Micallef | Assessing the effect of visualizations on bayesian reasoning through crowdsourcing[END_REF].

Several other information visualization studies have examined biases in probability and frequency estimation tasks (e.g., [START_REF] Correll | Error bars considered harmful: Exploring alternate encodings for mean and error[END_REF], [START_REF] Correll | Surprise! bayesian weighting for debiasing thematic maps[END_REF]), but the biases that were investigated were perceptual rather than cognitive.

Two recent studies have examined the impact of the anchoring effect in an information visualization context. Valdez et al. [START_REF] Valdez | Priming and anchoring effects in visualizations[END_REF] gave participants a series of class separability tasks on scatterplots, and found that their responses were influenced by the first scatterplot shown. Similarly, Cho et al. [START_REF] Cho | The anchoring effect in decision-making with visual analytics[END_REF] asked participants to explore a real Twitter dataset using a visual analytic system, and found that their responses to a quantitative estimation question were influenced by the presence of an anchor. While Cho et al. only found an effect when the anchor was presented as a numeral, user log analyses suggested that visualization anchors can affect participants' analytic process.

Xiong et al. [START_REF] Xiong | The curse of knowledge in visual data communication[END_REF] provided preliminary but compelling evidence for the existence of a curse of knowledge bias in visualization communication, using an estimation task. The curse of knowledge refers to people's tendency to overestimate how much they share their knowledge or expertise with other people [START_REF] Keysar | Speakers' overestimation of their effectiveness[END_REF] Xiong et al. [START_REF] Xiong | The curse of knowledge in visual data communication[END_REF] showed that participants who are exposed to a text narrative before seeing a visualization find the patterns related to the text narrative more salient. Crucially, participants tended to predict that the same patterns would be salient to viewers who were not exposed to the textual narrative.

In addition to empirical work, the information visualization literature has produced position papers that discuss how visualization designs might alleviate (rather than cause) estimation biases. For example, Dragicevic and Jansen [START_REF] Dragicevic | Visualization-mediated alleviation of the planning fallacy[END_REF] list four possible strategies to alleviate the planning fallacy using visualizations and data management tools, while Dimara et al. [START_REF] Dimara | Accounting for availability biases in information visualization[END_REF] suggest three ways visualization could be used to alleviate the availability bias. However, no tool has been developed and no experiment has been conducted to evaluate the effectiveness of these strategies.

Although self-reported confidence is a common metric in information visualization evaluation [START_REF] Du | Finding similar people to guide life choices: Challenge, design, and evaluation[END_REF], it can be subject to biases [START_REF] Lichtenstein | Do those who know more also know more about how much they know?[END_REF], [START_REF] Kruger | Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments[END_REF]. Findings from cognitive bias research suggest that confidence metrics need to be calibrated [START_REF] O'hagan | 4.3: The calibration of subjective probabilities: theories and explanations[END_REF] and put in context with task accuracy. Even if confidence judgments can be compatible with normative statistical principles [START_REF] Sanders | Signatures of a Statistical Computation in the Human Sense of Confidence[END_REF], they can be easily influenced by context. Previous visualization research suggests that indirect confidence assessment (e.g., "How likely are you to change your choice if a recommendation system gives you another suggestion?") may be more reliable than direct confidence ratings (e.g., "How confident are you in your choice?") [4]. However, to our knowledge, no previous visualization study examined biases related to performance estimation.

There is too little empirical data available at this point to provide strong guidelines for practitioners. One possible recommendation is that visualizations should be designed to minimize the number of estimations needed to derive answers to questions. One way of achieving this is by calculating and visually presenting relevant summary values to the user. However, it is typically impossible for the designer to anticipate all questions a user may have about the data. When users are likely to derive answers to unanticipated questions by combining several pieces of information, preliminary work on Bayesian estimation problems [START_REF] Micallef | Assessing the effect of visualizations on bayesian reasoning through crowdsourcing[END_REF] suggests that showing numbers next to (or on top of) visualizations can be counterproductive. The reason is that numbers prompt users to calculate, and miscalculations can yield errors much greater than imperfect approximations. Unless precise values are needed, it is advised to encode all quantitative values visually.

Biases in decision tasks

By decision task, we refer to any task involving the selection of one over several alternative options. Psychology experiments using such tasks are called choice studies. Study participants in these studies are "required to exhibit a preference for one of the several stimuli or make a different prescribed response to each of them" [START_REF] Authors | Psychology Today: An Introduction[END_REF]. For example, people can choose a car to purchase or a university to apply to.

The DECISION category includes all systematic biases that have been experimentally observed when participants are asked to make a decision. We identified 33 decision biases, listed in Table 2.

Psychology research

Some decision biases occur when people are dealing with uncertainty. For example, in ambiguity effect people tend to avoid decisions associated with ambiguous outcomes [START_REF] Ritov | Reluctance to vaccinate: Omission bias and ambiguity[END_REF]; or in the zero-risk bias, if the set of choices contains an alternative that eliminates risk completely they tend to stick to it even if it is not the optimal decision [START_REF] Baron | Attitudes toward managing hazardous waste: What should be cleaned up and who should pay for it?[END_REF]. People also often show different preferences based on whether the problem is a gain (e.g., allowances) or a loss (e.g., prohibitions) [START_REF] Thaler | The effect of myopia and loss aversion on risk taking: An experimental test[END_REF], known as loss aversion, or if it is simply framed as a gain or a loss, known as framing effect [START_REF]The framing of decisions and the psychology of choice[END_REF].

Nevertheless, not all decision biases are related to uncertain outcomes or framing. When people choose one alternative over the other, they are often unconsciously influenced by factors irrelevant to the decision to be made. In most situations, decision makers do not evaluate alternatives in isolation, but within the context in which the alternatives occur [23]. One well-studied example is the attraction effect, where one's decision between two alternatives is influenced by the presence of irrelevant (inferior) alternatives [START_REF] Huber | Adding asymmetrically dominated alternatives: Violations of regularity and the similarity hypothesis[END_REF].

In some biases such as the less is better effect people's decisions are affected by whether the alternatives are presented separately or juxtaposed [76], or by whether the alternatives are presented among more extreme ones (compromise effect) [START_REF] Simonson | Choice based on reasons: The case of attraction and compromise effects[END_REF], unavailable ones (phantom effect) [40], or more familiar alternatives (mere-exposure effect) [START_REF] Zajonc | Mere exposure: A gateway to the subliminal[END_REF].

Other cognitive biases refer to people who appear more attracted to alternatives for which they can receive an immediate reward such as the hyperbolic discounting [78], or for which they had previously invested self-effort, such as the IKEA effect [START_REF] Norton | The ikea effect: When labor leads to love[END_REF]. Examples also include attraction to alternatives which people owned in the past [START_REF] Morewedge | Explanations of the endowment effect: an integrative review[END_REF] (endowment effect), or avoiding to make any decision that requires a change of one's current state (status quo bias) [START_REF] Samuelson | Status quo bias in decision making[END_REF].

Visualization research

Decision tasks are common when using visualizations. Dimara et al. defined a decision task which articulates the link between decision making and multidimensional data visualizations named multi-attribute choice task. Several visualization systems exist that are explicitly designed to support multi-attribute choice tasks [START_REF] Gratzl | Lineup: Visual analysis of multi-attribute rankings[END_REF], [START_REF] Carenini | Valuecharts: analyzing linear models expressing preferences and evaluations[END_REF], [START_REF] Pajer | Weightlifter: Visual weight space exploration for multi-criteria decision making[END_REF] or decision making in general [START_REF] Asahi | Using treemaps to visualize the analytic hierarchy process[END_REF], [START_REF] Rudolph | Finvis: Applied visual analytics for personal financial planning[END_REF], [START_REF] Aseniero | Stratos: Using visualization to support decisions in strategic software release planning[END_REF]. Visualization researchers often mention decision biases under uncertainty [9], [10], [START_REF] Rudolph | Finvis: Applied visual analytics for personal financial planning[END_REF], [START_REF] Dasgupta | Bridging theory with practice: An exploratory study of visualization use and design for climate model comparison[END_REF] but there is very limited empirical work studying their existence in visualization [START_REF] Hullman | Hypothetical outcome plots outperform error bars and violin plots for inferences about reliability of variable ordering[END_REF], [START_REF] Fernandes | Uncertainty displays using quantile dotplots or cdfs improve transit decision-making[END_REF].

Exceptions include the attraction effect for which Dimara et al. showed that it also exists in scatterplot visualizations, and confirmed that even if data is correctly visualized and understood, the decision may still be irrational [START_REF] Dimara | The attraction effect in information visualization[END_REF]. Recent work [91] showed that visualizations can mitigate the attraction effect by allowing users to remove information from the display that should not affect a rational decision making process. Zhang et al. [92] further showed that startup companies presented with static tabular visualizations of star ratings tended to be subject to loss aversion bias.

Another example of a decision bias that was studied in a visualization context was the identifiable victim effect, where people are more likely to help a single concretely described person in need, compared to larger numbers of abstractly or statistically described people. [START_REF] Jenni | Explaining the identifiable victim effect[END_REF]. In contrast, Boy et al. [START_REF] Boy | Showing people behind data: Does anthropomorphiz-ing visualizations elicit more empathy for human rights data?[END_REF] found that data graphics that used simiarly concrete anthropomorphized icons did not increase a user's empathy for vulnerable populations. This result shows that it can be hard to anticipate the results of combining cognitive bias findings with visualization designs.

Visualizations can also be used to find evidence for a cognitive bias. An example of such a decision bias was found in government elections. Several scientific studies had long investigated the hypothesis that the order of candidates in the ballot papers can affect the result of the elections, but they only found inconclusive evidence. Wood et al. [START_REF] Wood | Ballotmaps: Detecting name bias in alphabetically ordered ballot papers[END_REF] collected data from 5000 candidates of the Greater London local elections held on the 6th May 2010, analyzed them using hierarchical spatially arranged visualizations, and showed that the position1 of candidate names on the ballot paper indeed influenced the number of votes they received. Wood et al.'s visual analytic techniques showed that an alphabetical tabular representation of candidates can lead to biased election results.

Although decision biases can be critical for visualization systems that target decision-support [START_REF] Padilla | Decision making with visualizations: a cognitive framework across disciplines[END_REF], most of the decision-support visualizations rarely evaluate the quality of users decisions (e.g., wrt to their consistency with personal preferences [4] or with rational principles [START_REF] Dimara | The attraction effect in information visualization[END_REF]).

Biases in hypothesis assessment tasks

By hypothesis assessment task, we refer to any task involving an investigation of whether one or more hypotheses are true or false. The term "hypothesis" here does not necessarily refer to a formal statistical hypothesis, but any statement, informal or formal, that can be either confirmed or disconfirmed using previous or new knowledge.

The HYPOTHESIS ASSESSMENT category includes all systematic biases that have been experimentally observed when participants were asked to assess if a statement is true or false. We identified 11 hypothesis assessment biases, listed in Table 2.

Psychology research

One of the best known and most impactful biases is the confirmation bias, according to which people tend to favor evidence that confirm an initial hypotheses while subconsciously ignoring disconfirming evidence [START_REF] Mahoney | Publication prejudices: An experimental study of confirmatory bias in the peer review system[END_REF]. As Nickerson puts it, "[the bias] appears to be sufficiently strong and pervasive that one is led to wonder whether the bias, by itself, might account for a significant fraction of the disputes, altercations, and misunderstandings that occur among individuals, groups, and nations." [START_REF] Nickerson | Confirmation bias: A ubiquitous phenomenon in many guises[END_REF]. Related biases in this category are the illusory truth effect, according to which people consider a proposition as true after repeated exposure to it [START_REF] Hasher | Frequency and the conference of referential validity[END_REF]; the congruence bias where people test if a hypothesis is true without considering alternative hypotheses [START_REF] Wason | On the failure to eliminate hypotheses in a conceptual task[END_REF]; and the illusory correlation bias when people consider a relationship between variables that does not exist [START_REF] Chapman | Illusory correlation as an obstacle to the use of valid psychodiagnostic signs[END_REF].

Scientists themselves are subject to biases in hypothesis assessment. For example, according to studies on the experimenter effect, experimenters can subconsciously influence participants to behave in a way that confirms their experimental hypotheses [START_REF] Rosnow | People studying people: Artifacts and ethics in behavioral research[END_REF].

Visualization research

Hypothesis assessment tasks are common in data analysis and reasoning using visualization tools, e.g., exploring whether trucks have more accidents than regular cars; if the horsepower of a car is correlated to its weight; or if earth temperatures are increasing. Keim et al. refer to these type of high-level tasks as confirmatory analysis [START_REF] Keim | Challenges in visual data analysis[END_REF] and Amar and Stasko [START_REF] Amar | A Knowledge Task-Based Framework for Design and Evaluation of Information Visualizations[END_REF] characterize them as confirm hypotheses tasks in their taxonomy.

Although hypothesis assessment biases have been mentioned as critical challenges for information visualization [9], [10], we are not aware of any empirical study that tries to assess them. A natural first step would be to empirically confirm that hypothesis assessment biases indeed occur while using visualizations.

To mitigate the confirmation bias in particular, several strategies have been proposed in the psychology literature. These include the "analysis of competing hypotheses" and "evidence marshalling" [START_REF] Heuer | Analysis of competing hypotheses[END_REF]. These methods respectively encourage analysts to generate multiple hypotheses and to carefully record evidence confirming or rejecting each of them before reaching any conclusion. Some software tools help users follow those methods [START_REF] Wright | The sandbox for analysis: concepts and methods[END_REF] by facilitating the recording and the linking of evidence with hypotheses. These approaches could likely mitigate other biases in this category (such as the the congruence bias and the illusory truth effect), opening new opportunities for research.

As visualization designers, we could consider other possible design features as possible ways to mitigate hypothesis assessment biases. For example, we could study if the confirmation and other related biases can be reduced by showing what data has already been examined in our visualizations, and what has been ignored [5]. Or if they can be reduced when we suggest or require specific analytic workflows to be followed in our tools. While we strive to label all displays clearly, there may be cases where temporarily hiding labels could reduce hypothesis assessment biases by emulating a "blind test" situation. For example, consider an analyst examining evidence about crime or input statistics based on ethnicity. If at the initial review of income or crime data, the visualization displayed simple labels (such as A, B and C), instead of the actual ethnicity value, it could remove possible preconceptions about ethnicity and aid analysts consider all evidence at their disposal.

Beyond pinpointing opportunities to test for these biases and to mitigate them in visualizations, we also hope to increase awareness of the existence of hypothesis assessment biases within our community, since as researchers we can be prone to them. We hope this awareness will help visualization researchers adopt themselves methods that are more robust to these biases. For example, confirmation bias can be reduced by conducting more risky hypothesis tests [START_REF] Meehl | Theory-testing in psychology and physics: A methodological paradox[END_REF] (e.g., including tasks that might refute our hypotheses), while strategies also exist to address experimenter effects [START_REF] Rosenthal | Artifacts in Behavioral Research: Robert Rosenthal and Ralph L. Rosnow's Classic Books[END_REF].

Biases in causal attribution tasks

By causal attribution task, we refer to any task involving an assessment of causality [START_REF] Kelley | The processes of causal attribution[END_REF]. In social psychology, the attribution theory studies how people explain the causes of behavior and events [START_REF] Kelley | The processes of causal attribution[END_REF]. For example, when interpreting an event, people can either attribute the event to external factors (e.g., John had a car accident because the road was in bad condition) or to internal ones (e.g., John had a car accident because he is not a good driver).

The CAUSAL ATTRIBUTION category includes all systematic biases that have been experimentally observed when participants were asked to provide explanations of events or behaviors. We identified 12 causal attribution biases, listed in Table 2.

Psychology research

Attribution biases have been mostly studied in social psychology, so experimental scenarios typically focus on judgments of human behavior. They reveal people's tendency to favor themselves over others in the explanations they give. For example, the egocentric bias suggests that people tend to overestimate their contribution when asked to explain why a joint achievement was successful [START_REF] Ross | Egocentric biases in availability and attribution[END_REF]. Similarly, the self-serving bias suggests that people tend to attribute success to their own abilities and efforts, but ascribe failure to external factors [START_REF] Campbell | Self-threat magnifies the selfserving bias[END_REF]. For example, a student can attribute a good exam grade to their own effort, but a poor one to external factors such as the poor quality of their teacher or the unfair questions in the exam. When it comes to failures, according to the actor-observer bias, people tend to attribute their own to situational factors, but attribute the failures of others to personality weaknesses [START_REF] Miller | Actor-observer differences in perceptions of effective control[END_REF]. For example, they are more likely to attribute a car accident they had to bad road conditions or other drivers, but attribute the car accidents of others to their poor driving skills. People also tend sometimes to attribute others' ambiguous behavior to intentionally negative reasons, e.g., I see my peers laugh, they may be laughing about me [START_REF] Graham | Attributional and emotional determinants of aggression among african-american and latino young adolescents[END_REF] (hostile attribution bias).

Attribution biases occur not only when people unfairly evaluate their actions against those of others, but also the actions of members of their group (in-group members) to people outside it (out-group members). For example, in the ultimate attribution error, when Hindu and Muslim participants were asked to explain undesirable acts performed by Hindus or Muslims, Hindus attributed external causes to the acts of fellow Hindus, but internal causes (e.g., related to personality) for undesirable acts committed by Muslims, and vice versa [START_REF] Pettigrew | The ultimate attribution error: Extending allport's cognitive analysis of prejudice[END_REF]. When judging the actions of out-group members, people also tend to overgeneralize individual behaviors For example, in the group attribution error people tend to generalize decisions made by a group to individual people (e.g., the action of a whole nation is also the preference of an individual citizen) [START_REF] Allison | The group attribution error[END_REF].

Visualization research

Causal attribution tasks can also be common when using visualizations. Such tasks are explicitly identified by Amar and Stasko [START_REF] Amar | A Knowledge Task-Based Framework for Design and Evaluation of Information Visualizations[END_REF] as formulate cause and effect tasks in their taxonomy. In principle, any analytic task can involve causal attribution when users try to explain why a phenomenon occurs while exploring their data, like trying to explain peaks or outliers. For example, there are tasks when an analyst is trying to determine "Why are there more mass killings in the US than other countries?" or "What caused the recent decrease in road fatalities in France?".

Data analytic activities include describing patterns in data, but can also include prescribing decisive steps based on those patterns, often relying on the user's internal causal model of what factors affect what outcomes in the data. Some visual tools exist already to help conduct such analysis (e.g., the cause and effect analysis diagrams [START_REF] Ishikawa | Guide to Quality Control[END_REF]), but further study of their effectiveness is needed.

Although causal attribution biases have not been the subject of substantial work in visualization, the area is potentially ripe for research. Past studies have identified situations where analysts reached wrong causal conclusions based on the existence of correlations. For example, when it was observed that hormone replacement therapy (HRT) patients also had a lower-than-average incidence of coronary heart disease, doctors proposed that HRT protected against hear disease; a later analysis indicated that it was more likely that because HRT patients came from higher socio-economic groups they followed better-than-average diet and exercise regimens [START_REF] Lawlor | Commentary: The hormone replacementcoronary heart disease conundrum: is this the death of observational epidemiology?[END_REF]. It might be fruitful to study visualization designs that might minimize false causal attributions when correlations are present.

Another possible research direction comes from anecdotal data that dashboard users are more likely to make unwarranted causal claims from visualized data. Upon seeing a graph showing that drivers using a new GPS system get in more accidents (1 percent per year) compared to users of older systems (2 percent per year), viewers are anecdotally more likely to incorrectly posit that the new GPS device leads to more accidents [START_REF] Shapiro | 3 ways data dashboards can mislead you[END_REF]. To draw the correct conclusions, viewers have to consider pre-existing driver behavior. In this case, most safe drivers had not used the new system. It was mostly risk-taking drivers (that tend to have more accidents anyway) that had decided to use the new GPS system, thus inflating the number of accidents for the new system. In fact, the new system improved safety for both driver categories when considered separately. It is important to collect empirical data to demonstrate the existence of this types of potentially critical errors that stem from the choice of data combinations to visualize.

As we saw in the previous section, another side effect of faulty causal attribution is that people can not properly monitor the result of a joint action (for example neglecting or undervaluing the contributions of others. Effective collaboration though can be essential in visual data analysis, for example when multiple investigators are monitoring different suspicious individuals as a dangerous situation evolves in real time. It would be interesting to investigate if adding visualizations that demonstrate colleagues activity can promote a more balanced appreciation of others' work.

Biases in recall tasks

The RECALL category includes all systematic biases that have been experimentally observed when participants were asked to recall or recognize previous material. We identified 39 recall biases, listed in Table 2.

Psychology research

Memories are not copies of past experiences, but are instead reconstructed at the time of recall [23].

This means that post-event information can change a remembered event, known as misinformation effect [117]. More generally, people tend to better recall visual representations over words [START_REF] Mcbride | A comparison of conscious and automatic memory processes for picture and word stimuli: A process dissociation analysis[END_REF], auditory information over visual information [START_REF] Ginns | Meta-analysis of the modality effect[END_REF], self-generated content over read content [START_REF] Slamecka | The generation effect: Delineation of a phenomenon[END_REF], pleasant over unpleasant emotions [START_REF] Walker | The fading affect bias: But what the hell is it for?[END_REF], interrupted tasks over completed ones [START_REF] Einstein | Forgetting of intentions in demanding situations is rapid[END_REF], humorous [START_REF] Summerfelt | The effect of humor on memory: Constrained by the pun[END_REF] or bizarre items [START_REF] Mcdaniel | The bizarreness effect: it's not surprising, it's complex[END_REF], and information that took more work to comprehend [START_REF] Craik | Depth of processing and the retention of words in episodic memory[END_REF] or easy to find through a search engine (known as the Google effect) [START_REF] Sparrow | Google effects on memory: Cognitive consequences of having information at our fingertips[END_REF]. People can also mistake the ideas of others as original thoughts, [START_REF] Brown | Cryptomnesia: Delineating inadvertent plagiarism[END_REF], which can be an unintentional cause of plagiarism. Conversely, people consider some imaginary events as real [START_REF] Brainerd | Fuzzy-trace theory and false memory[END_REF], a phenomenon often observed in criminal witness interviews after misleading suggestions [START_REF] Lindsay | The eyewitness suggestibility effect and memory for source[END_REF].

Visualization research

In theory, using visualizations should help a viewer overcome limited (and biased) memory recall processes, by supplementing memory with unbiased views of all relevant information. But in complex datasets, not all information could be displayed, and even if it could, it could not be all be processed by the viewer. In addition, the viewer would still rely on a limited and biased memory system to link viewed data points with examples, context, and emotions. So visualizations have the ability to decrease, but not to eliminate, biased recall processes. One solution to this problem may be annotation systems that guide an observer to record observations and judgments for subsets of viewed data, but then later organize and provide automated comparisons of the user's preferences. Some properties of visualizations can make them more memorable. Including real-world images or objects or making visualizations distinct from others can lead participants to better recall that they saw those visualizations (e.g., [START_REF] Borkin | Beyond memorability: Visualization recognition and recall[END_REF]). Some work is beginning to additionally show that memory can be improved for the data patterns depicted in visualizations. Converting a traditional bar graph to a stack of iconic pictures of objects (e.g., depicting a number of baseball games as a stack of baseballs) can improve shortterm memory for depicted information [START_REF] Haroz | Isotype visualization: Working memory, performance, and engagement with pictographs[END_REF]. Linking data patterns to real-world objects (e.g., noting that an uptick in global temperatures looks like a hockey stick) can improve long-memory for those patterns, when testing weeks later [START_REF] Bateman | Useful junk?: the effects of visual embellishment on comprehension and memorability of charts[END_REF]. An automated system that gives suggestions for data shape mnemonics could bolster limited human memory.

Biases in opinion reporting tasks

The OPINION REPORTING category includes all systematic biases that have been experimentally observed when participants were asked to answer questions regarding their beliefs or opinions on political, moral, or social issues. We identified 21 opinion reporting biases, listed in Table 2.

Even though participants' opinions can play a role in much of the other bias categories, in the OPINION RE-PORTING category the task is to explicitly report this opinion (e.g., Americans are smart). In contrast, in the CAUSAL ATTRIBUTION category, the task is to explain a phenomenon (e.g, the US enjoys economic growth because Americans are smart). In the HYPOTHESIS ASSESS-MENT category the goal is to investigate if a statement is true or false (e.g., according to these data, i.e., US IQ scores, articles, facts, are Americans smart or not?). In the ESTIMATION category, the goal is to assess a quantity or predict an outcome (e.g., the US will likely grow, because Americans are smart). OPINION REPORTING biases differ from other categories, as people who have certain beliefs will not necessarily reason or predict the future based on these beliefs.

Psychology research

According to the bandwagon effect, people's reported beliefs on issues such as abortion can change according to the majority opinion [START_REF] Nadeau | New evidence about the existence of a bandwagon effect in the opinion formation process[END_REF]. Yet, people tend to believe that others are more biased (naive cynicism) [START_REF] Kruger | naive cynicism" in everyday theories of responsibility assessment: On biased assumptions of bias[END_REF] and more affected by mass media propaganda (third-person effect) [START_REF] Antonopoulos | Web third-person effect in structural aspects of the information on media websites[END_REF], compared to themselves. People also tend to generalize some characteristics from a member of a group (e.g., race, ethnicity, gender, age) to the entire group, often ignoring conflicting evidence (stereotyping) [START_REF] Correll | The police officer's dilemma: using ethnicity to disambiguate potentially threatening individuals[END_REF]. Finally, people tend to assign moral blame depending on outcomes, not on actions (moral luck) -for example, not wearing a seatbelt is seen as more particularly irresponsible if an accident happens [START_REF] Cushman | Crime and punishment: Distinguishing the roles of causal and intentional analyses in moral judgment[END_REF].

Visualization research

The OPINION REPORTING category is inherently linked to people's attitudes, moral beliefs and behavior, rather than biases observed in more general analytical tasks, and may be less useful to visualization researchers. However, similarly to all other bias categories, the possible connection of such errors to visualization systems is an unexplored topic, in particular when it comes to bias alleviation.

Other tasks

The last OTHER category includes all systematic biases that have been experimentally observed without being tied to any of the tasks discussed previously. We identified 5 other task biases, listed in Table 2.

Several biases in this category involve observing behavior rather than assessing responses. For example, according to the unit bias, people tend to eat more food in bigger containers [START_REF] Geier | Unit bias: A new heuristic that helps explain the effect of portion size on food intake[END_REF]. Another example is the tendency of investors to monitor their portfolios less frequently when they show negative information [START_REF] Karlsson | The ostrich effect: Selective attention to information[END_REF] (the ostrich effect). People also tend to develop more risky behavior once their perceived safety increases, e.g., to drive faster with a car with better airbags [START_REF] Hedlund | Risky business: safety regulations, risk compensation, and individual behavior[END_REF] (risk compensation).

Although these biases are not tied to a specific large category of tasks, they may be relevant for visualization design. For example, the unit bias might be relevant for visual judgments of quantity, e.g. increased white space around a collection of scatterplot points, due to axis scaling choices, might affect judgments of how many data points are present. The ostrich effect would be relevant for any data or analysis display where people might downplay or ignore information that the viewer would consider negative, and suggests that automated systems could highlight this information to counteract the bias. Biases similar to risk compensation might arise when a viewer is considering how to set thresholds for data values to appear in a view, and changing that threshold based on new, but irrelevant, information about other parameters.

DISCUSSION

Benefits of a task-based approach

The proposed taxonomy is organized by task, as opposed to previous efforts that were based on often untested, hard to grasp and even conflicted explanations of why a cognitive bias occurs. We believe that this organization will make it easier for visualization researchers to find out which biases may be relevant to their system or research area. It assumes that a task analysis has been performed (which is a standard user interface design practice), rather than requiring visualization researchers to guess which inner cognitive processes users may have to follow.

Moreover, this new task-based classification of cognitive biases may reveal new patterns by presenting biases from a different angle. For example, similarities between tasks may reveal biases with the same root.

Finally, our taxonomy preserves the pointers to the original experiments, which may help visualization researchers conduct new evaluations using methodologies that are wellestablished in other fields.

It is likely that additional biases will be identified and the list of biases will have to be further expanded. To our knowledge, this taxonomy is by far the largest in the literature and includes biases studied in different research domains (e.g., psychology, consumer research, sociology).

Visualization tasks

We derived our categories of biases from an analysis of experimental tasks used to detect those biases. These categories therefore capture tasks that do not necessarily align with visualization tasks that are described in visualization taxonomies (e.g., look-up, explore, identify, compare [START_REF] Brehmer | A multi-level typology of abstract visualization tasks[END_REF]) and used in empirical visualization work [START_REF] Kim | Assessing Effects of Task and Data Distribution on the Effectiveness of Visual Encodings[END_REF], [START_REF] Saket | Task-Based Effectiveness of Basic Visualizations[END_REF]. Such tasks tend to be lower-level but can be building blocks to many of the higher-level tasks in our taxonomy (e.g., identifying and comparing options before making a DECISION ). However, as seen in our categories, some of our tasks are indeed shared with visualization taxonomies, such as hypothesis assessment or cause and effect formulation [START_REF] Amar | A Knowledge Task-Based Framework for Design and Evaluation of Information Visualizations[END_REF].

Nevertheless, all tasks we identified are highly relevant to the goals of visualization systems and studies. For example, users of decision-support systems often have to make choices (e.g., multi-attribute choices [4]), and the DECISION category reveals the biases that are likely to be a factor when users perform such tasks. Similarly, visualization researchers interested in the memorability of visualization designs [START_REF] Borkin | Beyond memorability: Visualization recognition and recall[END_REF], [START_REF] Healey | Attention and visual memory in visualization and computer graphics[END_REF] can focus on RECALL biases. Researchers who study confirmatory analysis tasks [START_REF] Keim | Challenges in visual data analysis[END_REF] could start with the HYPOTHESIS ASSESSMENT category and researchers working on uncertainty visualization [9] may want to focus on the ESTIMATION category.

Opportunities for future research

There are so few studies of cognitive biases in visualization that the topic offers many opportunities for future visualization research. Researchers can draw from the rich set of cognitive biases provided in Table 2 by choosing a bias, testing whether the bias persists when standard visualizations are provided, and if so, investigate whether the bias can be alleviated by using improved designs [91].

Previous visualization research provides examples of methodological approaches for studying cognitive biases in an information visualization context and sometimes discuss pitfalls. For example, Micallef et al. [START_REF] Micallef | Assessing the effect of visualizations on bayesian reasoning through crowdsourcing[END_REF] suggest that when an experiment includes a task whose answer can be calculated numerically, it is recommended to i) use a continuous error metric rather than a dichotomous "correct/incorrect" metric, and ii) include conditions where no numeral is provided, in order to force participants to derive the answer from the provided visualization. Another pitfall consists of not presenting the same information in all conditions [START_REF] Khan | Benefits of visualization in the mammography problem[END_REF]: in order to demonstrate that a visualization can alleviate a cognitive bias, it is crucial to ascertain that the improvement over the baseline is due to the visualization itself, and not differences in the information presented.

Studying cognitive biases in an information visualization context also provides opportunities to extend methods and results from psychology. For example, the attraction effect was a decision bias that was only defined with three alternatives in numerical tables. In a visualization study, Dimara et al. [START_REF] Dimara | The attraction effect in information visualization[END_REF] extended the definition of the attraction effect to more than three alternatives and proposed a procedure for constructing a stimuli dataset.

The psychology literature often suggests ways a bias could be alleviated, and some of the strategies may be applied to visualization experiments [91]. Since alleviation strategies are bias-specific [91], it is impossible to cover them all in this article. As previous work has illustrated [START_REF] Dimara | Accounting for availability biases in information visualization[END_REF], [START_REF] Dragicevic | Visualization-mediated alleviation of the planning fallacy[END_REF], [91], each bias needs its own survey of the literature. We hope our taxonomy will facilitate such surveys by providing references as starting points.

Limitations

154 biases is a lot, but it is likely that more will be discovered. While all biases are listed and classified in table Table 2 the paper itself could only discuss a subset of the biases. We focused our discussion on the biases that we felt were most important to visualization, most well-established in psychology, and most reflected a rationality violation. Furthermore, each bias was assigned a single category. Our methodology for creating the taxonomy leaves open the possibility that the same cognitive bias exists in more than one task type, across different studies. Even though most academic studies tend to consistently replicate the same tasks, this concern is indeed a possibility, but not necessarily a limitation. The assumption behind the classification of our taxonomy is that different user tasks should be approached differently by researchers. A good example of such a case where the same bias has been observed in two different tasks exists in the literature on the attraction effect. The attraction effect has been massively replicated as a decision task among three commercial products. Some papers exist that tested the attraction effect in visual judgments tasks, such as identifying which of the two rectangles is bigger [START_REF] Trueblood | Not just for consumers context effects are fundamental to decision making[END_REF] or finding similarities in circle and line pairs [START_REF] Choplin | Comparison-induced decoy effects[END_REF]. Even though these cases appear similar to the attraction effect and likely have similar roots, it is best if they are approached as perceptual biases, since people mainly fail to encode the visual property of an object.

Also, the initial coding and sorting was conducted by a single person. This was mitigated by having multiple reviews and an iterative process involving all five authors. To keep the number of overall citations manageable, the search of the representative paper stopped at the first that satisfied the source eligibility requirements and was not exhaustive. Our taxonomy is a starting point, but to study a specific bias in depth requires a separate literature review.

Finally, a cognitive bias assumes by definition a "deviation from reality", a notion that is complex and controversial. We still do not have a definitive proof that the known cognitive biases actually reflect irrationality. Therefore, In-foVis researchers should attempt to verify in their studies that erroneous responses really reflect irrationality, and not some optimal strategy based on alternative interpretations of the task. We also encourage visualization researchers to remain updated about the current debates in cognitive bias research surrounding the concept of irrationality.

CONCLUSION

This paper classified 154 cognitive biases, cases where people systematically and involuntarily deviate from what is expected to be a rational "reality". For example, their decisions are often influenced by reasons irrelevant to the objective qualities of the decision alternatives. Our classification is task-based (when the bias occurs), rather than explanatory (why it occurs), to help visualization researchers identify possible bias that could affect their visualization tasks.

Cognitive biases are often mentioned as important in the visualization literature [9]. Some works indeed discuss cognitive biases in the context of visualizations [START_REF] Micallef | Assessing the effect of visualizations on bayesian reasoning through crowdsourcing[END_REF], [START_REF] Boy | Showing people behind data: Does anthropomorphiz-ing visualizations elicit more empathy for human rights data?[END_REF], but they do not provide evidence of detecting or of alleviating the bias when using visualizations. In our review we only found one empirical study [91] that alleviates a cognitive bias using visualization designs. More generally, it seems that there are very few visualization studies (e.g., [START_REF] Dimara | The attraction effect in information visualization[END_REF]) that even provide evidence for the existence of cognitive biases in visualizations. We believe this space provides ample opportunities for research in visualization, and we hope the directions we suggest in the different bias categories will inspire future work.

Fig. 1 :

 1 Fig. 1: Overview of 154 cognitive biases organized by experimental task. Dots represent the different cognitive biases.

Fig. 2 :

 2 Fig. 2: Our taxonomy organized by the flavors discussed in section 3.4. Each dot is a cognitive bias. Colors encode taskcategory (ESTIMATION , DECISION , HYPOTHESIS ASSESSMENT , CAUSAL ATTRIBUTION , RECALL , OPINION REPORTING , and OTHER ).

  Evanthia Dimara is a research scientist at the ISIR Laboratory (HCI group) of Sorbonne University. Her fields of research are human-computer interaction and information visualization. Her focus is on decision makinghow to help people make unbiased and informed decisions alone or in groups.Steven FranconeriSteven Franconeri is a Professor of Psychology at Northwestern University, and Director of the Northwestern Cognitive Science Program. He studies visuospatial thinking and visual communication, across psychology, education, and information visualization. Catherine Plaisant Catherine Plaisant is a research scientist at the University of Maryland, College Park and assistant director of research of the University of Maryland Human-Computer Interaction Lab. Anastasia Bezerianos Anastasia Bezerianos is assistant professor in Univ. Paris-Sud and part of the ILDA Inria team, France. Her interests include interaction and visualization designs for large displays, visual perception, user evaluation, and collaborative work. Pierre Dragicevic Pierre Dragicevic is a permanent research scientist at Inria, France, in the Aviz team. His research interests include data physicalization, and visualizations for judgment and decision making.

TABLE 1

 1

	: Color legend of the relation of each cognitive bias to visualization research
	#8 #7 #6 #5 #4 #3 #2 #1	Evidence for the alleviation of the cognitive bias in visualization Evidence for the existence of the cognitive bias in visualization Studied in visualization, but no clear evidence of existence or alleviation Discussed in visualization research as important, but not yet studied Not discussed in visualization but likely relevant Probably relevant to visualization Potentially relevant to visualization Relevance to visualization currently unclear

TABLE 2 :

 2 Our taxonomy of cognitive biases classified by the tasks: ESTIMATION , DECISION , HYPOTHESIS ASSESSMENT , CAUSAL ATTRIBUTION , RECALL , OPINION REPORTING , and OTHER . The first column shows the task category color of each bias. The column "Flavor" describes the phenomenon behind the bias (discussed in Sec. 3.4). The column "Cognitive bias" shows the name of each bias. The column "Ref" is a representative peer-reviewed paper for each bias ( described in Sec. 3.2). The column "Relevance to InfoVis" indicates whether each cognitive bias has been examined in information visualization research and reports the reference of the paper (discussed in Sec. 4 ). The color coding of the "Relevance to InfoVis" column is explained in detail in legend Table1

	#	Flavor Cognitive bias	Ref	Relevance to InfoVis Short description
	TASK: ESTIMATION
	1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33	Association Baseline Inertia Outcome Self-perspective	Availability bias Conjunction fallacy Empathy gap Time-saving bias Anchoring effect Base rate fallacy Dunning-Kruger effect Gambler's fallacy Hard-easy effect Hot-hand fallacy Insensitivity to sample size Regressive bias Subadditivity effect Weber-Fechner law Conservatism Exaggerated expectation Illusion of validity Impact bias Outcome bias Planning fallacy Restraint bias Sexual overperception bias Curse of knowledge Extrinsic incentives bias False consensus effect Illusion of control Illusion of transparency Naive cynicism Optimism bias Out-group homogeneity bias Pessimism bias Spotlight effect Worse-than-average effect	[52] [47] [147] [148] [50] [46] [57] [17] [56] [150] [17] [151] [152] [42] [48] [154] [17] [155] [156] [53] [157] [158] [66] [159] [160] [161] [162] [134] [51] [163] [164] [165] [166]	#5 [26], [146] #5 [9] #1 #4 #7 [7], [64] #6 [59], [60] #5 [149] #4 #3 #5 [146] #5 [9], [10] #4 #4 #6 [153] #7 [92] #4 #5 [9] #1 #2 #5 [67] #1 #1 #7 [65] #1 #2 #3 #1 #2 #4 #4 #4 #1 #3	Events more probable if easy to remember Specific outcomes more probable than general Estimations affected by not recognizing the role of current emotional state Overestimate time saved when increasing speed Estimation affected by first piece of information Ignore base rate probability of general population Low-ability people overestimate their performance (opposite for high-ability) Current outcome that is more frequent will be less frequent in future Overconfidence for hard tasks, underconfidence for easy Current outcome that is more frequent will be more frequent in future Estimate probability ignoring sample size Overestimate high probabilities, underestimate low ones Overall probability less than the probabilities of the parts Failure to perceive small differences in large quantities New information insufficiently updates probability estimates Exaggerating evidence to fit a conclusion Overconfidence in judgment based on intuition and anecdotes Predict future emotional reactions as more intense Evaluate decision maker only by choice outcome Overoptimistic task completion predictions, especially for self Overestimate of ability to resist temptation Over or underestimate of romantic interest from others Experts assume that novices have same knowledge Others have extrinsic motivations (e.g.money), self are intrinsic (e.g.learning) Overestimate the agreement of others with own opinions Overestimation of one's influence on an external event Overestimate insight of others into own mental state, and vice versa Predict that the others will be more egocentrically biased Positive outcomes more probable for oneself than others Estimate out-group will be more homogenous than in-group members Positive outcomes less probable for oneself than others Overestimate probability that people notice one's appearance/behavior Underestimate own achievements relative to others in difficult tasks
	TASK: DECISION
	34		Ambiguity effect	[72]
		Association

TABLE 3 :

 3 Alternative names for cognitive biases Subjective validation or Acceptance phenomenon or Personal validation Fundamental attribution error Correspondence bias or Attribution e Gambler's fallacy Monte Carlo fallacy or Fallacy of the maturity of chances Illusory truth e t Truth e t or Illusion-of-truth e t or Reiteration e t or Validity e or Frequency-validity relationship or Availability cascade Illusory superiority Above-average e t or Superiority bias or Leniency error or Sense of relative superiority or Primus inter pares e or Lake Wobegon e Impact bias Durability bias In-group favoritism In-group bias or In-group/out-group bias or Intergroup bias

	Bias	Synonym / Similar Biases
	Actor-observer asymmetry Anchoring e t Anthropomorphism Attraction e t Availability bias Base rate fallacy Childhood amnesia Choice-supportive bias Clustering illusion Confirmation bias Conjunction fallacy Conservatism Cross-race e t Cryptomnesia Cue-dependent forgetting Digital amnesia Duration neglect Hard-easy e t Hyperbolic discounting Empathy gap Endowment e t Escalation of commitment Experimenter e t Fading a t bias Focusing e t Barnum e t t or Hindsight bias Actor-observer bias Focalism or Anchoring Personification Decoy e t or Asymmetric dominance e Availability heuristic Base rate neglect or Base rate bias or Extension neglect Infantile amnesia Post-purchase rationalization Texas sharpshooter fallacy Confirmatory bias or Myside bias or Semmelweis reflex Linda problem Belief revision Cross-race bias or Other-race bias or Own-race bias Inadvertent plagiarism Context e t or Retrieval failure Google e Extension neglect or Peak-end rule Discriminability e t or Difficulty e Dynamic inconsistency Projection bias Divestiture aversion or Mere ownership e Irrational escalation of commitment or Commitment bias or Sunk cost fallacy Experimenter-expectancy e t or Observer-expectancy e or Experimenter's bias or Expectancy bias or Subject-expectancy e FAD Focusing illusion Forer e Knew-it-all-along e t or Creeping determinism Hostile attribution bias Hostile attribution o Hot-hand fallacy Hot hand phenomenon or Hot hand n o i t a c fi i t s u j t r o f f E t c e f f e A E K I r e m m a h n e d l o G r o l e v a g s ' w o l s a M r o t c e f f e g n i c a p S t c e f f e g a L Mere exposure e t Familiarity principle Money illusion Price illusion Negativity bias Negativity e Not invented here NIH Optimism bias Unrealistic optimism or Comparative optimism Pessimism bias Pessimistic bias Rhyme-as-reason e t Eaton-Rosen phenomenon Risk compensation Peltzman e Self-generation e t Generation e Self-reference e t Self-relevance e Serial position e t Recency e t or Primacy e Survivorship bias Survival bias Telescoping e t Telescoping bias Testing e t Retrieval practice or Practice testing or Test-enhanced learning f o w a L s ' n o s n i k r a P y t i l a i v i r T Von Restor e

t Isolation e Weber-Fechner law Weber's law or Fechner's law Worse-than-average e t Below-average e

The ballot names bias shares some similarities with the serialpositioning effect[START_REF] Murdock | The serial position effect of free recall[END_REF] where people better recall the first (primacy) and last (recency) items in a list. However, it is not the same bias as the ballot task is to choose (a candidate) and not to recall her name. But it is possible that people chose the candidates who were easier to remember.

ACKNOWLEDGMENTS

We thank G. Bailly and E. Lee for their precious feedback.