Nicholas Asher

When learning becomes impossible

Keywords: learning, bias, agent modeling, echo chambers, Bayesian learning, philosophical foundations

We formally analyze an epistemic bias we call interpretive blindness (IB), in which under certain conditions a learner will be incapable of learning. IB is now common in our society, but it is a natural consequence of Bayesian inference and what we argue are mild assumptions about the relation between belief and evidence. IB a special problem for learning from testimony, in which one acquires information only from text or conversation. We show that IB follows from a codependence between background beliefs and interpretation in a Bayesian setting and the nature of contemporary testimony. We argue that a particular characteristic of contemporary testimony, argumentative completeness, can preclude learning in hierarchical Bayesian settings, even in the presence of constraints that are designed to promote good epistemic practices.

INTRODUCTION

In this paper, we describe and formally analyze a simple epistemic bias, in which people end up being unable to learn, unable to shift their beliefs in the face of evidence. We call this bias interpretive blindness (IB). IB is now familiar and all around us-people dying of Covid but refusing to believe that the disease exists despite the evidence, people who believe elections are fraudulent in spite of overwhelming evidence to the contrary, the list goes on. IB is based on a codependence between what beliefs we have and what evidence we use to update those beliefs; it then results from exploiting simple rules of Bayesian inference in a dynamic, iterative process whereby a learner's background beliefs and biases lead her to update her beliefs based on a body of testimony 𝑇 , while biases inherent in 𝑇 come back to reinforce her beliefs and her trust in 𝑇 , further biasing her towards accepting 𝑇 for future updates. We look at how this codependence affects human agents whose beliefs are guided and shaped by testimony-perhaps the primary way that most people acquire information nowadays. We show that an inability to learn can result, and we show how this inability is enabled and exacerbated by modern machine learning algorithms that can govern what testimony we have access to.

When learning through testimony, an agent acquires beliefs through conversations with other agents, or from books, newspapers or social networks, and so on. Typically, such people lack direct access to the phenomena described via that testimony or cannot analyze the phenomena themselves [START_REF] Millgram | The great endarkenment: philosophy for an age of hyperspecialization[END_REF]]. Typically too, humans only pay attention to a restricted set of bodies of testimony from a limited number of sources for their information-which makes sense in terms of an agent's limited resources and attention span. These conditions are the fertile ground for the learning inability of IB. In IB, agents' biases preclude learning when an agent tries to exploit new data that are incompatible with or simply distinct from 𝑇 ; agents will discount any evidence that challenges their beliefs. IB is enabled and exacerbated by algorithms that tailor the testimony they provide to that which the agent is already disposed to believe. Our paper formally analyzes the strategic epistemic consequences of these algorithms and of IB. We use Wolpert's 2018 extended Bayesian framework to prove our results.

IB is not only problematic for first order Bayesian approaches but for hierarchical ones [START_REF] Gelman | Bayesian data analysis[END_REF] as well, because testimony from sources like Facebook and other social media, 24/7 media outlets and web interest groups is often argumentatively complete, a notion we analyze precisely in Section 4. In an argumentatively complete body of testimony 𝑇 , the authors of that testimony can respond to and undercut any doubts raised by other data or arguments in a body 𝑇 ′ that might threaten 𝑇 's credibility. A skillful climate denier, for example, will always find a way to undercut the most scientifically careful argument for climate change. Argumentatively complete testimony can undermine higher order constraints and good epistemic practices that guide first order learning.

Our paper starts in Section 2 by introducing the codependence of belief and interpretation relative to testimony and the hypotheses that support it. In Section 3 we formally show how IB can result in first-order Bayesian learning.

Section 4 shows how IB can come about in a hierarchical Bayesian learning setting. Section 5 discusses related work while Section 6 develops a game theoretic setting to investigate the complexity of IB. We investigate whether IB is rationally refutable; on certain commonly accepted epistemic assumptions, we conclude that it is not.

TESTIMONY AND SOURCES

IB arises in learning because of a codependence between beliefs and the interpretation of evidence, in this case written or linguistically conveyed information. The interpretations we are interested in are judgments about the evidence's trustworthiness. In updating our beliefs with new evidence 𝐸, our beliefs, particularly about the reliability of 𝐸's source, color how we interpret 𝐸, how much in short we believe it. But new evidence 𝐸 ′ can update our beliefs about the reliability of sources which in turn confer a possibly new degree of belief concerning 𝐸. This codependence between evidence and belief dictates how we learn.

Let us look at this codependence more closely in conjunction with a body of testimony. A body of testimony 𝑇 is a collection of information conveyed by one or more sources that may "promote" or vouch for certain descriptions of events and cast doubt on or disparage others. The New York Times, Fox News, CNN, Facebook, 4Chan, all provide bodies of testimony; their union is also a body of testimony. While such bodies may be consistent or inconsistent, we restrict ourselves here to consistent 𝑇 . Importantly such bodies are also dynamic; they evolve over time as they are updated with new descriptions of events. Dynamic bodies of testimony are ubiquitous in our communicative landscape: on-line, 24/7 news sources as well as particular groups on social media provide continuously evolving, updated coverage of new events. To model this, we shall say that 𝑇 comes in "stages", where stages can be defined by times or even conversational turns, and each stage 𝑇 𝑖 is the body of evidence accumulated up to stage 𝑖. 𝑇 = {𝑇 1 ,𝑇 2 , ...,𝑇 𝑛 , ...} is the collection of all the stages of a dynamic body of evidence. Thus, a body of evidence invites us to iteratively update our beliefs about the trustworthiness of that very body.

Let T be a collection of (potentially conflicting) bodies of testimony 𝑇 about some phenomenon 𝑃, and assume that a learner f does not have independent access to 𝑃 and can thus only learn about it via T . Learning from T will require f to judge some body of testimony 𝑇 in T as credible or trustworthy. Let H be a set of evaluation hypotheses, where each ℎ ∈ H evaluates the bodies of testimony 𝑇 in T . The hypotheses ℎ ∈ H -background beliefs that may take into account 𝑇 's source, subject matter, past known accuracy, appeal, and perhaps other elements-define a conditional probability 𝑃 (𝑇 |ℎ) for each 𝑇 ∈ T , which we will sometimes write as ℎ(𝑇 ), where ℎ(𝑇 ) = 0 means 𝑇 is untrustworthy according to ℎ, and ℎ(𝑇 ) = 1 means 𝑇 is trustworthy according to ℎ. Following Wolpert's 2018 extended Bayesian framework, f determines his belief in 𝑇 relative to H .

Our learner f also has a probability distribution over the evaluation hypotheses in H . Since the bodies of evidence 𝑇 in T are dynamic, f updates this distribution relative to the stages 𝑇 𝑖 of 𝑇 as each 𝑇 evolves. This is intuitive; testimony 𝑇 to which f attends should be relevant to how f updates its beliefs about the trustworthiness of 𝑇 . The collection T is also most likely restricted, which is intuitive too: most if not all of us acquire new information from a restricted set of bodies of evidence-a reasonable choice given the balance rational agents need to find between exploiting an already acquired body of evidence and gathering data from other bodies of evidence. And finally, because we restrict ourselves to consistent bodies of evidence 𝑇 , each 𝑇 will push a particular point of view on events.

But as the probabilities of the background beliefs, the evaluation hypotheses, are updated over the stages 𝑇 𝑖 , so too will f update her belief in 𝑇 , using the updated probabilities for her evaluation hypotheses. This codependence can lead to a problem in learning: when we rely on testimony to learn and we restrict the testimony we pay attention to, the confirming evidence for the background beliefs, the evaluation hypotheses and the testimony 𝑇 they support mutually confirm each other to form a barrier to learning about events that are not mentioned in 𝑇 . Algorithms of the sort used in social media to build testimony for agents are optimized to continue the themes and ultimately the content of what those agents have previously consulted by exploiting the agent's history of choices that reveal her likes. We will formalize this in terms of a relation between the algorithm's choices and the presence of certain evaluation hypotheses.

Such algorithms naturally accelerate the process of what we call below bias hardening and IB in the presence of iterated Bayesian updating, despite our learner's rational epistemic practices. We turn to this in the next section.

IB IN A FIRST ORDER BAYESIAN SETTING

To formalize IB and its consequences, we first present a simple experiment to show how the codependence of interpretation and belief leads to bias hardening.1 To illustrate, suppose that f considers a consistent dynamic body of testimony 𝑇 = {𝑇 1 ,𝑇 2 , ...,𝑇 𝑛 , ...} and has two evaluation hypotheses ℎ 1 , ℎ 2 , where the prior probabilities assigned to ℎ 1 and ℎ 2 by f are:

𝑃 (ℎ 1 ) = .6, 𝑃 (ℎ 2 ) = .4 (1)
and the evaluation hypotheses assign probabilities to 𝑇 as it evolves through stages 𝑇 𝑖 as follows:

𝑃 (𝑇 𝑖 |ℎ 1 ) = .8, 𝑃 (𝑇 𝑖 |ℎ 2 ) = .2 for all i (2)
We can now calculate the probability of 𝑇 1 using the general rule for marginal probabilities in 3. Let B be f 's background beliefs; and let the set of all ℎ 𝑖 , the alternative hypotheses that are consistent with or assigned non-zero conditional probability relative to B [START_REF] Thomas L Griffiths | Bayesian models of cognition[END_REF][START_REF] Lampinen | Bayesian approach for neural networks-review and case studies[END_REF][START_REF] Tenenbaum | Theory-based Bayesian models of inductive learning and reasoning[END_REF][START_REF] Tenenbaum | How to grow a mind: Statistics, structure, and abstraction[END_REF], be the set of evaluation hypotheses ℎ 𝑖 (so {ℎ 1 , ℎ 2 }, in our example).

𝑃 (𝑥) = 𝑖=𝑘 𝑖=1 𝑃 (𝑥 |ℎ 𝑖 , B).𝑃 (ℎ 𝑖 , B) (3)
Then using (1), (2), and (3), we have:

𝑃 (𝑇 1 ) = 𝑃 (𝑇 1 |ℎ 1 ).𝑃 (ℎ 1 ) + 𝑃 (𝑇 1 |ℎ 2 ).𝑃 (ℎ 2 ) = .56. (4)
This is our estimation of our belief in the body of evidence 𝑇 based on what we have so far. We will continue to update the probability of 𝑇 given new stages 𝑇 𝑖 below by distinguishing prior probabilities 𝑃 𝑝𝑟𝑖𝑜𝑟 and updated probabilities 𝑃 𝑝𝑜𝑠𝑡 . Now suppose there is a new conversational turn in 𝑇 , a new stage of evidence 𝑇 2 . Given our assumptions, 𝑃 (𝑇 2 |ℎ 1 ) = .8, while 𝑃 (𝑇 2 |ℎ 2 ) = .2, 𝑇 2 is supported by ℎ 1 but not by ℎ 2 -ℎ 1 and ℎ 2 are consistent with their roles on 𝑇 1 .

Given the dependence of beliefs and interpretation of evidence, 𝑇 2 also leads us to re-evaluate our evaluation hypotheses by adapting Bayes' formula to our evaluation hypotheses:

𝑃 (ℎ 𝑖 |𝑇 𝑛+1 ) = 𝑃 (𝑇 𝑛+1 |ℎ 𝑖 ).𝑃 𝑝𝑟𝑖𝑜𝑟 (ℎ 𝑖 ) 𝑃 𝑝𝑜𝑠𝑡 (𝑇 𝑛 ) (5)
Given 𝑇 2 , whose initial probability we set to what the posterior calculated for 𝑇 1 -i.e., 𝑃 𝑝𝑜𝑠𝑡 (𝑇 1 ) = 𝑃 𝑝𝑟𝑖𝑜𝑟 (𝑇 2 ), we can update our confidence in ℎ 1 as follows:

𝑃 (ℎ 1 |𝑇 2 ) = 𝑃 (𝑇 2 |ℎ 1 ).𝑃 𝑝𝑟𝑖𝑜𝑟 (ℎ 1 ) 𝑃 𝑝𝑜𝑠𝑡 (𝑇 1 ) ≈ .86. (6)
Thus, we have posterior probabilities for our evaluation hypotheses as well as for stages of bodies of evidence. The similarly updated probability for ℎ 2 now drops to roughly .14. Using the updated values for ℎ 1 and ℎ 2 , we see that 𝑇 2 , which includes 𝑇 1 , is now even more believable: 𝑃 𝑝𝑜𝑠𝑡 (𝑇 2 ) = .74. Now suppose that a new bit of evidence, 𝑇 3 , is added to 𝑇 . As before, we set 𝑃 𝑝𝑜𝑠𝑡 (𝑇 2 ) = 𝑃 𝑝𝑟𝑖𝑜𝑟 (𝑇 3 ). Given our assumptions about our source functions, 𝑃 (𝑇 3 |ℎ 1 ) = .8, we have 𝑃 (ℎ 1 |𝑇 3 ) = .96, while 𝑃 (ℎ 2 |𝑇 3 ) ≈ 0.04, and confidence in 𝑇 3 is also updated: 𝑃 𝑝𝑜𝑠𝑡 (𝑇 3 ) = .776 ≈ .78. Updating ℎ 1 's probability conditional on new evidence 𝑇 4 now yields a value of .989 ≈ .99, while 𝑃 (ℎ 2 |𝑇 4 ) = 0.008 ≈ 0.01. By the time we get to 𝑇 5 , the probability of ℎ 1 will have gone to 1, while 𝑃 (ℎ 2 ) = 0, and 𝑃 (𝑇 5 ) = .8. In sum, as 𝑛 increases, the updated probabilities of ℎ 1 go to 1 and 𝑃 (𝑇 𝑛 ) → 𝑃 (𝑇 |ℎ 1 ), that is, to the strength of ℎ 1 's support for 𝑇 .

Our codependence of belief and evidence suggests a loopy structure (cyclic graph) for updating. However, by exploiting stages, we can disentangle such structures; and efficient approximations are possible in disentangled structures [START_REF] Murphy | Loopy belief propagation for approximate inference: An empirical study[END_REF]. Proposition 1 below shows a convergence under certain assumptions. Let P 𝑛 (ℎ 𝑖 ) be the probability of ℎ 𝑖 after conditionalizing on 𝑇 𝑛 and P 𝑛 (𝑇 ) the value of 𝑇 after n conditional updates as defined above.

Let's now move to a more general setting. Let f 's evaluation hypotheses H f , come with a probability distribution.

An agent could have, among the many evaluation hypotheses that she countenances, an evaluation hypothesis ℎ for which the conditional probability of 𝑇 given ℎ increases as 𝑇 evolves. The support for 𝑇 might increase (or decrease) as 𝑇 gets more extended with more and more stages.

Definition 1. An evaluation hypothesis ℎ ∈ H f is positive sensitive to 𝑇 = {𝑇 1 ,𝑇 2 , ...} iff 𝑃 (𝑇 𝑛 |ℎ) > .5 and is monotone increasing for all 𝑛.

How bodies of testimony are constituted for ordinary learners is not always clear. Learners can assemble their own body of testimony, and typically, they must concentrate on some testimony to the exclusion of other testimony. What are the criteria? Well, sometimes other actors can guide the acquisition of testimony. What we shall call Facebook-like algorithms from our epistemic perspective are a way social media and news organizations can steer learners to a certain body of testimony. Their role is to bring testimony to an agent's attention that feeds and updates, in fact constructs, an evaluation hypothesis that keeps the learner coming back to the same type of information, often the same set of sources of information. More formally, the role of such an algorithm is to construct a positive sensitive evaluation hypothesis in H f . Definition 2. A Facebook-like (FB) algorithm 𝑔 for f constructs a body of testimony from a set of bodies of testimony T that f uses to update her hypotheses and 𝑔(T

) = 𝑇 = {𝑇 1 ,𝑇 2 , ...} with 𝑔 𝑛 (T ) = 𝑇 𝑛 iff there is an ℎ ∈ H f such that: (i) ∃𝑛 𝑃 (𝑔 𝑛 (T )|ℎ) > .5 and (ii) ∀𝑚 > 𝑛 ∃𝑘 (𝑃 (𝑔 𝑘+𝑚 (T )|ℎ) > 𝑃 (𝑔 𝑚 (T )|ℎ) as long as 𝑃 (𝑔 𝑘+𝑚 (T )|ℎ) ≠ 1).
An FB algorithm 𝑔 𝑛 (T ) at each stage 𝑛 provides information that does not decrease ℎ's support for 𝑇 . Thus 𝑔 makes ℎ positive sensitive to 𝑇 through a choice of stages 𝑇 𝑛 . In addition, however, 𝑔 will eventually keep on increasing the support of ℎ for 𝑔(T ). We assume that if 𝑔 is an FB algorithm for f , then f updates her hypotheses based on the testimony fed by 𝑔. We suspect that actual social media algorithms are FB though we do not have a proof of this. The possibility, however, seems to us very real.

Proposition 1. Suppose testimony 𝑇 = {𝑇 1 ,𝑇 2 , ...,𝑇 𝑛 , ...}, and with ℎ 1 ∈ H positive sensitive to 𝑇 and with 𝑃 (ℎ 1 ) ≠ 0, while 𝑃 (𝑇 𝑛 |ℎ 𝑗 ) < .5 and is monotone decreasing for all 𝑛 and for all ℎ 𝑗 ∈ H, ℎ 𝑗 ≠ ℎ 1 . Then:

As 𝑛 → ∞, P 𝑛 (𝑇 ) → 𝑙𝑖𝑚𝑠𝑢𝑝 (𝑃 (𝑇 𝑛 |ℎ 1 )), P 𝑛 (ℎ 1 ) → 1 and P 𝑛 (ℎ 𝑗 ) → 0 for 𝑗 ≠ 1
Given the calculations above and using standard updating rules for the probabilities 𝑃 assigned by f , if 𝑃 (𝑇 𝑖 |ℎ 1 ) is monotonic increasing with respect to 𝑖 and 𝑃 (𝑇 𝑖 |ℎ 𝑗 ) for any 𝑗 ≠ 1 is monotonic decreasing, then the updates of 𝑃 (𝑇 𝑖 ), 𝑃 (ℎ 1 |𝑇 𝑖 ) and 𝑃 (ℎ 𝑗 |𝑇 𝑖 ) will follow the pattern of our experiment above and converge to the support of ℎ 1 , 1, and 0 respectively. □ Corollary 1. If f uses an FB algorithm and ℎ 𝑗 , 𝑗 ≠ 1 are as in Proposition 1

As 𝑛 → ∞, P 𝑛 (𝑇 ) → 𝑙𝑖𝑚𝑠𝑢𝑝 (𝑃 (𝑇 𝑛 |ℎ 1 )), P 𝑛 (ℎ 1 ) → 1 and P 𝑛 (ℎ 𝑗 ) → 0 for 𝑗 ≠ 1.

Given that an FB algorithm entails the existence of a positive sensitive hypothesis ℎ 1 to 𝑇 and that ℎ 1 must have non 0 probability, the result follows from Proposition 1 □.

We now introduce three important properties of evaluation hypotheses. Definition 3. An evaluation hypothesis ℎ for a set of bodies of testimony

T is consistent iff for 𝑇 ,𝑇 ′ ∈ T , if 𝑇 ∪ 𝑇 ′ is inconsistent, then 𝑃 (𝑇 |ℎ) = 1 -𝑃 (𝑇 ′ |ℎ).
An evaluation hypothesis ℎ is probability-wise model complete (PWMC) for 𝑇 and some topic 𝑡 iff: for any putative piece of evidence 𝜙 on 𝑡 if for no stage 𝑇 𝑖 𝑇 𝑖 |= 𝜙 (𝜙 is not predicted or included in any

stage of 𝑇 ), then 𝑃 (𝜙 |ℎ) = 1 -𝑃 (𝑇 |ℎ). Definition 4. An evaluation hypothesis ℎ ∈ H makes 𝑇 potentially trustworthy (ℎ |= 𝑇 ), if as 𝑛 → ∞, 𝑃 (𝑇 𝑛 |ℎ) → 1.
Proposition 2. Suppose 𝑔(T ) = 𝑇 and 𝑔 is an 𝐹 𝐵 algorithm for f . Then there is an evaluation hypothesis ℎ such that ℎ |= 𝑇 .

Let 𝑔 be an 𝐹 𝐵 algorithm for f . Then as we showed above, there is an ℎ that is positive sensitive to 𝑇 and by the property (ii) of Facebook algorithms, 𝑙𝑖𝑚 𝑛→∞ 𝑃 (𝑇 𝑛 |ℎ) = 1. □ While consistency seems a basic requirement of evaluation hypotheses, a potentially trustworthy evaluation hypothesis is a kind of "soundness" or accuracy assumption about a body of evidence. For an agent who remains wedded to a body of testimony, such an assumption also seems rational.

PWMC hypotheses generalize consistent hypotheses, but what is their rationale? FB algorithms by themselves don't lead to PWMC. But they do facilitate it. As 𝑔 feeds testimony to f that lends more and more support to an eventually trustworthy hypothesis ℎ, a natural thought is for f to assume that 𝑇 𝑖+1 provides a more complete coverage of the pertinent facts than 𝑇 𝑖 . So elements 𝜙 that are not mentioned in any 𝑇 𝑖 or are in some way incompatible with 𝑇 are either not relevant, or just false. The PWMC condition codifies this for evaluation hypotheses in terms of an operation akin to negation as failure in Prolog; if ℎ makes 𝑇 probability wise model complete, then if given some newsworthy and relevant topic 𝑡 and𝑇 doesn't mention 𝜙, then ℎ supports ¬𝜙 to the extent that ℎ supports 𝑇 . For simplicity, we will assume that the topic parameter is constant in our discussions below and drop it from the formalism. Proposition 3. Let T be a set of consistent bodies of testimony and let 𝑔 be a FB algorithm for f with 𝑔(T ) = 𝑇 with ℎ 1 ∈ H f the entailed positive sensitive hypothesis to 𝑇 . Let the priors on ℎ 𝑖 ∈ H, ℎ 𝑖 ≠ ℎ 1 be as in Proposition 1 Then:

As 𝑛 → ∞, P 𝑛 (𝑇 ) → 1.
(1)

Suppose in addition, ℎ 1 is PWMC for 𝑇 and 𝑇 ̸ |= 𝑇 ′ . Then:

As 𝑛 → ∞, P 𝑛 (𝑇 ′ ) → 0 (2)
To show (1), note that 𝑃 (𝑇 1 |ℎ 1 ) > .5 and since as 𝑛 → ∞, P 𝑛 (ℎ 1 ) → 1, after a certain point 𝑃 (𝑇 𝑛 |ℎ 1 ) is monotone increasing. Then by Proposition 1, P 𝑛 (𝑇 ) → 𝑃 (𝑇 𝑛 |ℎ 1 ). Since ℎ 1 makes 𝑇 potentially trustworthy, as 𝑛 → ∞, P 𝑛 (𝑇 ) = 1.

To show (2), suppose

ℎ 1 is PWMC for 𝑇 . Given that 𝑇 ̸ |= 𝑇 ′ , ℎ 𝑖 (𝑇 ′ 𝑖 ) = 1 -ℎ 𝑖 (𝑇 𝑖 )
for each 𝑖, and the expected probability of 𝑇 ′ will decrease strictly monotonically over n, as P 𝑛 (ℎ 1 ) → 1. So as 𝑛 → ∞, P 𝑛 (𝑇 ′ ) = 0. □ Corollary 2. Let the priors on ℎ 𝑖 ∈ H, ℎ 𝑖 ≠ ℎ 1 be as in Proposition 1. Suppose in addition, 𝑇 |= ¬𝑇 ′ , then:

As 𝑛 → ∞, P 𝑛 (𝑇 ′ ) → 0 (2)
Note that our agent may have many evaluation hypotheses and the result of Proposition 3 is unchanged. Crucially f has updated his beliefs only on 𝑇 due to 𝑔. This matches our intuitions about what agents actually do. As long as the codependence between background beliefs and bodies of evidence holds and certain bodies of evidence are supported more than others, belief in some bodies of evidence 𝑇 ∈ T will be strengthened, while belief in bodies of evidence in conflict with 𝑇 or just different from 𝑇 will be weakened. Importantly, this can happen merely by 𝑇 𝑖 repeating content already in 𝑇 𝑘 for 𝑖 > 𝑘. Such repetitions of content are commonplace on social media sites and news sites that broadcast continuously. In addition, the assumption of a PWMC evaluation hypothesis is rather mild; it reflects an agent's mistrust of bodies of evidence other than the ones he relies on-a rather common situation.

Proposition 3 impacts the marginalization of new data, because if its assumptions are met, as P 𝑛 (𝑇 ′ ) → 0, f discounts evidence from 𝑇 ′ , despite the presence of evaluation hypotheses supporting 𝑇 ′ .

Proposition 4. Suppose evidence 𝜙 such that 𝑇 ′ |= 𝜙, 𝑇 ̸ |= 𝜙 and 𝑇 ,𝑇 ′ , and f 's evaluation hypotheses are as in Proposition 3 and f conforms to Bayesian learning. Then:

As 𝑛 → ∞, P 𝑛 (𝜙) → 0.
Since f conforms to Bayesian learning, the marginal probability for 𝜙 is based on Equation 3 and the set of hypotheses ℎ 𝑖 in Equation 3 is the set H that for f pronounce on testimony that mentions or asserts 𝜙. By Proposition 1, as 𝑛 → ∞, P 𝑛 (ℎ 1 ) → 1. By Proposition 3, P 𝑛 ((𝑇 ′ ℎ 1 ) → 0. But for all other ℎ 𝑘 such that ℎ 𝑘 (𝑇 ′ ) ≠ 0, by Proposition 1 again, as 𝑛 → ∞, P 𝑛 (ℎ 𝑘 ) → 0. But then P 𝑛 (𝜙 |ℎ 𝑖 , B) → 0 for all relevant ℎ 𝑖 . Given Equation 3, the result follows. □ In this situation, f assigns no credence to 𝜙. The prior beliefs of f may so limit the alternative hypotheses ℎ 𝑖 such that even an actual fact 𝜙 will have a marginal probability of 0; f will discount 𝜙 completely. Now consider general learning in this situation, defined in Wolpert's 2018 extended Bayesian framework via Bayes's formula.

𝑃 (ℎ|𝑥, B) = 𝑃 (𝑥 |ℎ, B).𝑃 (ℎ|B) 𝑖=𝑘 𝑖=1 𝑃 (𝑥 |ℎ 𝑖 , B).𝑃 (ℎ 𝑖 |B) (7)
To learn a hypothesis 𝜓 about some event 𝑒, f 's estimation of 𝜓 at some stage 𝑃 𝑛 (𝜓 ) based on her evidence should be closer to the objective or ideal assignment (posterior) 𝜓 𝑝 , than to her prior probability for 𝜓 , 𝑃 0 (𝜓 ). Similarly for marginal probabilities: P 𝑛 (𝑥) should track 𝑥 𝑝 , the posterior of 𝑥, given a random sampling of 𝑋 . We consider loss functions L (P 𝑛 (𝜓 ),𝜓 𝑝 ) and L (P 𝑛 (𝑥), 𝑥 𝑝 ). The greater divergence between the ideal posterior probability and the Bayesian subjective estimation of that probability, the worse will be the score for f 's learning. We say that f cannot learn 𝜓 if her updates do not eventually decrease loss; i.e. we cannot show 𝑙𝑖𝑚 𝑛→∞ L (P 𝑛 (𝜓 ),𝜓 𝑝 ) < L (P 0 (𝜓 ),𝜓 𝑝 ).

Proposition 5. Suppose f is a Bayesian learner with evaluation hypotheses and testimony 𝑇 , 𝑇 ′ as in Proposition 3. Let 𝜓 be a new hypothesis with all evidence 𝑒 confirming 𝜓 such that 𝑇 ′ |= 𝑒. Then f is incapable of learning 𝜓 .

Consider 𝑒 such that 𝑇 ′ |= 𝑒 and e confirms 𝜓 . So the true posterior 𝑃 𝑝 (𝜓 |𝑒) > 𝑃 (𝜓 ), with 𝑃 (𝜓 ) the prior on 𝜓 . Suppose f 's evaluation hypotheses and probabilities have been updated via 𝑇 as in Proposition 3 and let that give the "prior" probability for the new hypothesis 𝜓 . By Proposition 4, as 𝑛 → ∞, P 𝑛 (𝑒) → 0. In the limit, Bayesian learning as specified by equation ( 4) simply isn't defined when P 𝑛 (𝑒) = 0. So assuming 𝑒 is discounted as evidence in updating, we set 𝑃 (𝜓 |𝑒,𝑇 𝑛 ) = 𝑃 (𝜓 |𝑇 𝑛 ). But this is just 𝑃 0 (𝜓 ), f 's prior on 𝜓 . It follows that as 𝑙𝑖𝑚 𝑛→∞ L (P 𝑛 (𝜓 ),𝜓 𝑝 ) ≮ L (P 0 (𝜓 ),𝜓 𝑝 ). □

Proposition 5 is a formal statement of IB in a first order setting. It shows that under certain conditions, f will be incapable of learning any hypothesis that involves a dependence on testimony not in 𝑇 , upon which f has formed his beliefs. f is interpretively blind to any possibilities outside of 𝑇 .

IB IN HIERARCHICAL BAYESIAN LEARNING

It's not unreasonable to rule out new evidence from unreliable testimony, provided the assignment of one's evaluation hypotheses to the testimony is reasonable. But nothing in our discussion above forces the evaluation hypotheses to be be reasonable. Without any constraints, f 's evaluation hypotheses may rule out evidence that is completely grounded in reality and comes from testimony that an ideal rational agent would trust.

To solve this problem, we need to correct the background beliefs B. Ideally, a rational agent should control for the biases in testimony by consulting several different bodies of testimony. However, B cannot be corrected itself by evidence, because that evidence is already discounted if it conflicts with B. Very clearly, background beliefs can be a source of bad epistemic biases, and they can prevent straightforward corrections to improve one's beliefs as Bayesian learning would have us do.

Hierarchical Bayesian models were designed to address this problem [START_REF] Gelman | Bayesian data analysis[END_REF]. In hierarchical Bayesian models, a Bayesian learning model like the one we have discussed in Section 3 has certain parameters; the one parameter we have is our evaluation hypotheses providing the reliability of testimony. At a second level of the hierarchy, we could have a Bayesian learning model concerning evaluation hypotheses, in which we could detail factors that would allow us to estimate reliably the accuracy of an evaluation hypothesis. Abstractly, we would have evaluation hypotheses about evaluation hypotheses that would discuss factors like the consistency or the predictive accuracy of a testimony source, or the extent to which testimony from other sources agrees with its content. One could also require a longer or more thorough exploration of the data about the phenomenon before the agent's restricting himself to a small subset for exploitation (once again an application of the work in [Cesa-Bianchi and Lugosi 2006]). All of these ideas and more have been proposed.

Simply requiring evaluation hypotheses that obey exogenous constraints, however, begs the question of why f should accept them. In fact, the interdependence of testimony, new information and background beliefs can make the resort to higher order parameters to resolve IB a failure because a body of dynamic testimony 𝑇 , when directed by a conversational agent for the purposes of persuading and keeping his audience, can react to and attack not only a conflicting body of testimony 𝑇 ′ but also sources supporting it. This behavior provides arguments for or against not only first order evaluation hypotheses, as we've seen with the notion of consistency, but also for higher order functions and in fact sequences of evaluation hypotheses.

To formalize this picture, we assume a hierarchy of sets of evaluation hypotheses where,

ℎ 𝑛+1 : ℎ 𝑛 → [0, 1], for ℎ 𝑛+1 ∈ H 𝑛+1 .
Hypotheses at level 𝑛 + 1 are related to evaluation hypotheses at level 𝑛 via rationality.

Definition 5. A set of sets of evaluation hypotheses

H = {H 1 , H 2 , ..., H 𝑛 } is rational iff for all 𝑚 < 𝑛, ℎ 𝑚 𝑘 ∈ H 𝑚 , 𝑃 (ℎ 𝑚 𝑘 ) = 𝜆 ℎ 𝑗 ∈H 𝑚+1 𝑃 (ℎ 𝑚+1 𝑗 ).𝑃 (ℎ 𝑚 𝑘 |ℎ 𝑚+1 𝑗
) for some normalizing factor 𝜆.

A rational set of sets of evaluation hypotheses is one in which the probability of evaluation hypotheses at one level reflects what higher levels say about it. We will assume rationality of H .

Given rational H = {H 1 , H 2 , ..., H 𝑛 }, we now lift our notions of support to sequences. We define a H 𝑛 sequence 𝜎 ∈ 𝑛 𝑖=1 H 𝑖 of consistent evaluation hypotheses to support 𝑇 (𝜎 | ≈𝑇 ) (or that make 𝑇 potentially trustworthy-𝜎 |= 𝑇 )

iff the H 1 element ℎ 1 𝜎 of 𝜎 is positive sensitive to 𝑇 (makes 𝑇 potentially trustworthy) and every element of 𝜎 has non-0 probability given H . Conversely, we say that 𝑇 | ≈𝜎 iff for each element ℎ 𝑖 𝜎 of 𝜎 𝑃 (ℎ 𝑖 𝜎 |𝑇 𝑗 ) is eventually monotone increasing for all stages 𝑇 𝑗 . We note that 𝜎 | ≈𝑇 → 𝑇 | ≈𝜎.

Let 𝜎 𝑘 be the subsequence of 𝜎 such that 𝜎 𝑘 = 𝜎 ↾ ( 𝑛 𝑖=𝑘 H 𝑖 ). For ℎ 1 ∈ H 1 , 𝜎 2 (ℎ 1 ) signifies the support ℎ 1 receives from the higher order functions in 𝜎 via Definition 5. Definition 6. An H 𝑛 sequence 𝜎 1 undercuts 𝑇 iff for any

𝜎 𝑖 ∈ H 𝑖 , if 𝜎 𝑖 | ≈𝑇 , then 𝜎 𝑖+1 1 (ℎ 𝑖 𝜎 ) = 1 -𝑃 (𝑇 |ℎ 1 𝜎 ) Definition 7. 𝜙 disagrees with 𝑇 ′ just in case 𝑃 (𝑇 |𝜙) < 𝑃 (𝑇 ). Definition 8. Given H = {H 1 , H 2 , ..., H 𝑛 } 𝑇 attacks 𝑇 ′ iff (i) if 𝜎 | ≈𝑇 , then 𝑃 (𝑇 ′ |ℎ 1 𝜎 ) = 1 -𝑃 (𝑇 |ℎ 1 𝜎 ) and (ii) for any H 𝑚 sequence 𝜎 𝑚 , 𝑚 < 𝑛 if 𝜎 𝑚 | ≈𝑇 ′ , ∃ℎ 𝑚+1 ∈ H 𝑚+1 such that (𝑃 (ℎ 𝑚+1 |𝑇 ) > .5 and ℎ 𝑚+1 (𝜎) = 0). Definition 9. 𝑇 is argumentatively complete iff: (i) (𝑇 ′ |= 𝜙 and Disagree(𝜙,𝑇 )) → Attack(𝑇 ,𝑇 ′ ); (ii) If 𝑇 𝑛 ̸ |= 𝜙 but 𝑃 (𝑇 𝑛 |𝜙) ≥ 𝑃 (𝑇 𝑛 ), then 𝑇 𝑛+1 |= 𝜙. (iii) for any T undercutting sequence 𝜎 𝑚 , ∃ℎ 𝑚+1 ∈ H 𝑚+1 such that (𝑃 (ℎ 𝑚+1 |𝑇 ) > .5, ℎ 𝑚+1 (ℎ 𝑚 𝜎 ) = 0) (iv)
𝜎 ) = 1 -𝑃 (𝑇 |ℎ 1 𝜎 ). So ℎ 1 𝜎 is PWMC. □ Proposition 7.
Let 𝑇 be argumentatively complete with a rational set of evaluation hypotheses H with ℎ 1 ∈H 1 𝑃 (ℎ 1 ) ≠ 0 and probabilities updated on 𝑇 .

As 𝑛 → ∞, P 𝑛 (𝑇 ) → 1.

(1)

In addition suppose there is a 𝑇 ′ ⊈ 𝑇 .

As 𝑛 → ∞, P 𝑛 (𝑇 ′ ) → 0.

(2)

We first show (1). Since 𝑇 is argumentatively complete, ∃𝜎 ∈ H such that ℎ 1 𝜎 |= 𝑇 . We need to show that for some such ℎ 1 𝜎 , 𝑃 (ℎ 1 𝜎 ) ≠ 0 relative to H . Suppose that 𝑃 (ℎ 1 𝜎 ) = 0, for all ℎ 1 𝜎 such that ℎ 1 𝜎 |= 𝑇 . By rationality, for each such

ℎ 1 𝜎 , 𝑃 (ℎ 1 𝜎 ) = 𝜆 ℎ 𝑗 ∈H 2 𝑃 (ℎ 2 𝑗 ).𝑃 (ℎ 1 𝜎 |ℎ 2 𝑗 ) = 0.
Thus, all the non-0 probability mass of H falls on 𝑇 undercutting sequences 𝜎 𝑖 . But for each such 𝑇 undercutting 𝜎 𝑖 of length 𝑚, since 𝑇 is argumentatively complete, there is an evaluation hypothesis ℎ 𝑚+1 supported by 𝑇 such that 𝑃 (𝜎 𝑖 |ℎ 𝑚+1 ) = 0. Since H has only finitely many levels, at some level 𝑘 all T undercutting sequences 𝜎 𝑗 get 0 probability. This, together with the fact that ℎ 1 ∈H 1 𝑃 (ℎ 1 ) ≠ 0, contradicts the assumption that 𝑃 (ℎ 1 𝜎 ) = 0. Since 𝑇 is argumentatively complete, any sequence supporting any ℎ 1 where 𝑃 (𝑇 |ℎ 1 ) < 𝑃 (𝑇 ) will eventually get probability 0; so {ℎ 1 :𝑃 (𝑇 |ℎ 1 ) ≥𝑃 (𝑇 ) } 𝑃 (ℎ 1 ) = ℎ 1 ∈H 1 𝑃 (ℎ 1 ). Moreover, as P 𝑛 gets updated, as 𝑛 → ∞, the ℎ 1 such that P 𝑛 (𝑇 |ℎ 1 ) ≥ P 𝑛 (𝑇 ) turn out to be such that ℎ 1 |= 𝑇 . The conditions on first order evaluation hypotheses in H of Proposition 1 are now met. By Propositions 1 and 3, as 𝑛 → ∞, P 𝑛 (ℎ 1 𝜎 ) → 1, P 𝑛 (ℎ 1 𝑖 ) → 0 for 𝑖 ≠ 1. By Proposition 3, P 𝑛 (𝑇 ) → 1.

To show (2), by Proposition 6, ℎ 1 𝜎 is also PWMC for 𝑇 . As 𝑛 → ∞, since P 𝑛 (ℎ 1 𝜎 ) → 1, P 𝑛 (𝑇 ′ ) → 0. □ Proposition 8. Suppose 𝑇 is argumentatively complete. Let f be a hierarchical Bayesian learner whose evaluation hypotheses are rational and are updated on 𝑇 . If 𝑇 ′ ⊊ 𝑇 such confirms a hypothesis ℎ that 𝑇 does not, then f is incapable of learning ℎ.

Claim 2 of Proposition 7 shows that P 𝑛 (𝑇 ′ ) → 0. Then apply Proposition 5. □ Argumentatively complete testimony reduces the case of higher order Bayesian frameworks to our first order setting for IB. What is troubling about IB is that our learner f may hold onto an argumentatively complete 𝑇 regardless of how inadequate it is in the eyes of others or standard epistemic criteria; an argumentatively complete theory will always eventually find a reply to any attack or any doubt f might acquire.

Argumentatively complete testimony isn't just an abstract concept; many social media and news sites already approximate this condition. Outlets like NewsMax or One Amercan News Network that have a particular political bias will attack the credibility of stories from other bodies of testimony that have gone against a narrative they were and are promoting; darker conspiracy spinning websites like those promoting QAnon will attack arguments against their theories once they become aware of them.2 In anecdotal support of our claims, consider Michelle Goldberg's "It's Marjorie Taylor Greene's Party Now" New York Times, 2/2/2021) description of a group in IB: "American conservatism -particularly its evangelical strain -has fostered derangement in its ranks for decades, insisting that no source of information outside its own self-reinforcing ideological bubble is trustworthy. "

A crucial component of argumentatively complete testimony 𝑇 is that it promotes evaluation hypotheses that both make 𝑇 eventually trustworthy but also PWMC for 𝑇 . Sources like the New York Times embody this in their slogan all the news that's fit to print, but there's a commercial reason for this outcome; news sites and social media are out to capture market share and so they naturally promote themselves as accurate and complete at least in a certain domain.

The nature of contemporary testimony leads agents naturally to a situation where IB occurs.

But how does argumentative completeness relate to FB algorithms? We suspect that FB algorithms don't entail argumentative completeness, but they facilitate it like they do for PWMC. An FB algorithm 𝑔's choosing testimony 𝑇 𝑗 for f should ideally detract from other possible additions to f 's evidence that might lead f to find other testimony besides that provided by 𝑔. Since the goal of providing 𝑔 is to reinforce 𝑇 and the evaluation hypotheses supporting it, a possible consequence is for 𝑔 to provide testimony that not only supports 𝑇 but attacks any evidence that might detract from 𝑇 .

COMPARISONS TO PRIOR WORK

IB is an epistemological bias that we briefly described in a preliminary abstract [START_REF] Asher | Interpretive blindness and the impossibility of learning from testimony[END_REF]. IB is clearly related to confirmation bias [START_REF] Lord | Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence[END_REF][START_REF] Raymond S Nickerson | Confirmation bias: A ubiquitous phenomenon in many guises[END_REF][START_REF] Oswald | Cognitive illusions: A handbook on fallacies and biases in thinking, judgement and memory[END_REF], in which agents interpret new evidence in a way that confirms their beliefs, and to the framing biases of [Tversky andKahneman 1975, 1985]. People tend to see in the evidence what they believe. These forms of bias, however, concern how beliefs and bias influence interpretation, painting only part of the picture of IB. [START_REF] Asher | Strategic conversation under imperfect information: epistemic Message Exchange games[END_REF] shows a codependence between beliefs and the interpretation of ambiguous or underspecified elements in a text and postulates a similar circular structure to that which we have exploited for belief and interpretation of evidence in analyzing IB. Further, unlike much of the psychological and philosophical literature which either claims that biases like IB arise from bad epistemic practices or aren't really beliefs at all, or finds epistemologically exogenous justifications for it [START_REF] Cassam | Vice epistemology[END_REF][START_REF] Dardenne | Confirmation Bias as a Social Skill[END_REF][START_REF] Ichino | Non-doxastic conspiracy theories[END_REF] Räikkä 2020], we show how IB is a natural outcome of Bayesian updating, rational resource management and the belief interpretation codependence.

IB is also related to what [START_REF] Hall | Echo chamber: Rush Limbaugh and the conservative media establishment[END_REF][START_REF] Nguyen | Echo chambers and epistemic bubbles[END_REF]] have called echo chambers and epistemic bubbles. Epistemic bubbles are the sort of epistemic structures that result from Proposition 1. Contrary to what [START_REF] Nguyen | Echo chambers and epistemic bubbles[END_REF]] claims, we show that they are not so easy to get rid of. Once the potentially trustworthy hypothesis is sufficiently entrenched, which can happen very quickly as we show in Section 3, simply bringing evidence inconsistent with an accepted body of evidence will not liberate the learner from his IB predicament. Evidence inconsistent with the primary source of our learner's beliefs will be simply rejected, as we show formally in Corollary 2. Following [START_REF] Hall | Echo chamber: Rush Limbaugh and the conservative media establishment[END_REF], [START_REF] Nguyen | Echo chambers and epistemic bubbles[END_REF]] claims that echo chambers are much more problematic epistemically. We have shown this rigorously via Proposition 3 as well as in Propositions in Section 4. As we show below in Section 6 (see Proposition 10), there is no escape from these echo chambers, once certain epistemic principles are admitted to. We have also shown conceptual and formal links between epistemic bubbles and echo chambers. In fact epistemic bubbles lead naturally to echo chambers, as we show in Sections 3 and 4.

IB is also related to so called epistemic bootstrapping [START_REF] Douven | Proper bootstrapping[END_REF][START_REF] Vogel | Epistemic bootstrapping[END_REF][START_REF] Weisberg | Bootstrapping in general[END_REF]]. Epistemic bootstrapping is a phenomenon in which an agent 𝐴 exploits a hypothesis ℎ she is interested in confirming in the very process of confirming ℎ. IB is somewhat different. In IB, 𝐴 has a hypothesis ℎ that confers a certain probability on evidence 𝐸. Suppose ℎ assigns a relatively high probability to 𝐸. As more evidence 𝐸 ′ comes in, however, and 𝐸 ′ tends to confirm 𝐸, then 𝐴's confidence in ℎ should increase. And 𝐴 doesn't just exploit one hypothesis but other hypotheses that may be opposed and that would tend to disconconfirm ℎ. As [START_REF] Douven | Proper bootstrapping[END_REF] argues, bootstrapping is not in itself bad; in fact this is just what scientists do. What matters is whether the process involved in confirming ℎ could disconfirm ℎ. So in principle, the inductive practices that lead to IB aren't easily criticizable as instances of bad epistemic bootstrapping. Most likely unbeknownst to 𝐴, however, the epistemic deck is stacked against her, when an FB algorithm is used to feed her testimony. In principle this is quite a different situation epistemically from a case of bad epistemic bootstrapping.

One can also see IB as a concrete application of work on determining an optimal allocation of resources to the exploration and exploitation of sources [START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF][START_REF] Jeffrey | Switching costs and the Gittins index[END_REF][START_REF] Burnetas | Optimal adaptive policies for Markov decision processes[END_REF]Cesa-Bianchi and Lugosi 2006;[START_REF] Garivier | The KL-UCB Algorithm for Bounded Stochastic Bandits and Beyond[END_REF][START_REF] Leung | Asymptotically efficient adaptive allocation rules[END_REF][START_REF] Whittle | Multi-armed bandits and the Gittins index[END_REF]. It is also related to work on generalization in machine learning. Epistemic biases affect generalization and learning capacity in ways that are still not fully understood [START_REF] Kawaguchi | Generalization in deep learning[END_REF][START_REF] Lampinen | Bayesian approach for neural networks-review and case studies[END_REF][START_REF] Behnam Neyshabur | Exploring generalization in deep learning[END_REF][START_REF] Zhang | Understanding deep learning requires rethinking generalization[END_REF]]. [START_REF] Zhang | Understanding deep learning requires rethinking generalization[END_REF] show that standard techniques in machine learning for promoting good epistemic biases and generalization-training error minimization, regularization techniques like weight decay or dropout, or complexity measures used to minimize generalization error (the difference between training error and test error)-do not necessarily lead to good generalization and test performance. Argumentatively complete testimony 𝑇 incorporates an adversarial attack mechanism against any good epistemic practices that might discount 𝑇 . It's this mechanism that guarantees IB.

The argumentation literature [START_REF] Amgoud | An argumentation-based approach for reasoning about trust in information sources[END_REF][START_REF] Minh | On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games[END_REF] is also relevant to IB. If testimony 𝑇 is argumentatively complete, then 𝑇 always provides a counterargument to an attack against 𝑇 -much like an acceptable argument in [START_REF] Minh | On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games[END_REF]]. In addition, however, an argumentatively complete 𝑇 also supports higher order evaluation hypotheses that support hypotheses that support 𝑇 . There are also important connections to the literature on trust [START_REF] Castelfranchi | Trust theory: A socio-cognitive and computational model[END_REF]; in our set up learning agents trust certain sources over others, and our higher order setting invokes a hierarchy of reasons. Nevertheless, the argumentation and trust-based work of which we are aware is complementary to our approach. An argumentation framework takes a possibly inconsistent belief base and imposes a static constraint on inference in such a setting. Similarly, trust is typically modeled in some sort of static modal framework. By contrast, ME learning games and the whole Bayesian framework are dynamic, with beliefs evolving under evidence and game strategies evolving under agent interaction. It is this dynamic evolution that is crucial to our approach and, we think, to modeling agents and learning. In sum, we are not looking at the problem of consistency, but rather the problems of entrenchment and bias.

THE COMPLEXITY OF IB

IB is a result about learning. IB is a suboptimal but natural outcome of the way contemporary bodies of evidence are set up and how humans interpret them. Given our set up, everything turns on what body of evidence on which to update and with which evaluation hypotheses.

If IB is suboptimal, its effects are still more worrisome, because agents in the grip of IB are often unwilling or incapable of changing their beliefs so as to be able to learn. Of course, our learner might just be happy with 𝑇 ; perhaps he needs no more accurate or more truthful body of testimony. He may not be interested in learning anything beyond what 𝑇 presents him with. In this section, however, we assume a learner who might be interested in learning but has difficulting escaping his IB prison. We assume a rational learner f who updates according to his evaluation hypotheses; so if he has an evaluation hypothesis that confers a high probability on some 𝑇 , he will update on 𝑇 . We've seen that f can get IB when he unduly restricts the bodies of evidence which serve as the basis of update or when he attends to an argumentatively complete testimony. So key to removing IB is to get f to change his hypotheses and consider other evidence that that to which he is wedded.

Anecdotally, we have a lot of evidence that IB is hard to escape3 In general, however, we lack a precise analysis of its difficulty. In this section, we introduce a game theoretic method that shows IB is not only hard to defeat but it can even be hard to detect (leading to self-deception). We will see that the choice of epistemic paradigms is important.

To motivate our approach, consider how an actual conversation might go between our learner f in the grip of IB C. 𝐸 may only add sequences of evaluation hypotheses sanctioned by the Jury. We assume this to be a finite set H 𝐽 . D. Both players may only propose consistent and rational sequences.

E. f has learned from some body of evidence 𝑇 , which is common knowledge. F. f may only not accept a proposal 𝜎 of 𝐸, if he has a reason to do so-i.e., if he has an evaluation hypothesis

ℎ 𝑛+1 ∈ H 𝑛+1 f such that ℎ 𝑛+1 (𝜎) = 0.
We say a sequence 𝜎 ∈ 𝑛 1 H 𝑖 to be positive if for each element ℎ 𝑚+1 and ℎ 𝑚 of 𝜎 ℎ 𝑚+1 (ℎ 𝑚 ) >> 0. A sequence 𝜎 nullifies a sequence 𝜎 1 , if for all 𝑚 and for ℎ 𝑚 1 of 𝜎 1 , the ℎ 𝑚+1 of 𝜎 is such that ℎ 𝑚+1 (ℎ 𝑚 1 ) = 0. We can have two sequences each one nullifying the other. This formally represents an 𝑛 round argument, with each round 𝑗 + 1 offering a counterargument to the argument of round 𝑗. We will say that a hypothesis ℎ 1 is 𝑇 positive if ℎ 1 is positive and

𝑃 (𝑇 |ℎ 1 ) = 1
We now define the moves of a game G, in which we suppose a body of evidence 𝑇 that f has attended to and a body of evidence 𝑇 ′ inconsistent with 𝑇 . 𝐸 plays first, then f then replies. The game ends if f plays ACCEPT, which implies that he adds a hypothesis ℎ 1 * to H 1 f , with a non-0 probability mass and with with high

𝑃 (𝑇 ′ |ℎ 1 * ), where 𝑇 ′ ∪ 𝑇 is inconsistent. (m1) 𝐸 proposes 𝑇 ′ -positive ℎ 1 ∈ H 1 𝐽 to be added to H 1 f . (m2) Suppose at round 𝑘 ≥ 1 of 𝜌 in G 𝐸 has proposed a 𝑇 ′ positive ℎ 1 . At 𝑘 + 1 f may play ACCEPT. (m3) Suppose at round 𝑘 of 𝜌 in G 𝐸 has proposed a a 𝑇 ′ positive ℎ 1 . At 𝑘 + 1 f may play a nullifying ℎ 2 ∈ H 1 f such that ℎ 2 (ℎ 1 ) = 0, if there exists such ℎ 2 ∈ H 2 f .
her proposed sequence of evaluation hypotheses, f can rationally resort to 𝑇 to counter her argument. Extracting someone from higher order IB is thus impossible by purely epistemic means. There is no way of getting someone, even a rational agent, out of higher order IB by purely epistemic arguments, given our assumptions. This pessimistic is borne out empirically: some people in the grip of right wing conspiracy theories in the US were dying of Covid19 in December of 2020 and January 2021 but continued to refuse to believe that it was that disease that was killing them-despite all the evidence and arguments they were given, they refused to let go of an obviously faulty but argumentatively complete 𝑇 .

Of course, people sometimes do change their minds and do escape the grip of argumentatively complete theories, many times for epistemically exogenous reasons. 4 But by challenging one of our assumptions, rational agents can of course also reject IB. The weak link in our argument is assumption 𝐴, the "knowledge first" assumption. Perhaps f should accept evaluation hypotheses even if 𝑇 attacks them. More likely, f should not accept all attacks equally.

Probably, he should be skeptical of any body of evidence 𝑇 that promotes PWMC for 𝑇 and 𝑇 eventually trustworthy evaluation hypotheses while attacking any point of view at variance with it.

We now explore the play between 𝐸 and f in an ME learning game G where Win = P before f has accepted enough of the argumentatively complete 𝑇 to close off learning from alternative bodies of evidence. Suppose 𝑇 is argumentatively complete but comes in stages; if 𝑇 ′ 𝑖 attacks 𝑇 𝑖 , then 𝑇 𝑖+1 but not 𝑇 𝑖 attacks 𝑇 ′ 𝑖 . That is, an argumentatively complete 𝑇 reacts to attacks but does not forsee all attacks in advance. Suppose a set of consistent first order evaluation hypotheses H 1 = {ℎ 1 1 , ℎ 1 2 , ...}, with 𝑃 (ℎ 1 1 ) = .6, 𝑃 (ℎ 1 2 ) = .4, and 𝑃 (𝑇 𝑖 |(ℎ 1 ) = 1 = 𝑃 (𝑇 ′ 𝑖 |ℎ 2 ). Now suppose 𝑇 ′ 1 ∪ 𝑇 1 is inconsistent and 𝐸 proposes ℎ 1 2 since ℎ 1 2 |= 𝑇 ′ 𝑖 . Since the ℎ 1 𝑖 are consistent, 𝑃 (𝑇 1 |ℎ 1 2 ) = 0 = 𝑃 (𝑇 ′ 2 |ℎ 1 1 ). At this point, f could accept 𝐸's proposal under constraint (A), G ends and 𝐸 wins. f will continue to update over stages 𝑇 and 𝑇 ′ with the marginal probabilities 𝑃 (𝑇 𝑖 ) = .6 and 𝑃 (𝑇 ′ 𝑖 ) = .4 remaining stationary. On the other hand, f may decide to wait to see what the next stage 𝑇 2 of 𝑇 brings. As 𝑇 is argumentatively complete, 𝑇 2 will attack 𝑇 ′ 2 , and add a nullifying ℎ 2 ∈ H 2 supported by 𝑇 2 . Should f accept ℎ 2 , the probability of ℎ 1 2 will go to 0 in H . But now suppose we have a constraint, Discount, that discounts any nullifying sequence from 𝑇 . It would be unreasonable for f to wipe out alternatives in the face of this level of uncertainty; at this stage, 𝑃 (𝑇 2 ) = .6 and 𝑃 (𝑇 ′ 2 ) = .4. Summarizing:

Proposition 11. Suppose an ME learning game G with constraint 𝐴 replaced by Discount and with Win = P and f as described in Proposition 7. 𝐸 then has a winning strategy in 𝐺, and IB does not arise for f .

CONCLUSIONS

Interpretive blindness results from a dynamic, iterative process whereby a learner's background beliefs and biases lead her to update her beliefs based on a body of testimony 𝑇 , and then biases inherent in 𝑇 come back to reinforce her beliefs and her trust in 𝑇 's source(s), further biasing her towards these sources for future updates. We have introduced and formally characterized IB. We have shown that IB can prevent learning even in higher order Bayesian frameworks for learning from argumentatively complete testimony, despite the presence of constraints designed to promote good epistemic practices. We also shown that IB is computationally complex as a co-r.e. set via a game theoretic analysis, and that an agent may rationally remain in IB in the face of epistemic arguments. Our game theoretic analysis can also be extended to cases where the agent falls out of IB but then is a recidivist and becomse a prisoner once more. We leave that for future work. Investigating IB alas is not just an academic enterprise. IB really does happen, with sometimes tragic or dangerous results. We think a careful formal analysis is urgent for society. Finally, we note that while we have focused on IB as a problem for learning from testimony, the problem it raises for learning extends to any case in which we do not have unmediated access to ground truth and our data is "theory laden" [START_REF] Russell | Patterns of discovery: An inquiry into the conceptual foundations of science[END_REF]].

  How general are the results in Propositions 7 and 8? PAC, Statistical Physics Framework, VC, and supervised Bayesian learning are four different instantiations of Wolpert's extended Bayesian formalism that we use [Wolpert 2018]. Thus our results should hold for other frameworks.

  ∃H 𝑛 sequence 𝜎 such that 𝜎 |= 𝑇 . Proposition 6. If 𝑇 is argumentatively complete, then there is an ℎ such that ℎ is PWMC for 𝑇 . Assume that 𝑇 is argumentatively complete. Then ∃𝜎 ∈ H 𝑛 such that 𝜎 |= 𝑇 . Since 𝜎 |= 𝑇 , 𝜎 | ≈𝑇 . Now assume 𝑇 𝑛 ̸ |= 𝜙 for some 𝜙 for all stages 𝑛. But then 𝑃 (𝑇 𝑛 |𝜙) < 𝑃 (𝑇 𝑛 ) for each stage 𝑇 𝑛 of 𝑇 . But then 𝑇 and 𝜙 disagree and so 𝑇 attacks 𝜙. By the definition of attack, 𝑃 (𝜙 |ℎ 1

  and a person 𝐸 who wants to correct his problem. 𝐸 might question f 's reasons for believing some proposition 𝜙; she might try getting f to consider different bodies of evidence 𝑇 ′ that might disconfirm 𝜙. f might accept 𝑇 ′ or he might argue against it-by providing, for example, reasons why 𝑇 ′ is not trustworthy or why the arguments supporting 𝑇 ′ are faulty. 𝐸 might attack those arguments or provide new evaluation hypotheses for consideration. Our ME games formalize this interaction.In an ME learning game G = (𝑉 ∞ , Win), the two players, our investigator 𝐸 and our Bayesian learner f , construct a larger "conversation" by consecutively playing finite strings from the vocabulary 𝑉 . Win specifies the winning condition of 𝐸. The vocabulary 𝑉 of an ME learning game G consists of sequences of evaluation hypotheses (with some abuse of notation, we'll take a single ℎ 𝑛 𝑗 to be a one place sequence) and a predicate ACCEPT. ACCEPT means that f accepts the last suggestion by 𝐸 and confers upon it a non zero probability mass. Our ME learning games are subject to several constraints.A. Knowledge first[START_REF] Williamson | Knowledge and its Limits[END_REF]]: this is a constraint from formal epistemology; f only adds a sequence 𝜎 to 𝑛 1 H 𝑖 f for H 𝑖 f 𝑖-th level evaluation hypotheses in H f if he has no argument that attacks 𝜎-in other words no evaluation hypothesis ℎ 𝑛+1 ∈ H 𝑛+1 f such that ℎ 𝑛+1 (𝜎) = 0. B. The Jury in an ME learning game is epistemologically competent; i.e. it sanctions only evaluation hypotheses that advance learning.

[Kelly

2008] describes an informal description of this phenomenon.

FAccT '22, June 21-24, 2022, Seoul, Republic of Korea

See Stuart A. Thompson, "Three Weeks Inside a Pro-Trump QAnon Chat Room" NY Times, Jan 26, 2021).

See Thompson, cited in note 2.

For instance, the satisfaction they derived from belonging to a particular community supported by a particular body of testimony might and does wane.

ACKNOWLEDGMENTS

This work was supported by the AI Interdisciplinary Institute ANITI, funded by the French program "Investing for the Future -PIA3" under Grant agreement no. ANR-19-PI3A-0004, the ANR PRCI grant SLANT (no. ANR-19-CE23-0022), and also by the ICT 38 H2020 grant COALA (no. 957296).

(m4) Suppose 𝐸 has proposed a positive sequence 𝜎 of length 𝑚 and with ℎ 1 𝜎 𝑇 ′ positive at round 𝑘 of 𝜌 in G. At round 𝑘 + 1 f may respond with sequence of length 𝑚 + 1 nullifying 𝜎.

(m5) Suppose at round 𝑘 of 𝜌 of G, f has proposed an m-length sequence 𝜎 nullifying a positive 𝜎 * proposed by 𝐸, with 𝑇 ′ positive ℎ 1 𝜎 * . 𝐸 may respond at round 𝑘 + 1 of 𝜌 with a positive 𝑚 + 1 length sequence ℎ 𝑚+1 * .𝜎 * , with ℎ 𝑚+1 1 (ℎ 𝑚 ) ≠ 1 for ℎ 𝑚 in 𝜎.

(m6) Suppose at round 𝑘 of 𝜌 in G, 𝐸 has proposed a positive sequence 𝜎 of length 𝑚 and with ℎ 1 𝜎 𝑇 ′ positive. At round 𝑘 + 1 f may play ACCEPT, which implies that he adds 𝜎 to 𝑛 1 H 𝑖 f .

We note that if move (m6) occurs, f assigns ℎ 1 * and 𝑇 ′ a non-0 probability mass and updates with evidence 𝑇 ′ , which makes the ACCEPT move coherent.

Suppose that in an ME learning game G, 𝐸's winning condition is simply to discover that f is interpretively blind, if he is. Call this condition 𝐼𝐵. We establish the complexity of 𝐸's attempt to achieve 𝐼𝐵. The first order case with a finite H where the game is restricted to moves m1,m2,m3, is rather trivial. More interesting is the case of an ME learning game G = (𝑉 ∞ , Win) with Win = 𝐼𝐵 and in which 𝐸 and f play higher order evaluation hypotheses. Proposition 9. Suppose an ME learning game G = (𝑉 ∞ , Win) with Win = 𝐼𝐵 in which f plays moves described in (m4)-(m7). Then f is not interpretively blind iff play stops at some finite ordinal 𝑛.

Suppose that in the play of G, f accepts at some level 𝑛 to add the sequence of evaluation hypotheses proposed by 𝐸.

Then by the construction of the sequence and the requirement of coherence (constraint D), this confers upon some evaluation hypothesis 𝑠 * 1 a non zero probability such that 𝑃 (𝑇 ′ |ℎ 1 * ) = 1, where 𝑇 ′ is incompatible with the body of evidence 𝑇 . By accepting, f will have an evaluation hypothesis ℎ 1 * with non zero probability such that 𝑃 (𝑇 ′ |ℎ 1 * ) = 1, where 𝑇 ′ is incompatible with the body of evidence 𝑇 , which f has proposed as a source of learning (constraint E). Now when f updates his belief in 𝑇 he must do so with respect to ℎ 1 * , and he must now update his confidence in his evaluation hypotheses with respect not only to 𝑇 but also 𝑇 ′ . In that case, 𝑃 (ℎ 1 * |𝑇 𝑛 ,𝑇 ′ 𝑛 ) ↛ 0 and P 𝑛 (𝑇 ′ ) ↛ 0. As a result, f will be able to learn from 𝑇 ′ , and so he is not interpretively blind with respect to 𝑇 .

If there is no stopping point at any finite ordinal, then 𝐸's is never able to get f to accept a 𝑇 ′ positive hypothesis. In which case, f continues to only update on 𝑇 and by Propositions 7 8, f is interpretively blind. □ Suppose 𝐸's winning condition for an ME learning G, is to get f to accept a 𝑇 ′ positive evaluation hypothesis. Call this winning condition for 𝐸 P (for persuasion).

Corollary 3. Suppose that in an ME learning game G with Win = P. The complexity of Win is an R.E. set. If Win = 𝐼𝐵 then Win is co-r.e. or Π 1 in the Borel Hierarchy.

Proposition 10. Suppose an ME learning game G with Win = P and f as described in Proposition 7. Then 𝐸 has no winning strategy in G.

Proposition 8 implies f 's evaluation hypotheses are updated on an argumentatively complete body of evidence 𝑇 . When implemented via an ME game G, the sequence of evaluation hypotheses in Proposition 7 provide a winning strategy for f . Suppose 𝐸 proposes an ℎ 1 supporting 𝑒 that is inconsistent with 𝑇 . Even if 𝐸 generates a suitable sequence of higher order 𝑇 ′ positive evaluation hypotheses ℎ 1 , ℎ 2 , ℎ 3 , . . ., given Constraint A above, f will only accept an evaluation hypothesis if he has no argument against it. But as 𝑇 will eventually supply such an argument, f can always counter 𝐸's proposals. So she has no winning strategy. □ Not only is IB computationally complex (Corollary 3 shows it is not computable), Proposition 10 shows formally that even if 𝐸 has rationally compelling arguments to show that f is better off (his payoff or reward is higher) in accepting