text:
Bayes' theorem

In probability theory and statistics, Bayes' theorem (alternatively Bayes' law or Bayes' rule), named after Thomas Bayes, describes the probability of an event, based on prior knowledge of conditions that might be related to the event. For example, if the risk of developing health problems is known to increase with age, Bayes' theorem allows the risk to an individual of a known age to be assessed more accurately by conditioning it relative to their age, rather than assuming that the individual is typical of the population as a whole.

One of the many applications of Bayes' theorem is Bayesian inference, a particular approach to statistical inference. When applied, the probabilities involved in the theorem may have different probability interpretations. With Bayesian probability interpretation, the theorem expresses how a degree of belief, expressed as a probability, should rationally change to account for the availability of related evidence. Bayesian inference is fundamental to Bayesian statistics. It has been considered to be "to the theory of probability what Pythagoras's theorem is to geometry."

Based on Bayes law both the prevalence of a disease in a given population and the error rate of an infectious disease test have to be taken into account to evaluate the meaning of a positive test result correctly and avoid the base-rate fallacy.

History
Bayes' theorem is named after the Reverend Thomas Bayes, also a statistician and philosopher. Bayes used conditional probability to provide an algorithm (his Proposition 9) that uses evidence to calculate limits on an unknown parameter. His work was published in 1763 as An Essay Towards Solving a Problem in the Doctrine of Chances. Bayes studied how to compute a distribution for the probability parameter of a binomial distribution (in modern terminology). On Bayes's death his family transferred his papers to a friend, the minister, philosopher, and mathematician Richard Price.

Over two years, Richard Price significantly edited the unpublished manuscript, before sending it to a friend who read it aloud at the Royal Society on 23 December 1763. Price edited Bayes's major work "An Essay Towards Solving a Problem in the Doctrine of Chances" (1763), which appeared in Philosophical Transactions, and contains Bayes' theorem. Price wrote an introduction to the paper which provides some of the philosophical basis of Bayesian statistics and chose one of the two solutions offered by Bayes. In 1765, Price was elected a Fellow of the Royal Society in recognition of his work on the legacy of Bayes. On 27 April a letter sent to his friend Benjamin Franklin was read out at the Royal Society, and later published, where Price applies this work to population and computing 'life-annuities'.

Independently of Bayes, Pierre-Simon Laplace in 1774, and later in his 1812 Théorie analytique des probabilités, used conditional probability to formulate the relation of an updated posterior probability from a prior probability, given evidence. He reproduced and extended Bayes's results in 1774, apparently unaware of Bayes's work. The Bayesian interpretation of probability was developed mainly by Laplace.

About 200 years later, Sir Harold Jeffreys put Bayes's algorithm and Laplace's formulation on an axiomatic basis, writing in a 1973 book that Bayes' theorem "is to the theory of probability what the Pythagorean theorem is to geometry".

Stephen Stigler used a Bayesian argument to conclude that Bayes' theorem was discovered by Nicholas Saunderson, a blind English mathematician, some time before Bayes; that interpretation, however, has been disputed. Martyn Hooper and Sharon McGrayne have argued that Richard Price's contribution was substantial: "By modern standards, we should refer to the Bayes–Price rule. Price discovered Bayes's work, recognized its importance, corrected it, contributed to the article, and found a use for it. The modern convention of employing Bayes's name alone is unfair but so entrenched that anything else makes little sense."

Statement of theorem
Bayes' theorem is stated mathematically as the following equation:

where $$A$$ and $$B$$ are events and $$P(B) \neq 0$$.

 * $$P(A\vert B)$$ is a conditional probability: the probability of event $$A$$ occurring given that $$B$$ is true. It is also called the posterior probability of $$A$$ given $$B$$.
 * $$P(B\vert A)$$ is also a conditional probability: the probability of event $$B$$ occurring given that $$A$$ is true. It can also be interpreted as the likelihood of $$A$$ given a fixed $$B$$ because $$P(B\vert A)=L(A\vert B)$$.
 * $$P(A)$$ and $$P(B)$$ are the probabilities of observing $$A$$ and $$B$$ respectively without any given conditions; they are known as the prior probability and marginal probability.

For events
Bayes' theorem may be derived from the definition of conditional probability:

 * $$P(A\vert B)=\frac{P(A \cap B)}{P(B)}, \text{ if } P(B) \neq 0, $$

where $$P(A \cap B)$$ is the probability of both A and B being true. Similarly,

 * $$P(B\vert A)=\frac{P(A \cap B)}{P(A)}, \text{ if } P(A) \neq 0. $$

Solving for $$P(A \cap B)$$ and substituting into the above expression for $$P(A\vert B)$$ yields Bayes' theorem:

 * $$P(A\vert B) = \frac{P(B\vert A) P(A)}{P(B)}, \text{ if } P(B) \neq 0.$$

For continuous random variables
For two continuous random variables X and Y, Bayes' theorem may be analogously derived from the definition of conditional density:

 * $$f_{X \vert Y=y} (x) = \frac{f_{X,Y}(x,y)}{f_Y(y)} $$
 * $$f_{Y \vert X=x}(y) = \frac{f_{X,Y}(x,y)}{f_X(x)} $$

Therefore,

 * $$f_{X \vert Y=y}(x) = \frac{f_{Y \vert X=x}(y) f_X(x)}{f_Y(y)}.$$

General case
Let $$P_Y^x $$ be the conditional distribution of $$Y$$ given $$X = x$$ and let $$P_X$$ be the distribution of $$X$$. The joint distribution is then $$P_{X,Y} (dx,dy) = P_Y^x (dy) P_X (dx)$$. The conditional distribution $$P_X^y $$ of $$X$$ given $$Y=y$$ is then determined by

$$P_X^y (A) = E (1_A (X) | Y = y)$$

Existence and uniqueness of the needed conditional expectation is a consequence of the Radon–Nikodym theorem. This was formulated by Kolmogorov in his famous book from 1933. Kolmogorov underlines the importance of conditional probability by writing "I wish to call attention to ... and especially the theory of conditional probabilities and conditional expectations ..." in the Preface. The Bayes theorem determines the posterior distribution from the prior distribution. Uniqueness requires continuity assumptions. Bayes' theorem can be generalized to include improper prior distributions such as the uniform distribution on the real line. Modern Markov chain Monte Carlo methods have boosted the importance of Bayes' theorem including cases with improper priors.

Recreational mathematics
Bayes' rule and computing conditional probabilities provide a solution method for a number of popular puzzles, such as the Three Prisoners problem, the Monty Hall problem, the Two Child problem and the Two Envelopes problem.

Drug testing
Suppose, a particular test for whether someone has been using cannabis is 90% sensitive, meaning the true positive rate (TPR) = 0.90. Therefore, it leads to 90% true positive results (correct identification of drug use) for cannabis users.

The test is also 80% specific, meaning true negative rate (TNR) = 0.80. Therefore, the test correctly identifies 80% of non-use for non-users, but also generates 20% false positives, or false positive rate (FPR) = 0.20, for non-users.

Assuming 0.05 prevalence, meaning 5% of people use cannabis, what is the probability that a random person who tests positive is really a cannabis user?

The Positive predictive value (PPV) of a test is the proportion of persons who are actually positive out of all those testing positive, and can be calculated from a sample as:
 * PPV = True positive / Tested positive

If sensitivity, specificity, and prevalence are known, PPV can be calculated using Bayes theorem. Let $$P(\text{User}\vert \text{Positive}) $$ mean "the probability that someone is a cannabis user given that they test positive," which is what is meant by PPV. We can write:

\begin{align} P(\text{User}\vert \text{Positive}) &= \frac{P(\text{Positive}\vert \text{User}) P(\text{User})}{P(\text{Positive})} \\ &= \frac{P(\text{Positive}\vert\text{User}) P(\text{User})}{P(\text{Positive}\vert\text{User}) P(\text{User}) + P(\text{Positive}\vert\text{Non-user}) P(\text{Non-user})} \\[8pt] &= \frac{0.90 \times 0.05}{0.90 \times 0.05 + 0.20 \times 0.95} = \frac{0.045}{0.045 + 0.19} \approx 19\% \end{align}$$

The fact that $$ P(\text{Positive}) = P(\text{Positive}\vert\text{User}) P(\text{User}) + P(\text{Positive}\vert\text{Non-user}) P(\text{Non-user}) $$ is a direct application of the Law of Total Probability. In this case, it says that the probability that someone tests positive is the probability that a user tests positive, times the probability of being a user, plus the probability that a non-user tests positive, times the probability of being a non-user. This is true because the classifications user and non-user form a partition of a set, namely the set of people who take the drug test. This combined with the definition of conditional probability results in the above statement.

In other words, even if someone tests positive, the probability that they are a cannabis user is only 19%—this is because in this group, only 5% of people are users, and most positives are false positives coming from the remaining 95%.

If 1,000 people were tested:
 * 950 are non-users and 190 of them give false positive (0.20 × 950)
 * 50 of them are users and 45 of them give true positive (0.90 × 50)

The 1,000 people thus yields 235 positive tests, of which only 45 are genuine drug users, about 19%.

Sensitivity or specificity
The importance of specificity can be seen by showing that even if sensitivity is raised to 100% and specificity remains at 80%, the probability of someone testing positive really being a cannabis user only rises from 19% to 21%, but if the sensitivity is held at 90% and the specificity is increased to 95%, the probability rises to 49%.

Cancer rate
Even if 100% of patients with pancreatic cancer have a certain symptom, when someone has the same symptom, it does not mean that this person has a 100% chance of getting pancreatic cancer. Assuming the incidence rate of pancreatic cancer is 1/100000, while 10/99999 healthy individuals have the same symptoms worldwide, the probability of having pancreatic cancer given the symptoms is only 9.1%, and the other 90.9% could be "false positives" (that is, falsely said to have cancer; "positive" is a confusing term when, as here, the test gives bad news).

Based on incidence rate, the following table presents the corresponding numbers per 100,000 people. Which can then be used to calculate the probability of having cancer when you have the symptoms:

\begin{align} P(\text{Cancer}|\text{Symptoms}) &= \frac{P(\text{Symptoms}|\text{Cancer}) P(\text{Cancer})}{P(\text{Symptoms})} \\ &= \frac{P(\text{Symptoms}|\text{Cancer}) P(\text{Cancer})}{P(\text{Symptoms}|\text{Cancer}) P(\text{Cancer}) + P(\text{Symptoms}|\text{Non-Cancer}) P(\text{Non-Cancer})} \\[8pt] &= \frac{1 \times 0.00001}{1 \times 0.00001 + (10/99999) \times 0.99999} = \frac1{11} \approx 9.1\% \end{align}$$

Defective item rate
A factory produces items using three machines—A, B, and C—which account for 20%, 30%, and 50% of its output respectively. Of the items produced by machine A, 5% are defective; similarly, 3% of machine B's items and 1% of machine C's are defective. If a randomly selected item is defective, what is the probability it was produced by machine C?

Once again, the answer can be reached without using the formula by applying the conditions to a hypothetical number of cases. For example, if the factory produces 1,000 items, 200 will be produced by Machine A, 300 by Machine B, and 500 by Machine C. Machine A will produce 5% × 200 = 10 defective items, Machine B 3% × 300 = 9, and Machine C 1% × 500 = 5, for a total of 24. Thus, the likelihood that a randomly selected defective item was produced by machine C is 5/24 (~20.83%).

This problem can also be solved using Bayes' theorem: Let Xi denote the event that a randomly chosen item was made by the i th machine (for i = A,B,C). Let Y denote the event that a randomly chosen item is defective. Then, we are given the following information:
 * $$P(X_A) = 0.2, \quad P(X_B) = 0.3, \quad P(X_C) = 0.5.$$

If the item was made by the first machine, then the probability that it is defective is 0.05; that is, P(Y | XA) = 0.05. Overall, we have

 * $$P(Y| X_A) = 0.05, \quad P(Y |X_B) = 0.03, \quad  P(Y| X_C) = 0.01.$$

To answer the original question, we first find P(Y). That can be done in the following way:

 * $$P(Y) = \sum_i P(Y| X_i) P(X_i) = (0.05)(0.2) + (0.03)(0.3) + (0.01)(0.5) = 0.024.$$

Hence, 2.4% of the total output is defective.

We are given that Y has occurred, and we want to calculate the conditional probability of XC. By Bayes' theorem,

 * $$P(X_C|Y) = \frac{P(Y | X_C) P(X_C)}{P(Y)} = \frac{0.01 \cdot 0.50}{0.024} = \frac{5}{24}$$

Given that the item is defective, the probability that it was made by machine C is 5/24. Although machine C produces half of the total output, it produces a much smaller fraction of the defective items. Hence the knowledge that the item selected was defective enables us to replace the prior probability P(XC) = 1/2 by the smaller posterior probability P(XC | Y) = 5/24.

Interpretations
The interpretation of Bayes' rule depends on the interpretation of probability ascribed to the terms. The two predominant interpretations are described below.

Bayesian interpretation
In the Bayesian (or epistemological) interpretation, probability measures a "degree of belief". Bayes' theorem links the degree of belief in a proposition before and after accounting for evidence. For example, suppose it is believed with 50% certainty that a coin is twice as likely to land heads than tails. If the coin is flipped a number of times and the outcomes observed, that degree of belief will probably rise or fall, but might even remain the same, depending on the results. For proposition A and evidence B,
 * P (A), the prior, is the initial degree of belief in A.
 * P (A | B), the posterior, is the degree of belief after incorporating news that B is true.
 * the quotient $P(B | A)⁄P(B)$ represents the support B provides for A.

For more on the application of Bayes' theorem under the Bayesian interpretation of probability, see Bayesian inference.

Frequentist interpretation

In the frequentist interpretation, probability measures a "proportion of outcomes". For example, suppose an experiment is performed many times. P(A) is the proportion of outcomes with property A (the prior) and P(B) is the proportion with property B. P(B | A) is the proportion of outcomes with property B out of outcomes with property A, and P(A | B) is the proportion of those with A out of those with B (the posterior).

The role of Bayes' theorem is best visualized with tree diagrams. The two diagrams partition the same outcomes by A and B in opposite orders, to obtain the inverse probabilities. Bayes' theorem links the different partitionings.

Example

An entomologist spots what might, due to the pattern on its back, be a rare subspecies of beetle. A full 98% of the members of the rare subspecies have the pattern, so P(Pattern | Rare) = 98%. Only 5% of members of the common subspecies have the pattern. The rare subspecies is 0.1% of the total population. How likely is the beetle having the pattern to be rare: what is P(Rare | Pattern)?

From the extended form of Bayes' theorem (since any beetle is either rare or common),

\begin{align} P(\text{Rare} \vert \text{Pattern}) &= \frac{P(\text{Pattern} \vert \text{Rare})P(\text{Rare})} {P(\text{Pattern})}\\ [8pt] &= \frac{P(\text{Pattern}\vert \text{Rare})P(\text{Rare})} {P(\text{Pattern} \vert \text{Rare}) P(\text{Rare}) + P(\text{Pattern}\vert \text{Common})P(\text{Common})}\\ [8pt] &= \frac{0.98 \times 0.001} {0.98 \times 0.001 + 0.05 \times 0.999}\\ [8pt] &\approx 1.9\% \end{align} $$

Simple form
For events A and B, provided that P(B) ≠ 0,

 * $$P(A| B) = \frac{P(B | A) P(A)}{P(B)} . $$

In many applications, for instance in Bayesian inference, the event B is fixed in the discussion, and we wish to consider the impact of its having been observed on our belief in various possible events A. In such a situation the denominator of the last expression, the probability of the given evidence B, is fixed; what we want to vary is A. Bayes' theorem then shows that the posterior probabilities are proportional to the numerator, so the last equation becomes:

 * $$P(A| B) \propto P(A) \cdot P(B| A) .$$

In words, the posterior is proportional to the prior times the likelihood.

If events A1, A2, ..., are mutually exclusive and exhaustive, i.e., one of them is certain to occur but no two can occur together, we can determine the proportionality constant by using the fact that their probabilities must add up to one. For instance, for a given event A, the event A itself and its complement ¬A are exclusive and exhaustive. Denoting the constant of proportionality by c we have

 * $$P(A| B) = c \cdot P(A) \cdot P(B| A) \text{ and } P(\neg A| B) = c \cdot P(\neg A) \cdot P(B| \neg A). $$

Adding these two formulas we deduce that

 * $$ 1 = c \cdot (P(B| A)\cdot P(A) + P(B| \neg A) \cdot P(\neg A)),$$

or

 * $$ c = \frac{1}{P(B| A)\cdot P(A) + P(B| \neg A) \cdot P(\neg A)} = \frac 1 {P(B)}. $$

Alternative form
Another form of Bayes' theorem for two competing statements or hypotheses is:

 * $$P(A| B) = \frac{P(B| A) P(A)}{ P(B| A) P(A) + P(B| \neg A) P(\neg A)}.$$

For an epistemological interpretation:

For proposition A and evidence or background B,

 * $$P(A)$$ is the prior probability, the initial degree of belief in A.
 * $$P(\neg A)$$ is the corresponding initial degree of belief in not-A, that  A is false,  where $$ P(\neg A) =1-P(A) $$
 * $$P(B| A)$$ is the conditional probability or likelihood, the degree of belief in B  given that proposition A is true.
 * $$P(B|\neg A)$$ is the conditional probability or likelihood, the degree of belief in B given that proposition A is false.
 * $$P(A| B)$$ is the posterior probability, the probability of A after taking into account B.

Extended form
Often, for some partition {Aj} of the sample space, the event space is given in terms of P(Aj) and P(B | Aj). It is then useful to compute P(B) using the law of total probability:

$$P(B)=\sum_{j}P(B \cap A_j),$$

Or (using the multiplication rule for conditional probability),

 * $$P(B) = {\sum_j P(B| A_j) P(A_j)},$$
 * $$\Rightarrow P(A_i| B) = \frac{P(B| A_i) P(A_i)}{\sum\limits_j P(B| A_j) P(A_j)}\cdot$$

In the special case where A is a binary variable:

 * $$P(A| B) = \frac{P(B| A) P(A)}{ P(B| A) P(A) + P(B| \neg A) P(\neg A)}\cdot$$

Random variables
Consider a sample space Ω generated by two random variables X and Y with known probability distributions. In principle, Bayes' theorem applies to the events A = {X = x} and B = {Y = y}.

 * $$P( X{=}x | Y {=} y) = \frac{P(Y{=}y | X{=}x) P(X{=}x)}{P(Y{=}y)}$$

However, terms become 0 at points where either variable has finite probability density. To remain useful, Bayes' theorem can be formulated in terms of the relevant densities (see Derivation).

Simple form
If X is continuous and Y is discrete,

 * $$f_{X | Y{=}y}(x) = \frac{P(Y{=}y| X{=}x) f_X(x)}{P(Y{=}y)}$$

where each $$f$$ is a density function.

If X is discrete and Y is continuous,

 * $$ P(X{=}x| Y{=}y) = \frac{f_{Y | X{=}x}(y) P(X{=}x)}{f_Y(y)}.$$

If both X and Y are continuous,

 * $$ f_{X| Y{=}y}(x) = \frac{f_{Y | X{=}x}(y) f_X(x)}{f_Y(y)}.$$

Extended form

A continuous event space is often conceptualized in terms of the numerator terms. It is then useful to eliminate the denominator using the law of total probability. For fY(y), this becomes an integral:

 * $$ f_Y(y) = \int_{-\infty}^\infty f_{Y| X = \xi}(y) f_X(\xi)\,d\xi .$$

Bayes' rule in odds form
Bayes' theorem in odds form is:

 * $$O(A_1:A_2\vert B) = O(A_1:A_2) \cdot \Lambda(A_1:A_2\vert B) $$

where

 * $$\Lambda(A_1:A_2\vert B) = \frac{P(B\vert A_1)}{P(B\vert A_2)}$$

is called the Bayes factor or likelihood ratio. The odds between two events is simply the ratio of the probabilities of the two events. Thus

 * $$O(A_1:A_2) = \frac{P(A_1)}{P(A_2)},$$

 * $$O(A_1:A_2\vert B) = \frac{P(A_1\vert  B)}{P(A_2\vert  B)},$$

Thus, the rule says that the posterior odds are the prior odds times the Bayes factor, or in other words,  the posterior is proportional to the prior times the likelihood.

In the special case that $$A_1 = A$$ and $$A_2 = \neg A$$, one writes $$O(A)=O(A:\neg A) =P(A)/(1-P(A))$$, and uses a similar abbreviation for the Bayes factor and for the conditional odds. The odds on $$A$$ is by definition the odds for and against $$A$$. Bayes' rule can then be written in the abbreviated form

 * $$O(A\vert B) = O(A) \cdot \Lambda(A\vert B) ,$$

or, in words, the posterior odds on $$A$$ equals the prior odds on $$A$$ times the likelihood ratio for $$A$$ given information $$B$$. In short, posterior odds equals prior odds times likelihood ratio.

For example, if a medical test has a sensitivity of 90% and a specificity of 91%, then the positive Bayes factor is $$\Lambda_+ = P(\text{True Positive})/P(\text{False Positive}) = 90\%/(100\%-91\%)=10$$. Now, if the prevalence of this disease is 9.09%, and if we take that as the prior probability, then the prior odds is about 1:10. So after receiving a positive test result, the posterior odds of actually having the disease becomes 1:1, which means that the posterior probability of having the disease is 50%. If a second test is performed in serial testing, and that also turns out to be positive, then the posterior odds of actually having the disease becomes 10:1, which means a posterior probability of about 90.91%. The negative Bayes factor can be calculated to be 91%/(100%-90%)=9.1, so if the second test turns out to be negative, then the posterior odds of actually having the disease is 1:9.1, which means a posterior probability of about 9.9%.

The example above can also be understood with more solid numbers: Assume the patient taking the test is from a group of 1000 people, where 91 of them actually have the disease (prevalence of 9.1%). If all these 1000 people take the medical test, 82 of those with the disease will get a true positive result (sensitivity of 90.1%), 9 of those with the disease will get a false negative result (false negative rate of 9.9%), 827 of those without the disease will get a true negative result (specificity of 91.0%), and 82 of those without the disease will get a false positive result (false positive rate of 9.0%). Before taking any test, the patient's odds for having the disease is 91:909. After receiving a positive result, the patient's odds for having the disease is

 * $$\frac{91}{909}\times\frac{90.1\%}{9.0\%}=\frac{91\times90.1\%}{909\times9.0\%}=1:1$$

which is consistent with the fact that there are 82 true positives and 82 false positives in the group of 1000 people.

Propositional logic
Using $$P(\neg B\vert A)=1-P(B\vert A)$$ twice, one may use Bayes' theorem to also express $$P(\neg B\vert \neg A)$$ in terms of $$P(A\vert B)$$ and without negations:
 * $$P(\neg B\vert \neg A) = 1 - \left(1-P(A\vert B)\right)\frac{P(B)}{P(\neg A)},$$

when $$P(\neg A) = 1 - P(A) \neq 0$$. From this we can read off the inference
 * $$P(A\vert B) = 1\implies P(\neg B\vert \neg A) = 1$$.

In words: If certainly $$B$$ implies $$A$$, we infer that certainly $$\neg A$$ implies $$\neg B$$. Where $$P(B)\neq 0$$, the two implications being certain are equivalent statements. In the probability formulas, the conditional probability $$P(A \vert B)$$ generalizes the logical implication $$B \implies A$$, where now beyond assigning true or false, we assign probability values to statements. The assertion of $$B \implies A$$ is captured by certainty of the conditional, the assertion of $$P(A\vert B) = 1$$. Relating the directions of implication, Bayes' theorem represents a generalization of the contraposition law, which in classical propositional logic can be expressed as:

 * $$(B \implies A)\iff(\neg A \implies \neg B)$$.

In this relation between implications, the positions of $$A$$ resp. $$B$$ get flipped.

The corresponding formula in terms of probability calculus is Bayes' theorem, which in its expanded form involving the prior probability/base rate $$a$$ of only $$A$$, is expressed as:
 * $$P(A \vert B) = P(B \vert A) \frac{a(A)}{P(B\vert A)\,a(A)+P(B \vert \neg A)\,a(\neg A)}.$$

Subjective logic
Bayes' theorem represents a special case of deriving inverted conditional opinions in subjective logic expressed as:

 * $$(\omega^S_{A\tilde{|}B},\omega^S_{A\tilde{|}\lnot B}) = (\omega^S_{B\vert A}, \omega^S_{B\vert\lnot A}) \widetilde{\phi} a_A,$$

where $$\widetilde{\phi}$$ denotes the operator for inverting conditional opinions. The argument $$(\omega^S_{B\vert A},\omega^S_{B\vert\lnot A})$$ denotes a pair of binomial conditional opinions given by source $$S$$, and the argument $$a_{A}$$ denotes the prior probability (aka. the base rate) of $$A$$. The pair of derivative inverted conditional opinions is denoted $$(\omega^S_{A\tilde{|}B},\omega^{S}_{A\tilde{|}\lnot B})$$. The conditional opinion $$\omega^S_{A\vert B}$$ generalizes the probabilistic conditional $$P(A \vert B)$$, i.e. in addition to assigning a probability the source $$S$$ can assign any subjective opinion to the conditional statement $$(A\vert B)$$. A binomial subjective opinion $$\omega^{S}_{A}$$ is the belief in the truth of statement $$A$$ with degrees of epistemic uncertainty, as expressed by source $$S$$. Every subjective opinion has a corresponding projected probability $$P(\omega^{S}_{A})$$. The application of Bayes' theorem to projected probabilities of opinions is a homomorphism, meaning that Bayes' theorem can be expressed in terms of projected probabilities of opinions:

 * $$P(\omega^S_{A \tilde{|} B}) = \frac{P(\omega^S_{B \vert A}) a(A)}{P(\omega^S_{B\vert A}) a(A) + P(\omega^S_{B \vert \lnot A}) a(\lnot A)}. $$

Hence, the subjective Bayes' theorem represents a generalization of Bayes' theorem.

Bayes theorem for 3 events
A version of Bayes' theorem for 3 events results from the addition of a third event $$C$$, with $$P(C)>0,$$ on which all probabilities are conditioned:

 * $$P(A \vert B \cap C) = \frac{P(B \vert A \cap C) \, P(A \vert C)}{P(B \vert C)} $$

Derivation
Using the chain rule
 * $$P(A \cap B \cap C) = P(A \vert B \cap C) \, P(B \vert C) \, P(C)$$

And, on the other hand
 * $$P(A \cap B \cap C) = P(B \cap A \cap C) = P(B \vert A \cap C) \, P(A \vert C) \, P(C) $$

The desired result is obtained by identifying both expressions and solving for $$P(A \vert B \cap C)$$.

Use in genetics
In genetics, Bayes' rule can be used to estimate the probability of an individual having a specific genotype. Many people seek to approximate their chances of being affected by a genetic disease or their likelihood of being a carrier for a recessive gene of interest. A Bayesian analysis can be done based on family history or genetic testing, in order to predict whether an individual will develop a disease or pass one on to their children. Genetic testing and prediction is a common practice among couples who plan to have children but are concerned that they may both be carriers for a disease, especially within communities with low genetic variance.

Using pedigree to calculate probabilities
Example of a Bayesian analysis table for a female individual's risk for a disease based on the knowledge that the disease is present in her siblings but not in her parents or any of her four children. Based solely on the status of the subject's siblings and parents, she is equally likely to be a carrier as to be a non-carrier (this likelihood is denoted by the Prior Hypothesis). However, the probability that the subject's four sons would all be unaffected is 1/16 ($1/2$⋅$1/2$⋅$1/2$⋅$1/2$) if she is a carrier, about 1 if she is a non-carrier (this is the Conditional Probability). The Joint Probability reconciles these two predictions by multiplying them together. The last line (the Posterior Probability) is calculated by dividing the Joint Probability for each hypothesis by the sum of both joint probabilities.

Using genetic test results
Parental genetic testing can detect around 90% of known disease alleles in parents that can lead to carrier or affected status in their child. Cystic fibrosis is a heritable disease caused by an autosomal recessive mutation on the CFTR gene, located on the q arm of chromosome 7.

Bayesian analysis of a female patient with a family history of cystic fibrosis (CF), who has tested negative for CF, demonstrating how this method was used to determine her risk of having a child born with CF:

Because the patient is unaffected, she is either homozygous for the wild-type allele, or heterozygous. To establish prior probabilities, a Punnett square is used, based on the knowledge that neither parent was affected by the disease but both could have been carriers: Given that the patient is unaffected, there are only three possibilities. Within these three, there are two scenarios in which the patient carries the mutant allele. Thus the prior probabilities are $2/3$ and $1/3$.

Next, the patient undergoes genetic testing and tests negative for cystic fibrosis. This test has a 90% detection rate, so the conditional probabilities of a negative test are 1/10 and 1. Finally, the joint and posterior probabilities are calculated as before. After carrying out the same analysis on the patient's male partner (with a negative test result), the chances of their child being affected is equal to the product of the parents' respective posterior probabilities for being carriers times the chances that two carriers will produce an affected offspring ($1/4$).

Genetic testing done in parallel with other risk factor identification
Bayesian analysis can be done using phenotypic information associated with a genetic condition, and when combined with genetic testing this analysis becomes much more complicated. Cystic fibrosis, for example, can be identified in a fetus through an ultrasound looking for an echogenic bowel, meaning one that appears brighter than normal on a scan. This is not a foolproof test, as an echogenic bowel can be present in a perfectly healthy fetus. Parental genetic testing is very influential in this case, where a phenotypic facet can be overly influential in probability calculation. In the case of a fetus with an echogenic bowel, with a mother who has been tested and is known to be a CF carrier, the posterior probability that the fetus actually has the disease is very high (0.64). However, once the father has tested negative for CF, the posterior probability drops significantly (to 0.16).

Risk factor calculation is a powerful tool in genetic counseling and reproductive planning, but it cannot be treated as the only important factor to consider. As above, incomplete testing can yield falsely high probability of carrier status, and testing can be financially inaccessible or unfeasible when a parent is not present.

title: Bayes' theorem


text:
Bayesian inference

Bayesian inference ( or  ) is a method of statistical inference in which Bayes' theorem is used to update the probability for a hypothesis as more evidence or information becomes available. Fundamentally, Bayesian inference uses prior knowledge, in the form of a prior distribution in order to estimate posterior probabilities. Bayesian inference is an important technique in statistics, and especially in mathematical statistics. Bayesian updating is particularly important in the dynamic analysis of a sequence of data. Bayesian inference has found application in a wide range of activities, including science, engineering, philosophy, medicine, sport, and law. In the philosophy of decision theory, Bayesian inference is closely related to subjective probability, often called "Bayesian probability".

Formal explanation
Bayesian inference derives the posterior probability as a consequence of two antecedents: a prior probability and a "likelihood function" derived from a statistical model for the observed data. Bayesian inference computes the posterior probability according to Bayes' theorem: $$P(H \mid E) = \frac{P(E \mid H) \cdot P(H)}{P(E)},$$ where
 * $$H$$ stands for any hypothesis whose probability may be affected by data (called evidence below). Often there are competing hypotheses, and the task is to determine which is the most probable.
 * $$P(H)$$, the prior probability, is the estimate of the probability of the hypothesis $$H$$ before the data $$E$$, the current evidence, is observed.
 * $$E$$, the evidence, corresponds to new data that were not used in computing the prior probability.
 * $$P(H \mid E)$$, the posterior probability, is the probability of $$H$$ given $$E$$, i.e., after $$E$$ is observed. This is what we want to know: the probability of a hypothesis given the observed evidence.
 * $$P(E \mid H)$$ is the probability of observing $$E$$ given $$H$$ and is called the likelihood. As a function of $$E$$ with $$H$$ fixed, it indicates the compatibility of the evidence with the given hypothesis. The likelihood function is a function of the evidence, $$E$$, while the posterior probability is a function of the hypothesis, $$H$$.
 * $$P(E)$$ is sometimes termed the marginal likelihood or "model evidence". This factor is the same for all possible hypotheses being considered (as is evident from the fact that the hypothesis $$H$$ does not appear anywhere in the symbol, unlike for all the other factors) and hence does not factor into determining the relative probabilities of different hypotheses.
 * $$P(E)>0$$ (Else one has $$0/0$$.)

For different values of $$H$$, only the factors $$P(H)$$ and $$P(E \mid H)$$, both in the numerator, affect the value of $$P(H \mid E)$$ – the posterior probability of a hypothesis is proportional to its prior probability (its inherent likeliness) and the newly acquired likelihood (its compatibility with the new observed evidence).

In cases where $$\neg H$$ ("not $$H$$"), the logical negation of $$H$$, is a valid likelihood, Bayes' rule can be rewritten as follows: $$\begin{align} P(H \mid E) &= \frac{P(E \mid H) P(H)}{P(E)} \\ \\ &= \frac{P(E \mid H) P(H)}{P(E \mid H) P(H) + P(E \mid \neg H) P(\neg H)} \\ \\ &= \frac{1}{1 + \left(\frac{1}{P(H)} - 1\right) \frac{P(E \mid \neg H)}{P(E \mid H)} } \\ \end{align}$$ because $$ P(E) = P(E \mid H) P(H) + P(E \mid \neg H) P(\neg H) $$ and $$ P(H) + P(\neg H) = 1 .$$ This focuses attention on the term $$ \left(\tfrac{1}{P(H)} - 1\right) \tfrac{P(E \mid \neg H)}{P(E \mid H)} .$$  If that term is approximately 1, then the probability of the hypothesis given the evidence, $$ P(H \mid E) $$, is about $$\tfrac{1}{2}$$, about 50% likely - equally likely or not likely. If that term is very small, close to zero, then the probability of the hypothesis, given the evidence, $$ P(H \mid E) $$ is close to 1 or the conditional hypothesis is quite likely. If that term is very large, much larger than 1, then the hypothesis, given the evidence, is quite unlikely. If the hypothesis (without consideration of evidence) is unlikely, then $$P(H)$$ is small (but not necessarily astronomically small) and $$\tfrac{1}{P(H)}$$ is much larger than 1 and this term can be approximated as $$\tfrac{P(E \mid \neg H)}{P(E \mid H) \cdot P(H)} $$ and relevant probabilities can be compared directly to each other.

One quick and easy way to remember the equation would be to use rule of multiplication: $$P(E \cap H) = P(E \mid H) P(H) = P(H \mid E) P(E).$$

Alternatives to Bayesian updating
Bayesian updating is widely used and computationally convenient. However, it is not the only updating rule that might be considered rational.

Ian Hacking noted that traditional "Dutch book" arguments did not specify Bayesian updating: they left open the possibility that non-Bayesian updating rules could avoid Dutch books. Hacking wrote: "And neither the Dutch book argument nor any other in the personalist arsenal of proofs of the probability axioms entails the dynamic assumption. Not one entails Bayesianism. So the personalist requires the dynamic assumption to be Bayesian. It is true that in consistency a personalist could abandon the Bayesian model of learning from experience. Salt could lose its savour."

Indeed, there are non-Bayesian updating rules that also avoid Dutch books (as discussed in the literature on "probability kinematics") following the publication of Richard C. Jeffrey's rule, which applies Bayes' rule to the case where the evidence itself is assigned a probability. The additional hypotheses needed to uniquely require Bayesian updating have been deemed to be substantial, complicated, and unsatisfactory.

Inference over exclusive and exhaustive possibilities
If evidence is simultaneously used to update belief over a set of exclusive and exhaustive propositions, Bayesian inference may be thought of as acting on this belief distribution as a whole.

General formulation

Suppose a process is generating independent and identically distributed events $$E_n,\ n = 1, 2, 3, \ldots$$, but the probability distribution is unknown. Let the event space $$\Omega$$ represent the current state of belief for this process. Each model is represented by event $$M_m$$. The conditional probabilities $$P(E_n \mid M_m)$$ are specified to define the models. $$P(M_m)$$ is the degree of belief in $$M_m$$. Before the first inference step, $$\{P(M_m)\}$$ is a set of initial prior probabilities. These must sum to 1, but are otherwise arbitrary.

Suppose that the process is observed to generate $$E \in \{E_n\}$$. For each $$M \in \{M_m\}$$, the prior $$P(M)$$ is updated to the posterior $$P(M \mid E)$$. From Bayes' theorem:

$$P(M \mid E) = \frac{P(E \mid M)}{\sum_m {P(E \mid M_m) P(M_m)}} \cdot P(M).$$

Upon observation of further evidence, this procedure may be repeated.

Multiple observations
For a sequence of independent and identically distributed observations $$\mathbf{E} = (e_1, \dots, e_n)$$, it can be shown by induction that repeated application of the above is equivalent to $$P(M \mid \mathbf{E}) = \frac{P(\mathbf{E} \mid M)}{\sum_m {P(\mathbf{E} \mid M_m) P(M_m)}} \cdot P(M),$$ where $$P(\mathbf{E} \mid M) = \prod_k{P(e_k \mid M)}.$$

Parametric formulation: motivating the formal description
By parameterizing the space of models, the belief in all models may be updated in a single step. The distribution of belief over the model space may then be thought of as a distribution of belief over the parameter space. The distributions in this section are expressed as continuous, represented by probability densities, as this is the usual situation. The technique is, however, equally applicable to discrete distributions.

Let the vector $$\boldsymbol{\theta}$$ span the parameter space. Let the initial prior distribution over $$\boldsymbol{\theta}$$ be $$p(\boldsymbol{\theta} \mid \boldsymbol{\alpha})$$, where $$\boldsymbol{\alpha}$$ is a set of parameters to the prior itself, or hyperparameters. Let $$\mathbf{E} = (e_1, \dots, e_n)$$ be a sequence of independent and identically distributed event observations, where all $$e_i$$ are distributed as $$p(e \mid \boldsymbol{\theta})$$ for some $$\boldsymbol{\theta}$$. Bayes' theorem is applied to find the posterior distribution over $$\boldsymbol{\theta}$$:

$$\begin{align} p(\boldsymbol{\theta} \mid \mathbf{E}, \boldsymbol{\alpha}) &= \frac{p(\mathbf{E} \mid \boldsymbol{\theta}, \boldsymbol{\alpha})}{p(\mathbf{E} \mid \boldsymbol{\alpha})} \cdot p(\boldsymbol{\theta} \mid \boldsymbol{\alpha}) \\ &= \frac{p(\mathbf{E} \mid \boldsymbol{\theta}, \boldsymbol{\alpha})}{\int p(\mathbf{E} \mid \boldsymbol{\theta}, \boldsymbol{\alpha}) p(\boldsymbol{\theta} \mid \boldsymbol{\alpha}) \, d\boldsymbol{\theta}} \cdot p(\boldsymbol{\theta} \mid \boldsymbol{\alpha}), \end{align}$$ where $$p(\mathbf{E} \mid \boldsymbol{\theta}, \boldsymbol{\alpha}) = \prod_k p(e_k \mid \boldsymbol{\theta}).$$

Definitions

 * $$x$$, a data point in general. This may in fact be a vector of values.
 * $$\theta$$, the parameter of the data point's distribution, i.e., $x \sim p(x \mid \theta)$. This may be a vector of parameters.
 * $$\alpha$$, the hyperparameter of the parameter distribution, i.e., $\theta \sim p(\theta \mid \alpha)$. This may be a vector of hyperparameters.
 * $$\mathbf{X}$$ is the sample, a set of $$n$$ observed data points, i.e., $$x_1, \ldots, x_n$$.
 * $$\tilde{x}$$, a new data point whose distribution is to be predicted.

Bayesian inference
= \frac{p(\mathbf{X} \mid \theta,\alpha) p(\theta \mid \alpha)}{p(\mathbf{X} \mid \alpha)} \propto p(\mathbf{X} \mid \theta,\alpha) p(\theta \mid \alpha).$$ This is expressed in words as "posterior is proportional to likelihood times prior", or sometimes as "posterior = likelihood times prior, over evidence". $$P_X^y (A) = E (1_A (X) | Y = y)$$Existence and uniqueness of the needed conditional expectation is a consequence of the Radon–Nikodym theorem. This was formulated by Kolmogorov in his famous book from 1933. Kolmogorov underlines the importance of conditional probability by writing "I wish to call attention to ... and especially the theory of conditional probabilities and conditional expectations ..." in the Preface. The Bayes theorem determines the posterior distribution from the prior distribution. Uniqueness requires continuity assumptions. Bayes' theorem can be generalized to include improper prior distributions such as the uniform distribution on the real line. Modern Markov chain Monte Carlo methods have boosted the importance of Bayes' theorem including cases with improper priors.
 * The prior distribution is the distribution of the parameter(s) before any data is observed, i.e. $$p(\theta \mid \alpha)$$ . The prior distribution might not be easily determined; in such a case, one possibility may be to use the Jeffreys prior to obtain a prior distribution before updating it with newer observations.
 * The sampling distribution is the distribution of the observed data conditional on its parameters, i.e. $p(\mathbf{X} \mid \theta)$. This is also termed the likelihood, especially when viewed as a function of the parameter(s), sometimes written $$\operatorname{L}(\theta  \mid \mathbf{X}) = p(\mathbf{X} \mid \theta)$$.
 * The marginal likelihood (sometimes also termed the evidence) is the distribution of the observed data marginalized over the parameter(s), i.e. $$p(\mathbf{X} \mid \alpha) = \int p(\mathbf{X} \mid \theta) p(\theta \mid \alpha) d\theta.$$ It quantifies the agreement between data and expert opinion, in a geometric sense that can be made precise. If the marginal likelihood is 0 then there is no agreement between the data and expert opinion and Bayes' rule cannot be applied.
 * The posterior distribution is the distribution of the parameter(s) after taking into account the observed data. This is determined by Bayes' rule, which forms the heart of Bayesian inference: $$p(\theta \mid \mathbf{X},\alpha) = \frac{p(\theta,\mathbf{X},\alpha)}{p(\mathbf{X},\alpha)} = \frac{p(\mathbf{X}\mid\theta,\alpha)p(\theta,\alpha)}{p(\mathbf{X}\mid\alpha)p(\alpha)}
 * In practice, for almost all complex Bayesian models used in machine learning, the posterior distribution $$p(\theta \mid \mathbf{X},\alpha)$$ is not obtained in a closed form distribution, mainly because the parameter space for $$\theta$$ can be very high, or the Bayesian model retains certain hierarchical structure formulated from the observations $$\mathbf{X}$$ and parameter $$\theta$$. In such situations, we need to resort to approximation techniques.
 * General case: Let $$P_Y^x $$ be the conditional distribution of $$Y$$ given $$X = x$$ and let $$P_X$$ be the distribution of $$X$$. The joint distribution is then $$P_{X,Y} (dx,dy) = P_Y^x (dy) P_X (dx)$$. The conditional distribution $$P_X^y $$ of $$X$$ given $$Y=y$$ is then determined by

Bayesian prediction

 * The posterior predictive distribution is the distribution of a new data point, marginalized over the posterior: $$p(\tilde{x} \mid \mathbf{X},\alpha) = \int p(\tilde{x} \mid \theta) p(\theta \mid \mathbf{X},\alpha) d\theta$$
 * The prior predictive distribution is the distribution of a new data point, marginalized over the prior: $$p(\tilde{x} \mid \alpha) = \int p(\tilde{x} \mid \theta) p(\theta \mid \alpha) d\theta$$

Bayesian theory calls for the use of the posterior predictive distribution to do predictive inference, i.e., to predict the distribution of a new, unobserved data point. That is, instead of a fixed point as a prediction, a distribution over possible points is returned. Only this way is the entire posterior distribution of the parameter(s) used. By comparison, prediction in frequentist statistics often involves finding an optimum point estimate of the parameter(s)—e.g., by maximum likelihood or maximum a posteriori estimation (MAP)—and then plugging this estimate into the formula for the distribution of a data point. This has the disadvantage that it does not account for any uncertainty in the value of the parameter, and hence will underestimate the variance of the predictive distribution.

In some instances, frequentist statistics can work around this problem. For example, confidence intervals and prediction intervals in frequentist statistics when constructed from a normal distribution with unknown mean and variance are constructed using a Student's t-distribution. This correctly estimates the variance, due to the facts that (1) the average of normally distributed random variables is also normally distributed, and (2) the predictive distribution of a normally distributed data point with unknown mean and variance, using conjugate or uninformative priors, has a Student's t-distribution. In Bayesian statistics, however, the posterior predictive distribution can always be determined exactly—or at least to an arbitrary level of precision when numerical methods are used.

Both types of predictive distributions have the form of a compound probability distribution (as does the marginal likelihood). In fact, if the prior distribution is a conjugate prior, such that the prior and posterior distributions come from the same family, it can be seen that both prior and posterior predictive distributions also come from the same family of compound distributions. The only difference is that the posterior predictive distribution uses the updated values of the hyperparameters (applying the Bayesian update rules given in the conjugate prior article), while the prior predictive distribution uses the values of the hyperparameters that appear in the prior distribution.

Interpretation of factor
$ \frac{P(E \mid M)}{P(E)} > 1 \Rightarrow P(E \mid M) > P(E)$. That is, if the model were true, the evidence would be more likely than is predicted by the current state of belief. The reverse applies for a decrease in belief. If the belief does not change, $ \frac{P(E \mid M)}{P(E)} = 1 \Rightarrow P(E \mid M) = P(E)$. That is, the evidence is independent of the model. If the model were true, the evidence would be exactly as likely as predicted by the current state of belief.

Cromwell's rule
If $$P(M) = 0$$ then $$P(M \mid E) = 0$$. If $$P(M) = 1$$ and $$P(E) > 0$$, then $$P(M|E) = 1$$. This can be interpreted to mean that hard convictions are insensitive to counter-evidence.

The former follows directly from Bayes' theorem. The latter can be derived by applying the first rule to the event "not $$M$$" in place of "$$M$$", yielding "if $$1 - P(M) = 0$$, then $$1 - P(M \mid E) = 0$$", from which the result immediately follows.

Asymptotic behaviour of posterior
Consider the behaviour of a belief distribution as it is updated a large number of times with independent and identically distributed trials. For sufficiently nice prior probabilities, the Bernstein-von Mises theorem gives that in the limit of infinite trials, the posterior converges to a Gaussian distribution independent of the initial prior under some conditions firstly outlined and rigorously proven by Joseph L. Doob in 1948, namely if the random variable in consideration has a finite probability space. The more general results were obtained later by the statistician David A. Freedman who published in two seminal research papers in 1963 and 1965 when and under what circumstances the asymptotic behaviour of posterior is guaranteed. His 1963 paper treats, like Doob (1949), the finite case and comes to a satisfactory conclusion. However, if the random variable has an infinite but countable probability space (i.e., corresponding to a die with infinite many faces) the 1965 paper demonstrates that for a dense subset of priors the Bernstein-von Mises theorem is not applicable. In this case there is almost surely no asymptotic convergence. Later in the 1980s and 1990s Freedman and Persi Diaconis continued to work on the case of infinite countable probability spaces. To summarise, there may be insufficient trials to suppress the effects of the initial choice, and especially for large (but finite) systems the convergence might be very slow.

Conjugate priors
In parameterized form, the prior distribution is often assumed to come from a family of distributions called conjugate priors. The usefulness of a conjugate prior is that the corresponding posterior distribution will be in the same family, and the calculation may be expressed in closed form.

Estimates of parameters and predictions
It is often desired to use a posterior distribution to estimate a parameter or variable. Several methods of Bayesian estimation select measurements of central tendency from the posterior distribution.

For one-dimensional problems, a unique median exists for practical continuous problems. The posterior median is attractive as a robust estimator.

If there exists a finite mean for the posterior distribution, then the posterior mean is a method of estimation. $$\tilde \theta = \operatorname{E}[\theta] = \int \theta \, p(\theta \mid \mathbf{X},\alpha) \, d\theta$$

Taking a value with the greatest probability defines maximum a posteriori (MAP) estimates: $$\{ \theta_{\text{MAP}}\} \subset \arg \max_\theta p(\theta \mid \mathbf{X},\alpha) .$$

There are examples where no maximum is attained, in which case the set of MAP estimates is empty.

There are other methods of estimation that minimize the posterior risk (expected-posterior loss) with respect to a loss function, and these are of interest to statistical decision theory using the sampling distribution ("frequentist statistics").

The posterior predictive distribution of a new observation $$\tilde{x}$$ (that is independent of previous observations) is determined by $$p(\tilde{x}|\mathbf{X},\alpha) = \int p(\tilde{x},\theta \mid \mathbf{X},\alpha) \, d\theta = \int p(\tilde{x} \mid \theta) p(\theta \mid \mathbf{X},\alpha) \, d\theta .$$

Probability of a hypothesis
Suppose there are two full bowls of cookies. Bowl #1 has 10 chocolate chip and 30 plain cookies, while bowl #2 has 20 of each. Our friend Fred picks a bowl at random, and then picks a cookie at random. We may assume there is no reason to believe Fred treats one bowl differently from another, likewise for the cookies. The cookie turns out to be a plain one. How probable is it that Fred picked it out of bowl #1?

Intuitively, it seems clear that the answer should be more than a half, since there are more plain cookies in bowl #1. The precise answer is given by Bayes' theorem. Let $$H_1$$ correspond to bowl #1, and $$H_2$$ to bowl #2. It is given that the bowls are identical from Fred's point of view, thus $$P(H_1)=P(H_2)$$, and the two must add up to 1, so both are equal to 0.5. The event $$E$$ is the observation of a plain cookie. From the contents of the bowls, we know that $$P(E \mid H_1) = 30/40 = 0.75$$ and $$P(E \mid H_2) = 20/40 = 0.5.$$ Bayes' formula then yields $$\begin{align} P(H_1 \mid E) &= \frac{P(E \mid H_1)\,P(H_1)}{P(E \mid H_1)\,P(H_1)\;+\;P(E \mid H_2)\,P(H_2)} \\ \\ \ & = \frac{0.75 \times 0.5}{0.75 \times 0.5 + 0.5 \times 0.5} \\ \\ \ & = 0.6 \end{align}$$

Before we observed the cookie, the probability we assigned for Fred having chosen bowl #1 was the prior probability, $$P(H_1)$$, which was 0.5. After observing the cookie, we must revise the probability to $$P(H_1 \mid E)$$, which is 0.6.

Making a prediction

An archaeologist is working at a site thought to be from the medieval period, between the 11th century to the 16th century. However, it is uncertain exactly when in this period the site was inhabited. Fragments of pottery are found, some of which are glazed and some of which are decorated. It is expected that if the site were inhabited during the early medieval period, then 1% of the pottery would be glazed and 50% of its area decorated, whereas if it had been inhabited in the late medieval period then 81% would be glazed and 5% of its area decorated. How confident can the archaeologist be in the date of inhabitation as fragments are unearthed?

The degree of belief in the continuous variable $$C$$ (century) is to be calculated, with the discrete set of events $$\{GD,G \bar D, \bar G D, \bar G \bar D\}$$ as evidence. Assuming linear variation of glaze and decoration with time, and that these variables are independent,

$$P(E=GD \mid C=c) = (0.01 + \frac{0.81-0.01}{16-11}(c-11))(0.5 - \frac{0.5-0.05}{16-11}(c-11))$$ $$P(E=G \bar D \mid C=c) = (0.01 + \frac{0.81-0.01}{16-11}(c-11))(0.5 + \frac{0.5-0.05}{16-11}(c-11))$$ $$P(E=\bar G D \mid C=c) = ((1-0.01) - \frac{0.81-0.01}{16-11}(c-11))(0.5 - \frac{0.5-0.05}{16-11}(c-11))$$ $$P(E=\bar G \bar D \mid C=c) = ((1-0.01) - \frac{0.81-0.01}{16-11}(c-11))(0.5 + \frac{0.5-0.05}{16-11}(c-11))$$

Assume a uniform prior of $ f_C(c) = 0.2$, and that trials are independent and identically distributed. When a new fragment of type $$e$$ is discovered, Bayes' theorem is applied to update the degree of belief for each $$c$$: $$f_C(c \mid E=e) = \frac{P(E=e \mid C=c)}{P(E=e)}f_C(c) = \frac{P(E=e \mid C=c)}{\int_{11}^{16}{P(E=e \mid C=c)f_C(c)dc}}f_C(c)$$

A computer simulation of the changing belief as 50 fragments are unearthed is shown on the graph. In the simulation, the site was inhabited around 1420, or $$c=15.2$$. By calculating the area under the relevant portion of the graph for 50 trials, the archaeologist can say that there is practically no chance the site was inhabited in the 11th and 12th centuries, about 1% chance that it was inhabited during the 13th century, 63% chance during the 14th century and 36% during the 15th century. The Bernstein-von Mises theorem asserts here the asymptotic convergence to the "true" distribution because the probability space corresponding to the discrete set of events $$\{GD,G \bar D, \bar G D, \bar G \bar D\}$$ is finite (see above section on asymptotic behaviour of the posterior).

In frequentist statistics and decision theory
A decision-theoretic justification of the use of Bayesian inference was given by Abraham Wald, who proved that every unique Bayesian procedure is admissible. Conversely, every admissible statistical procedure is either a Bayesian procedure or a limit of Bayesian procedures.

Wald characterized admissible procedures as Bayesian procedures (and limits of Bayesian procedures), making the Bayesian formalism a central technique in such areas of frequentist inference as parameter estimation, hypothesis testing, and computing confidence intervals. For example:
 * "Under some conditions, all admissible procedures are either Bayes procedures or limits of Bayes procedures (in various senses). These remarkable results, at least in their original form, are due essentially to Wald. They are useful because the property of being Bayes is easier to analyze than admissibility."
 * "In decision theory, a quite general method for proving admissibility consists in exhibiting a procedure as a unique Bayes solution."
 * "In the first chapters of this work, prior distributions with finite support and the corresponding Bayes procedures were used to establish some of the main theorems relating to the comparison of experiments. Bayes procedures with respect to more general prior distributions have played a very important role in the development of statistics, including its asymptotic theory." "There are many problems where a glance at posterior distributions, for suitable priors, yields immediately interesting information. Also, this technique can hardly be avoided in sequential analysis."
 * "A useful fact is that any Bayes decision rule obtained by taking a proper prior over the whole parameter space must be admissible"
 * "An important area of investigation in the development of admissibility ideas has been that of conventional sampling-theory procedures, and many interesting results have been obtained."

Model selection
Bayesian methodology also plays a role in model selection where the aim is to select one model from a set of competing models that represents most closely the underlying process that generated the observed data. In Bayesian model comparison, the model with the highest posterior probability given the data is selected. The posterior probability of a model depends on the evidence, or marginal likelihood, which reflects the probability that the data is generated by the model, and on the prior belief of the model. When two competing models are a priori considered to be equiprobable, the ratio of their posterior probabilities corresponds to the Bayes factor. Since Bayesian model comparison is aimed on selecting the model with the highest posterior probability, this methodology is also referred to as the maximum a posteriori (MAP) selection rule or the MAP probability rule.

Probabilistic programming
While conceptually simple, Bayesian methods can be mathematically and numerically challenging. Probabilistic programming languages (PPLs) implement functions to easily build Bayesian models together with efficient automatic inference methods. This helps separate the model building from the inference, allowing practitioners to focus on their specific problems and leaving PPLs to handle the computational details for them.

Statistical data analysis
See the separate Wikipedia entry on Bayesian statistics, specifically the statistical modeling section in that page.

Computer applications
Bayesian inference has applications in artificial intelligence and expert systems. Bayesian inference techniques have been a fundamental part of computerized pattern recognition techniques since the late 1950s. There is also an ever-growing connection between Bayesian methods and simulation-based Monte Carlo techniques since complex models cannot be processed in closed form by a Bayesian analysis, while a graphical model structure may allow for efficient simulation algorithms like the Gibbs sampling and other Metropolis–Hastings algorithm schemes. Recently Bayesian inference has gained popularity among the phylogenetics community for these reasons; a number of applications allow many demographic and evolutionary parameters to be estimated simultaneously.

As applied to statistical classification, Bayesian inference has been used to develop algorithms for identifying e-mail spam. Applications which make use of Bayesian inference for spam filtering include CRM114, DSPAM, Bogofilter, SpamAssassin, SpamBayes, Mozilla, XEAMS, and others. Spam classification is treated in more detail in the article on the naïve Bayes classifier.

Solomonoff's Inductive inference is the theory of prediction based on observations; for example, predicting the next symbol based upon a given series of symbols. The only assumption is that the environment follows some unknown but computable probability distribution. It is a formal inductive framework that combines two well-studied principles of inductive inference: Bayesian statistics and Occam's Razor. Solomonoff's universal prior probability of any prefix p of a computable sequence x is the sum of the probabilities of all programs (for a universal computer) that compute something starting with p. Given some p and any computable but unknown probability distribution from which x is sampled, the universal prior and Bayes' theorem can be used to predict the yet unseen parts of x in optimal fashion.

Bioinformatics and healthcare applications
Bayesian inference has been applied in different Bioinformatics applications, including differential gene expression analysis. Bayesian inference is also used in a general cancer risk model, called CIRI (Continuous Individualized Risk Index), where serial measurements are incorporated to update a Bayesian model which is primarily built from prior knowledge.

In the courtroom
Bayesian inference can be used by jurors to coherently accumulate the evidence for and against a defendant, and to see whether, in totality, it meets their personal threshold for "beyond a reasonable doubt". Bayes' theorem is applied successively to all evidence presented, with the posterior from one stage becoming the prior for the next. The benefit of a Bayesian approach is that it gives the juror an unbiased, rational mechanism for combining evidence. It may be appropriate to explain Bayes' theorem to jurors in odds form, as betting odds are more widely understood than probabilities. Alternatively, a logarithmic approach, replacing multiplication with addition, might be easier for a jury to handle.

If the existence of the crime is not in doubt, only the identity of the culprit, it has been suggested that the prior should be uniform over the qualifying population. For example, if 1,000 people could have committed the crime, the prior probability of guilt would be 1/1000.

The use of Bayes' theorem by jurors is controversial. In the United Kingdom, a defence expert witness explained Bayes' theorem to the jury in R v Adams. The jury convicted, but the case went to appeal on the basis that no means of accumulating evidence had been provided for jurors who did not wish to use Bayes' theorem. The Court of Appeal upheld the conviction, but it also gave the opinion that "To introduce Bayes' Theorem, or any similar method, into a criminal trial plunges the jury into inappropriate and unnecessary realms of theory and complexity, deflecting them from their proper task."

Gardner-Medwin argues that the criterion on which a verdict in a criminal trial should be based is not the probability of guilt, but rather the probability of the evidence, given that the defendant is innocent (akin to a frequentist p-value). He argues that if the posterior probability of guilt is to be computed by Bayes' theorem, the prior probability of guilt must be known. This will depend on the incidence of the crime, which is an unusual piece of evidence to consider in a criminal trial. Consider the following three propositions:
 * A – the known facts and testimony could have arisen if the defendant is guilty.
 * B – the known facts and testimony could have arisen if the defendant is innocent.
 * C – the defendant is guilty.

Gardner-Medwin argues that the jury should believe both A and not-B in order to convict. A and not-B implies the truth of C, but the reverse is not true. It is possible that B and C are both true, but in this case he argues that a jury should acquit, even though they know that they will be letting some guilty people go free. See also Lindley's paradox.

Bayesian epistemology
Bayesian epistemology is a movement that advocates for Bayesian inference as a means of justifying the rules of inductive logic.

Karl Popper and David Miller have rejected the idea of Bayesian rationalism, i.e. using Bayes rule to make epistemological inferences: It is prone to the same vicious circle as any other justificationist epistemology, because it presupposes what it attempts to justify. According to this view, a rational interpretation of Bayesian inference would see it merely as a probabilistic version of falsification, rejecting the belief, commonly held by Bayesians, that high likelihood achieved by a series of Bayesian updates would prove the hypothesis beyond any reasonable doubt, or even with likelihood greater than 0.

Other

 * The scientific method is sometimes interpreted as an application of Bayesian inference. In this view, Bayes' rule guides (or should guide) the updating of probabilities about hypotheses conditional on new observations or experiments. The Bayesian inference has also been applied to treat stochastic scheduling problems with incomplete information by Cai et al. (2009).
 * Bayesian search theory is used to search for lost objects.
 * Bayesian inference in phylogeny
 * Bayesian tool for methylation analysis
 * Bayesian approaches to brain function investigate the brain as a Bayesian mechanism.
 * Bayesian inference in ecological studies
 * Bayesian inference is used to estimate parameters in stochastic chemical kinetic models
 * Bayesian inference in econophysics for currency or stock market prediction
 * Bayesian inference in marketing
 * Bayesian inference in motor learning
 * Bayesian inference is used in probabilistic numerics to solve numerical problems

Bayes and Bayesian inference
The problem considered by Bayes in Proposition 9 of his essay, "An Essay Towards Solving a Problem in the Doctrine of Chances", is the posterior distribution for the parameter a (the success rate) of the binomial distribution.

History
The term Bayesian refers to Thomas Bayes (1701–1761), who proved that probabilistic limits could be placed on an unknown event. However, it was Pierre-Simon Laplace (1749–1827) who introduced (as Principle VI) what is now called Bayes' theorem and used it to address problems in celestial mechanics, medical statistics, reliability, and jurisprudence. Early Bayesian inference, which used uniform priors following Laplace's principle of insufficient reason, was called "inverse probability" (because it infers backwards from observations to parameters, or from effects to causes ). After the 1920s, "inverse probability" was largely supplanted by a collection of methods that came to be called frequentist statistics.

In the 20th century, the ideas of Laplace were further developed in two different directions, giving rise to objective and subjective currents in Bayesian practice. In the objective or "non-informative" current, the statistical analysis depends on only the model assumed, the data analyzed, and the method assigning the prior, which differs from one objective Bayesian practitioner to another. In the subjective or "informative" current, the specification of the prior depends on the belief (that is, propositions on which the analysis is prepared to act), which can summarize information from experts, previous studies, etc.

In the 1980s, there was a dramatic growth in research and applications of Bayesian methods, mostly attributed to the discovery of Markov chain Monte Carlo methods, which removed many of the computational problems, and an increasing interest in nonstandard, complex applications. Despite growth of Bayesian research, most undergraduate teaching is still based on frequentist statistics. Nonetheless, Bayesian methods are widely accepted and used, such as for example in the field of machine learning.

Elementary
The following books are listed in ascending order of probabilistic sophistication:
 * Stone, JV (2013), "Bayes' Rule: A Tutorial Introduction to Bayesian Analysis",  Download first  chapter here, Sebtel Press, England.
 * Bolstad, William M. (2007) Introduction to Bayesian Statistics: Second Edition, John Wiley ISBN 0-471-27020-2
 * Updated classic textbook. Bayesian theory clearly presented.
 * Lee, Peter M. Bayesian Statistics: An Introduction. Fourth Edition (2012), John Wiley ISBN 978-1-1183-3257-3
 * Bolstad, William M. (2007) Introduction to Bayesian Statistics: Second Edition, John Wiley ISBN 0-471-27020-2
 * Updated classic textbook. Bayesian theory clearly presented.
 * Lee, Peter M. Bayesian Statistics: An Introduction. Fourth Edition (2012), John Wiley ISBN 978-1-1183-3257-3
 * Lee, Peter M. Bayesian Statistics: An Introduction. Fourth Edition (2012), John Wiley ISBN 978-1-1183-3257-3

Intermediate or advanced

 * DeGroot, Morris H., Optimal Statistical Decisions. Wiley Classics Library. 2004. (Originally published (1970) by McGraw-Hill.) ISBN 0-471-68029-X.
 * Jaynes, E. T. (1998). Probability Theory: The Logic of Science.
 * O'Hagan, A. and Forster, J. (2003). Kendall's Advanced Theory of Statistics, Volume 2B: Bayesian Inference. Arnold, New York. ISBN 0-340-52922-9.
 * Pearl, Judea. (1988). Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference, San Mateo, CA: Morgan Kaufmann.
 * Pierre Bessière et al. (2013). "Bayesian Programming". CRC Press. ISBN 9781439880326
 * Francisco J. Samaniego (2010). "A Comparison of the Bayesian and Frequentist Approaches to Estimation". Springer. New York, ISBN 978-1-4419-5940-9
 * Pearl, Judea. (1988). Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference, San Mateo, CA: Morgan Kaufmann.
 * Pierre Bessière et al. (2013). "Bayesian Programming". CRC Press. ISBN 9781439880326
 * Francisco J. Samaniego (2010). "A Comparison of the Bayesian and Frequentist Approaches to Estimation". Springer. New York, ISBN 978-1-4419-5940-9
 * Francisco J. Samaniego (2010). "A Comparison of the Bayesian and Frequentist Approaches to Estimation". Springer. New York, ISBN 978-1-4419-5940-9

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

title: Bayesian inference


text:
Hooliganism

Hooliganism is disruptive or unlawful behavior such as rioting, bullying and vandalism, often in connection with crowds at sporting events. A hooligan is a person that engages in illicit reckless behaviors and is a public nuisance.

Etymology
There are several theories regarding the origin of the word hooliganism, which is a derivative of the word hooligan. The Compact Oxford English Dictionary states that the word may have originated from the surname of a rowdy Irish family in a music hall song of the 1890s. Clarence Rook, in his 1899 book Hooligan Nights, wrote that the word came from Patrick Hoolihan (or Hooligan), an Irish bouncer and thief who lived in London. In 2015, the BBC Scotland TV programme The Secret Life of Midges noted that the English commander-in-chief during the Jacobite rising of 1745, General Wade, misheard the local Scots Gaelic word for midge—meanbh-chuileag—and coined the word hooligan to describe his fury and frustration at the way the tiny biting creatures made the life of his soldiers and himself a misery; this derivation may be apocryphal.

Early usage
The word first appeared in print in London police court reports in 1894 referring to the name of a gang of youths in the Lambeth area of London—the Hooligan Boys, and later—the O'Hooligan Boys.

In August 1898 the murder of Henry Mappin in Lambeth committed by a member of the gang drew further attention to the word which was immediately popularised by the press. The London newspaper The Daily Graphic wrote in an article on 22 August 1898, "The avalanche of brutality which, under the name of 'Hooliganism' ... has cast such a dire slur on the social records of South London."

The inquest was carried out by Mr Braxton Hicks who "remarked that the activity of the gang he referred to was not confined to Lambeth, but extended to numerous other districts. It was composed of young fellows who scorned to do a stroke of work, and obtained a living by blackmailing. It was a common practice for three or four of these men to walk into a shop and offer the shopman the alternative of giving them a dollar for drink or having his shop wrecked. In connection with the Oakley-street tragedy intimidation had reached an unexampled case. Witnesses had been warned that it would be as much as their life was worth to give evidence against John Darcy. On Wednesday plain-clothes men escorted the witnesses from the court singly. He himself had been warned — not by anonymous letter but through a mysterious personal medium — that if seen in a certain neighbourhood he would be done for. A magistrate had also told him that he had been the recipient of a like indignity."

Arthur Conan Doyle wrote in his 1904 short story "The Adventure of the Six Napoleons", "It seemed to be one of those senseless acts of Hooliganism which occur from time to time, and it was reported to the constable on the beat as such." H. G. Wells wrote in his 1909 semi-autobiographical novel Tono-Bungay, "Three energetic young men of the hooligan type, in neck-wraps and caps, were packing wooden cases with papered-up bottles, amidst much straw and confusion."

According to Life magazine (30 July 1951), the comic strip artist and political cartoonist Frederick Burr Opper introduced a character called Happy Hooligan in 1900; "hapless Happy appeared regularly in U.S. newspapers for more than 30 years", a "naive, skinny, baboon-faced tramp who invariably wore a tomato can for a hat." Life brought this up by way of criticizing the Soviet U.N. delegate Yakov A. Malik for misusing the word. Malik had indignantly referred to anti-Soviet demonstrators in New York as "hooligans". Happy Hooligan, Life reminded its readers, "became a national hero, not by making trouble, which Mr. Malik understands is the function of a hooligan, but by getting himself help."

Modern usage
Later, as the meaning of the word shifted slightly, none of the possible alternatives had precisely the same undertones of a person, usually young, who belongs to an informal group and commits acts of vandalism or criminal damage, starts fights, and who causes disturbances but is not a thief. Hooliganism is now related to sport.

Violence in sports
The words hooliganism and hooligan began to be associated with violence in sports, in particular from the 1970s in the UK with football hooliganism. The phenomenon, however, long preceded the modern term; for example, one of the earliest known instances of crowd violence at a sporting event took place in ancient Constantinople. Two chariot racing factions, the Blues and the Greens, were involved in the Nika riots which lasted around a week in 532 CE; nearly half the city was burned or destroyed, in addition to tens of thousands of deaths.

Sports crowd violence continues to be a worldwide concerning phenomenon exacting at times a large number of injuries, damage to property and casualties. Individual, contextual, social and environmental factors interact and influence one another through a dynamic process occurring at different levels. Macro-sociological accounts suggest that structural strains, experiences of deprivation or a low socio-economic background can at times be instrumental to the acceptance and reproduction of norms that tolerate great levels of violence and territoriality, which is a common feature of football hooliganism. Furthermore, social cleavages within societies facilitate the development of strong in-groups bonds and intense feelings of antagonism towards outsiders which in turn can facilitate group identification and affect the likelihood of fan violence.

In British sports
Beginning in at least the 1960s, the United Kingdom gained a reputation worldwide for football hooliganism; the phenomenon was often dubbed the British or English Disease. However, since the 1980s and well into the 1990s the UK government has led a widescale crackdown on football related violence. While football hooliganism has been a growing concern in some continental European countries in recent years, British football fans now tend to have a better reputation abroad. Although reports of British football hooliganism still surface, the instances now tend to occur at pre-arranged locations rather than at the matches themselves.

In American sports
Football (soccer) and other sports hooliganism overall is rare in the United States in part because of stricter legal penalties for vandalism and physical violence, club markets having their own territory of fans, venues banning weapons, stricter security during games, and a stronger taboo on politics, class, race, and religion into the American sporting culture. Although isolated drunken fights at games do occur, they rarely escalate to major brawling comparable to Europe and Latin America.

In the Soviet Union and Russia

In the Soviet Union the word khuligan (хулиган – transliteration of the English word) was used to refer to scofflaws. Hooliganism (хулиганство, khuliganstvo) was listed as a criminal offense, similar to disorderly conduct in some other jurisdictions, and used as a catch-all charge for prosecuting unapproved behavior.

Olympic medalist Vasiliy Khmelevskiy was convicted of hooliganism for setting a costumed person on fire during a celebration in Minsk in 1979 and sentenced to five years of imprisonment. Mathias Rust was convicted of hooliganism, among other things, for his 1987 Cessna landing on Bolshoy Moskvoretsky Bridge next to Red Square.

Since the fall of the Soviet Union, nowadays, hooliganism is defined generally in the Criminal Code of Russia as an average gravity crime.

More recently, the same charge has been leveled against members of the feminist punk group Pussy Riot for which three members each received a two-year sentence on 17 August 2012. Hooliganism charges were also levelled against the Greenpeace protesters in October 2013. In March 2022, Marina Ovsyannikova, a Russian journalist who held up a banner protesting the Russian invasion of Ukraine during a national news broadcast, was convicted of flouting Russian anti-protest laws and fined ₽30,000 for her actions. The Kremlin called her actions an act of hooliganism.

In film

 * A Clockwork Orange (1971)
 * The Asphalt Jungle (1950)
 * Awaydays (2009)
 * Cass (2008)
 * Eden Lake (2008)
 * EuroTrip (2004)
 * The Firm (1988)
 * The Firm (2009)
 * The Football Factory (2004)
 * Green Street (2005)
 * Green Street 2: Stand Your Ground (2009)
 * Green Street 3: Never Back Down (2013)
 * I.D. (1995) and ID2: Shadwell Army (2016)
 * Neds (2010)
 * Rise of the Footsoldier (2007)
 * Tash Force (2011)
 * Training Day (2001)

title: Hooliganism