Chapter 1 Probabilty and Inference

Bayes’ Rule: If $A$ and $B$ are events in event space $F$, then Bayes’ rule states that $$P(A\mid B) = \frac{P(B\mid A)P(A)}{P(B)} = \frac{P(B\mid A)P(A)}{P(B\mid A)P(A) + P(B\mid A^c)P(A^c)}$$ Let

• $y$ be the data we will collect from an experiment,
• $K$ be everything we know for certain about the world (aside from $y$), and
• $\theta$ be anything we don’t know for certain.

A Bayesian statistician is an individual who makes decisions based on the probability distribution of those things we don’t know conditional on what we know, i.e. $p(\theta\mid y, K)$.

• Parameter estimation: $p(\theta\mid y, M)$, where $M$ is the model with parameter vector $\theta$
  
• Hypothesis testing: $p(M_j\mid y, M)$
• Prediction: $p(\tilde{y}\mid y, M)$

Parameter Estimation Example: exponential model

Let $Y\mid \theta \sim Exp(\theta)$, the likelihood is $p(y\mid \theta) = \theta \exp(-\theta y)$. Let’s assume a prior $\theta \sim Ga(a, b)$, $p(\theta) = \frac{b^a}{\Gamma(a)}\theta^{a-1}e^{-b\theta}$, then prior predictive distribution is $$p(y)=\int p(y \mid \theta) p(\theta) d \theta=\frac{b^{a}}{\Gamma(a)} \frac{\Gamma(a+1)}{(b+y)^{a+1}}$$ The posterior is $$p(\theta\mid y) = \frac{p(y\mid \theta)p(\theta)}{p(y)} = \frac{(b+y)^{a+1}}{\Gamma(a+1)}\theta^{a+1-1}e^{-(b+y)\theta}$$ thus $\theta\mid y \sim Ga(a+1, b+y)$.

If $p(y) < \infty$, we can use $p(\theta\mid y) \propto p(y\mid \theta)p(\theta)$ to find the posterior. In the example, $\theta^{a}e^{-(b+y)\theta}$ is the kernel of a $Ga(a+1, b+y)$ distribution.

Bayesian learning: $p(\theta) \rightarrow p(\theta\mid y_1) \rightarrow p(\theta\mid y_1, y_2) \rightarrow \ldots$

Model selection

Formally, to select a model, we use $p(M_j\mid y) \propto p(y\mid M_j)p(M_j)$. Thus, a Bayesian approach provides a natural way to learn about models, i.e. $p(M_j) \rightarrow p(M_j\mid y)$.

Prediction

$p(\tilde{y}\mid y) = \int p(\tilde{y}, \theta \mid y)dy = \int p(\tilde{y}\mid \theta)p(\theta\mid y)d\theta$. From the previous example, let $y_i \sim Exp(\theta)$, $\theta \sim Ga(a,b)$, $$\begin{aligned} p(\tilde{y} \mid y) &=\int p(\tilde{y} \mid \theta) p(\theta \mid y) d \theta \\ &=\int \theta e^{-\theta \tilde{y}} \frac{(b+n \bar{y})^{a+n}}{\Gamma(a+1)} \theta^{a+n-1} e^{-\theta(b+n \bar{y})} d \theta \\ &=\frac{(b+n \bar{y})^{a+n}}{\Gamma(a+n)} \int \theta^{a+n+1-1} e^{-\theta(b+n \bar{y}+\tilde{y})} d \theta \\ &=\frac{(b+n \bar{y})^{a+n}}{\Gamma(a+n)} \frac{\Gamma(a+n+1)}{(b+n \bar{y}+\tilde{y})^{a+n+1}} \\ &=\frac{(a+n)(b+n \bar{y})^{a+n}}{(\tilde{y}+b+n \bar{y})^{a+n+1}} \end{aligned}$$ which is called the Lomax distribution for $\tilde{y}$ with parameters $a + n$ and $b + n\bar y$.

Probabilty: A subjective probability describes an individual’s personal judgement about how likely a particular event is to occur.

Rational individuals can differ about the probability of an event by having different knowledge, i.e. $P(E \mid K_1) \neq P(E \mid K_2)$. But given enough data, we might have $P(E \mid K_1,Y) \approx P(E \mid K_2, y)$.