Chapter 4 Data Asymptotics

Suppose \(p(\theta\mid y)\) is unimodel and roughly symmetric, a Taylor series expansion of the logarithm of the posterior around the posterior mode \(\hat\theta\) is \[ \log p(\theta \mid y)=\log p(\hat{\theta} \mid y)-\frac{1}{2}(\theta-\hat{\theta})^{\top}\left[-\frac{d^{2}}{d \theta^{2}} \log p(\theta \mid y)\right]_{\theta=\hat{\theta}}(\theta-\hat{\theta})+\cdots \] Discarding the higher order terms, this expansionsion provides a normal approximation to the posterior, i.e. \(p(\theta\mid y) \stackrel{d}{\approx} N(\hat\theta, I(\hat\theta)^{-1})\).

Theorem 1: If the parameter space \(\Theta\) is discrete and \(P(\theta = \theta_0) > 0\), then \(P(\theta = \theta_0\mid y) \rightarrow 1\) as \(n \rightarrow \infty\).

Theorem 2: If the parameter space \(\Theta\) is continuous and \(A\) is a neighborhood around \(\Theta_0\) with \(P(\theta\in A) > 0\), then \(P(\theta \in A \mid y) \rightarrow 1\) as \(n \rightarrow \infty\).

An estimator is consistent, i.e. \(\hat\theta \stackrel{p}{\rightarrow} \theta_0\) if \(\lim_{n \rightarrow \infty} P(|\hat\theta - \theta_0| < \epsilon) = 1\). Under regularity conditions, \(\hat\theta_{MLE} \stackrel{p}{\rightarrow} \theta_0\) .

Example 1: Binomial example

Let \(y \sim Bin(n, \theta)\) and \(\theta \sim Be(a, b)\), then \(\theta\mid y \sim Be(a + y, b + n - y)\) and the posterior mode is \(\hat\theta = \frac{y'}{n'} = \frac{a + y - 1}{a + b + n - 2}\). Thus \(I(\hat\theta) = \frac{n'}{\hat\theta(1 - \hat\theta)}\) and \(p(\theta\mid y) \stackrel{d}{\approx } N\left(\hat\theta, \frac{\hat\theta(1 - \hat\theta)}{n'} \right)\).

Recall that \(\hat\theta_{MLE} = y/n\). The following estimators are all consistent

• Posterior mean: \(\frac{a + y}{a + b + n}\)
• Posterior median: \(\approx \frac{a + y - 1/3}{a + b + n - 2/3}\) for \(a, b > 1\)
• Posterior mode: \(\frac{a + y - 1}{a + b + n - 2}\)

since as \(n \rightarrow \infty\), these all converage to \(\hat\theta_{MLE} = y/n\).

Example 2: Normal example

Consider \(Y_i \stackrel{iid}{\sim} N(\theta, 1)\) with known and prior \(\theta \sim N(c, 1)\), then \(\theta\mid y \sim N\left(\frac{1}{n+1}c + \frac{n}{n+1}\bar y, \frac{1}{n+1} \right)\).

Recall that \(\hat\theta_{MLE} = \bar y\), and the posterior mean coverages to the MLE.

Asymptotic normality

For large \(n\), we have \[ \log p(\theta \mid y) \approx \log p(\hat{\theta} \mid y)-\frac{1}{2}(\theta-\hat{\theta})^{\top}\left[n \mathrm{I}\left(\theta_{0}\right)\right](\theta-\hat{\theta}) \] where \(\hat\theta\) is the posterior mode. Since \(\hat\theta \rightarrow \theta_0\) and \(I(\hat\theta) \rightarrow I(\theta_0)\) as \(n \rightarrow \infty\), we have \[ p(\theta\mid y )\propto \exp\left(-\frac{1}{2}(\theta - \hat\theta)^T\left[n I(\hat\theta) \right] (\theta - \hat\theta) \right). \] Thus, \(\theta\mid y \stackrel{d}{\rightarrow} N\left(\hat\theta, \frac{1}{n} I(\hat\theta)^{-1} \right)\), i.e. the posterior distribution is asymptotically normal.

Suppose that \(f(y)\) the true sampling distribution does not correspond to \(p(y\mid \theta)\) for some \(\theta = \theta_0\). Then the posterior \(p(\theta\mid y)\) converges to a \(\theta_0\) that is the smallest in Kullback-Leibler divergence to the true \(f(y)\) where \[ K L(f(y) \| p(y \mid \theta))=E\left[\log \left(\frac{f(y)}{p(y \mid \theta)}\right)\right]=\int \log \left(\frac{f(y)}{p(y \mid \theta)}\right) f(y) d y \] That is, we do about the best that we can given that we have assumed the wrong sampling distribution \(p(y \mid \theta)\).