3.6 Exercises

1. Write the distribution of the Bernoulli example in canonical form, and find the mean and variance of the sufficient statistic.

2. Given a random sample $\boldsymbol{Y} = [Y_1 \ Y_2 \ \dots \ Y_N]^{\top}$ from $N$ binomial experiments, each with known size $n_i$ and the same unknown probability $\theta$, show that $p(\boldsymbol{y} \mid \theta)$ is in the exponential family. Then, find the posterior distribution, the marginal likelihood, and the predictive distribution of the binomial-Beta model assuming the number of trials is known.

3. Given a random sample $\boldsymbol{Y} = [Y_1 \ Y_2 \ \dots \ Y_N]^{\top}$ from an exponential distribution, show that $p(\boldsymbol{y} \mid \lambda)$ is in the exponential family. Additionally, find the posterior distribution, the marginal likelihood, and the predictive distribution of the exponential-Gamma model.

4. Given that $\boldsymbol{Y} \sim N_N(\boldsymbol{\mu}, \boldsymbol{\Sigma})$, that is, a multivariate normal distribution, show that $p(\boldsymbol{y} \mid \boldsymbol{\mu}, \boldsymbol{\Sigma})$ is in the exponential family.

5. Find the marginal likelihood in the normal/inverse-Wishart model.

6. Find the posterior predictive distribution in the normal/inverse-Wishart model, and show that ${\boldsymbol{Y}}_0\mid {\boldsymbol{Y}}\sim T_{N_0,M}(\alpha_n-M+1,{\boldsymbol{X}}_0{\boldsymbol{B}}_n,{\boldsymbol{I}}_{N_0}+{\boldsymbol{X}}_0{\boldsymbol{V}}_n{\boldsymbol{X}}_0^{\top},{\boldsymbol{\Psi}}_n)$.

7. Show that $\delta_n=\delta_0+({\boldsymbol{y}}-{\boldsymbol{X}}\hat{\boldsymbol{\beta}})^{\top}({\boldsymbol{y}}-{\boldsymbol{X}}\hat{\boldsymbol{\beta}})+(\hat{\boldsymbol{\beta}}-\boldsymbol{\beta}_0)^{\top}(({\boldsymbol{X}}^{\top}{\boldsymbol{X}})^{-1}+{\boldsymbol{B}}_0)^{-1}(\hat{\boldsymbol{\beta}}-\boldsymbol{\beta}_0)$ in the linear regression model, and that ${\boldsymbol{\Psi}}_{n}={\boldsymbol{\Psi}}_{0}+{\boldsymbol{S}}+(\hat{\boldsymbol{B}}-{\boldsymbol{B}}_{0})^{\top}{\boldsymbol{V}}_{n}(\hat{\boldsymbol{B}}-{\boldsymbol{B}}_{0})$ in the linear multivariate regression model.

8. Show that in the linear regression model $\boldsymbol{\beta}_n^{\top}({\boldsymbol{B}}_n^{-1}-{\boldsymbol{B}}_n^{-1}{\boldsymbol{M}}^{-1}{\boldsymbol{B}}_n^{-1})\boldsymbol{\beta}_n={\boldsymbol{\boldsymbol{\beta}}}_{**}^{\top}{\boldsymbol{C}}{\boldsymbol{\boldsymbol{\beta}}}_{**}$ and $\boldsymbol{\beta}_{**}={\boldsymbol{X}}_0\boldsymbol{\beta}_n$.

9. Show that $({\boldsymbol{Y}}-{\boldsymbol{X}}{\boldsymbol{B}})^{\top}({\boldsymbol{Y}}-{\boldsymbol{X}}{\boldsymbol{B}})={\boldsymbol{S}}+({\boldsymbol{B}}-\widehat{\boldsymbol{B}})^{\top}{\boldsymbol{X}}^{\top}{\boldsymbol{X}}({\boldsymbol{B}}-\widehat{\boldsymbol{B}})$ where ${\boldsymbol{S}}= ({\boldsymbol{Y}}-{\boldsymbol{X}}\widehat{\boldsymbol{B}})^{\top}({\boldsymbol{Y}}-{\boldsymbol{X}}\widehat{\boldsymbol{B}})$, $\widehat{\boldsymbol{B}}= ({\boldsymbol{X}}^{\top}{\boldsymbol{X}})^{-1}{\boldsymbol{X}}^{\top}{\boldsymbol{Y}}$ in the multivariate regression model.

10. What is the probability that the Sun will rise tomorrow?

This is the most famous example by Richard Price, developed in the Appendix of Bayes’ theorem paper (Thomas Bayes 1763). Here, we implicitly use Laplace’s Rule of Succession to solve this problem. In particular, if we were a priori uncertain about the probability that the Sun will rise on a specified day, we can assume a uniform prior distribution over $(0,1)$, that is, a Beta(1,1) distribution. Then, what is the probability that the Sun will rise?

11. Using information from Public Policy Polling in September 27th-28th for the 2016 presidential five-way race in USA, there are 411, 373 and 149 sampled people supporting Hillary Clinton, Donald Trump and other, respectively.

• Find the posterior probability of the percentage difference of people supporting Hillary versus Trump according to this data using a non-informative prior, that is, $\alpha_0=[1 \ 1 \ 1]$ in the multinomial-Dirichlet model. What is the probability of having more supports of Hillary vs Trump?

• What is the probability that sampling one hundred independent individuals 44, 40 and 16 support Hillary, Trump and other, respectively?

12. Math test example continues

You have a random sample of math scores of size $N=50$ from a normal distribution, $Y_i\sim {N}(\mu, \sigma^2)$. The sample mean and variance are equal to $102$ and $10$, respectively. Using the normal-normal/inverse-gamma model where $\mu_0=100$, $\beta_0=1$, $\alpha_0=\delta_0=0.001$

• Get a 95% confidence and credible interval for $\mu$.
• What is the posterior probability that $\mu > 103$?

13. Demand of electricity example continues

Set $c_0$ such that maximizes the marginal likelihood in the specifications with and without electricity price in the example of demand of electricity (empirical Bayes). Then, calculate the Bayes factor, and conclude if there is evidence supporting the inclusion of the price of electricity in the demand equation.

14. Utility demand

Use the file Utilities.csv to estimate a multivariate linear regression model where $\boldsymbol{Y}_i=\left[\log(\text{electricity}_i) \ \log(\text{water}_i) \ \log(\text{gas}_i)\right]$ as function of $\log(\text{electricity price}_i)$, $\log(\text{water price}_i)$, $\log(\text{gas price}_i)$, $\text{IndSocio1}_i$, $\text{IndSocio2}_i$, $\text{Altitude}_i$, $\text{Nrooms}_i$, $\text{HouseholdMem}_i$, $\text{Children}_i$, and $\log(\text{Income}_i)$, where electricity, water and gas are monthly consumption of electricity (kWh), water (m$^3$) and gas (m$^3$), and other definitions are given in the Example of Section 3.3. Omit households that do not consume any of the utilities in this exercise.

Set a non-informative prior framework, $\boldsymbol{B}_0=\left[0\right]_{11\times 3}$, $\boldsymbol{V}_0=1000 \boldsymbol{I}_{11}$, $\boldsymbol{\Psi}_0=1000 \boldsymbol{I}_{3}$ and $\alpha_0=3$, where we have $K=11$ (regressors plus intercept) and $M=3$ (equations) in this exercise.

• Find the posterior mean estimates and the highest posterior density intervals at 95% of $\boldsymbol{B}$ and $\boldsymbol{\Sigma}$. Use the marginal distribution and the conditional distribution to obtain the posterior estimates of $\boldsymbol{B}$, and compare the results.
• Find the Bayes factor comparing the baseline model in this exercise with the same specification but using the income in dollars. Now, calculate the Bayes factor using the income in thousand dollars. Is there any difference?
• Find the predictive distribution for the monthly demand of electricity, water and gas in the baseline specification of a household located in the lowest socioeconomic condition in a municipality located below 1000 meters above the sea level, 2 rooms, 3 members with children, a monthly income equal to USD 500, an electricity price equal to USD/kWh 0.15, a water price equal to USD/M$^3$ 0.70, and a gas price equal to USD/M$^3$ 0.75.

15 Ph.D. students sleeping hours (J. Albert 2009)

We are interested in learning about the proportion of Ph.D. students who sleep at least 6 hours per day. We have a sample of 52 students, where 15 report sleeping at least 6 hours, and the remaining 37 report not sleeping at least 6 hours. The prior distribution is a Beta distribution, with hyperparameters calibrated so that the prior probabilities of the proportion of students who sleep least than 6 hours being less than 0.4 and 0.75 are 0.6 and 0.95, respectively. Estimate the 95% posterior credible interval for the proportion of Ph.D. students who sleep at least 6 hours per day. Then, assume there is a group of experts whose beliefs about the proportion of Ph.D. students sleeping at least 6 hours are represented in the following table:

Table 3.1: Probability distribution: Ph.D students that sleep at least 6 hours per day.

$h$	0.05	0.10	0.15	0.20	0.25	0.30	0.35	0.40	0.45	0.50	0.15
$P(p=h)$	0.05	0.07	0.10	0.12	0.15	0.17	0.15	0.11	0.06	0.01	0.01

Use this Table as prior information, and find the posterior distribution of the proportion of students that sleep at least 6 hours.

References

Albert, Jim. 2009. Bayesian Computation with r. 2nd ed. Use r! New York, NY: Springer. https://doi.org/10.1007/978-0-387-92297-3.

Bayes, Thomas. 1763. “LII. An Essay Towards Solving a Problem in the Doctrine of Chances. By the Late Rev. Mr. Bayes, FRS Communicated by Mr. Price, in a Letter to John Canton, AMFR S.” Philosophical Transactions of the Royal Society of London, no. 53: 370–418.