11 Day 11 (February 23)

11.1 Announcements

  • Status update on proposals and projects

  • Go over a few things from Assignment 5

  • Questions about Assignment 6?

  • Read Ch. 11 on Linear Regression

11.2 The Bayesian Linear Model

  • The classic paper is Lindley and Smith (1972)

  • Implemented with MCMC from Gelfand and Smith (1990)

  • Recent recent elaborations include the Bayesian Lasso (see Park and Casella 2008)

  • The model

    • Distribution of the data: “\(\mathbf{y}\) given \(\boldsymbol{\beta}\) and \(\sigma_{\varepsilon}^{2}\)\[[\mathbf{y}|\boldsymbol{\beta},\sigma_{\varepsilon}^{2}]\equiv\text{N}(\mathbf{X}\boldsymbol{\beta},\sigma_{\varepsilon}^{2}\mathbf{I})\] This is also sometimes called the data model.
    • Distribution of the parameters: \(\boldsymbol{\beta}\) and \(\sigma_{\varepsilon}^{2}\) \[[\boldsymbol{\beta}]\equiv\text{N}(\mathbf{0},\sigma_{\beta}^{2}\mathbf{I})\] \[[\sigma_{\varepsilon}^{2}]\equiv\text{inverse gamma}(q,r)\]
  • Computations and statistical inference

    • Using Bayes rule (Bayes 1763) we can obtain the joint posterior distribution \[[\boldsymbol{\beta},\sigma_{\varepsilon}^{2}|\mathbf{y}]=\frac{[\mathbf{y}|\boldsymbol{\beta},\sigma_{\varepsilon}^{2}][\boldsymbol{\beta}][\sigma_{\varepsilon}^{2}]}{\int\int [\mathbf{y}|\boldsymbol{\beta},\sigma_{\varepsilon}^{2}][\boldsymbol{\beta}][\sigma_{\varepsilon}^{2}]d\boldsymbol{\beta}d\sigma_{\varepsilon}^{2}}\]
      • Statistical inference about a parameters is obtained from the marginal posterior distributions \[[\boldsymbol{\beta}|\mathbf{y}]=\int[\boldsymbol{\beta},\sigma_{\varepsilon}^{2}|\mathbf{y}]d\sigma_{\varepsilon}^{2}\] \[[\sigma_{\varepsilon}^{2}|\mathbf{y}]=\int[\boldsymbol{\beta},\sigma_{\varepsilon}^{2}|\mathbf{y}]d\boldsymbol{\beta}\]
    • In practice it is difficult to find closed-form solutions for the marginal posterior distributions, but easy to find closed-form solutions for the conditional posterior distribtuions.
      • Full-conditional distribtuions: \(\boldsymbol{\beta}\) and \(\sigma_{\varepsilon}^{2}\) \[[\boldsymbol{\beta}|\sigma_{\varepsilon}^{2},\mathbf{y}]\equiv\text{N}\bigg{(}\big{(}\mathbf{X}^{\prime}\mathbf{X}+\frac{1}{\sigma_{\beta}^{2}}\mathbf{I}\big{)}^{-1}\mathbf{X}^{\prime}\mathbf{y},\sigma_{\varepsilon}^{2}\big{(}\mathbf{X}^{\prime}\mathbf{X}+\frac{1}{\sigma_{\beta}^{2}}\mathbf{I}\big{)}^{-1}\bigg{)}\] \[[\sigma_{\varepsilon}^{2}|\mathbf{\boldsymbol{\beta}},\mathbf{y}]\equiv\text{inverse gamma}\left(q+\frac{n}{2},(r+\frac{1}{2}(\mathbf{y}-\mathbf{X}\boldsymbol{\beta})^{'}(\mathbf{y}-\mathbf{X}\boldsymbol{\beta}))^-1\right)\]
      • Gibbs sampler for the Bayesian linear model
        • Write out on the whiteboard