4 Day 4 (June 8)

4.1 Announcements

• Intro resources for learning R and SAS

  • Stat 725 and 726

• If office hours times don’t work for you let me know

• Recommended reading

  • Chapters 1 and 2 (pgs 1 - 28) in Linear Models with R
  • Chapter 2 in Applied Regression and ANOVA Using SAS

4.2 Matrix algebra

• Column vectors
  • $\mathbf{y}\equiv(y_{1},y_{2},\ldots,y_{n})^{'}$
  • $\mathbf{x}\equiv(x_{1},x_{2},\ldots,x_{n})^{'}$
  • $\boldsymbol{\beta}\equiv(\beta_{1},\beta_{2},\ldots,\beta_{p})^{'}$
  • $\boldsymbol{1}\equiv(1,1,\ldots,1)^{'}$
  • In R

  y <- matrix(c(1,2,3),nrow=3,ncol=1)
  y

  ##      [,1]
  ## [1,]    1
  ## [2,]    2
  ## [3,]    3

• Matrices
  • $\mathbf{X}\equiv(\mathbf{x}_{1},\mathbf{x}_{2},\ldots,\mathbf{x}_{p})$
  • In R

  X <- matrix(c(1,2,3,4,5,6),nrow=3,ncol=2,byrow=FALSE)
  X

  ##      [,1] [,2]
  ## [1,]    1    4
  ## [2,]    2    5
  ## [3,]    3    6

• Vector multiplication
  • $\mathbf{y}^{'}\mathbf{y}$
  • $\mathbf{1}^{'}\mathbf{1}$
  • $\mathbf{1}\mathbf{1}^{'}$
  • In R

  t(y)%*%y    

  ##      [,1]
  ## [1,]   14

• Matrix by vector multiplication
  • $\mathbf{X}^{'}\mathbf{y}$
  • In R

  t(X)%*%y

  ##      [,1]
  ## [1,]   14
  ## [2,]   32

• Matrix by matrix multiplication
  • $\mathbf{X}^{'}\mathbf{X}$
  • In R

  t(X)%*%X

  ##      [,1] [,2]
  ## [1,]   14   32
  ## [2,]   32   77

• Matrix inversion
  • $(\mathbf{X}^{'}\mathbf{X})^{-1}$
  • In R

  solve(t(X)%*%X)

  ##            [,1]       [,2]
  ## [1,]  1.4259259 -0.5925926
  ## [2,] -0.5925926  0.2592593

• Determinant of a matrix
  • $|\mathbf{I}|$
    
  • In R

  I <- diag(1,3)
  I

  ##      [,1] [,2] [,3]
  ## [1,]    1    0    0
  ## [2,]    0    1    0
  ## [3,]    0    0    1

  det(I)

  ## [1] 1

• Quadratic form
  • $\mathbf{y}^{'}\mathbf{S}\mathbf{y}$
• Derivative of a quadratic form (Note $\mathbf{S}$ is a symmetric matrix; e.g., $\mathbf{X}^{'}\mathbf{X}$)
  • $\frac{\partial}{\partial\mathbf{y}}\mathbf{y^{'}\mathbf{S}\mathbf{y}}=2\mathbf{S}\mathbf{y}$
• Other useful derivatives
  • $\frac{\partial}{\partial\mathbf{y}}\mathbf{\mathbf{x^{'}}\mathbf{y}}=\mathbf{x}$
  • $\frac{\partial}{\partial\mathbf{y}}\mathbf{\mathbf{X^{'}}\mathbf{y}}=\mathbf{X}$

4.3 Introduction to linear models

• What is a model?

• What is a linear model?

  • Most widely used model in science, engineering, and statistics

  • Vector form: $\mathbf{y}=\beta_{0}+\beta_{1}\mathbf{x}_{1}+\beta_{2}\mathbf{x}_{2}+\ldots+\beta_{p}\mathbf{x}_{p}+\boldsymbol{\varepsilon}$

  • Matrix form: $\mathbf{y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\varepsilon}$

  • Which part of the model is the mathematical model

  • Which part of the model makes the linear model a “statistical” model

  • Visual

• Which of the four below are a linear model $$\mathbf{y}=\beta_{0}+\beta_{1}\mathbf{x}_{1}+\beta_{2}\mathbf{x}^{2}_{1}+\boldsymbol{\varepsilon}$$ $$\mathbf{y}=\beta_{0}+\beta_{1}\mathbf{x}_{1}+\beta_{2}\text{log(}\mathbf{x}_{1}\text{)}+\boldsymbol{\varepsilon}$$ $$\mathbf{y}=\beta_{0}+\beta_{1}e^{\beta_{2}\mathbf{x}_{1}}+\boldsymbol{\varepsilon}$$ $$\mathbf{y}=\beta_{0}+\beta_{1}\mathbf{x}_{1}+\text{log(}\beta_{2}\text{)}\mathbf{x}_{1}+\boldsymbol{\varepsilon}$$

• Why study the linear model?

  • Building block for more complex models (e.g., GLMs, mixed models, machine learning, etc)
  • We know the most about it

4.4 Estimation

• Three options to estimate $\boldsymbol{\beta}$
  • Minimize a loss function
  • Maximize a likelihood function
  • Find the posterior distribution
  • Each option requires different assumptions

4.5 Loss function approach

• Define a measure of discrepancy between the data and the mathematical model

  • Find the values of $\boldsymbol{\beta}$ that make $\mathbf{X}\boldsymbol{\beta}$ “closest” to $\mathbf{y}$
  • Visual

• Classic example $$\underset{\boldsymbol{\beta}}{\operatorname{argmin}}\sum_{i=1}^{n}(y_i-\mathbf{x}_{i}^{\prime}\boldsymbol{\beta})^2$$ or in matrix form $$\underset{\boldsymbol{\beta}}{\operatorname{argmin}}(\mathbf{y} - \mathbf{X}\boldsymbol{\beta})^{\prime}(\mathbf{y} - \mathbf{X}\boldsymbol{\beta})$$ which results in $$\hat{\boldsymbol{\beta}}=(\mathbf{X}^{\prime}\mathbf{X})^{-1}\mathbf{X}^{\prime}\mathbf{y}$$ -In R

  y <- c(0.16,2.82,2.24)
  X <- matrix(c(1,1,1,1,2,3),nrow=3,ncol=2,byrow=FALSE)
  
  solve(t(X)%*%X)%*%t(X)%*%y

  ##       [,1]
  ## [1,] -0.34
  ## [2,]  1.04

  optim(par=c(0,0),method = c("Nelder-Mead"),fn=function(beta){t(y-X%*%beta)%*%(y-X%*%beta)})

  ## $par
  ## [1] -0.3399977  1.0399687
  ## 
  ## $value
  ## [1] 1.7496
  ## 
  ## $counts
  ## function gradient 
  ##       61       NA 
  ## 
  ## $convergence
  ## [1] 0
  ## 
  ## $message
  ## NULL

  lm(y~X-1)

  ## 
  ## Call:
  ## lm(formula = y ~ X - 1)
  ## 
  ## Coefficients:
  ##    X1     X2  
  ## -0.34   1.04