5 Day 5 (June 7)

5.1 Announcements

  • Tutoring for program R
    • Dickens Hall room 108
    • 12:30 - 1:30 Monday - Friday

5.2 Estimation

  • Three options to estimate \(\boldsymbol{\beta}\)
    • Minimize a loss function
    • Maximize a likelihood function
    • Find the posterior distribution
    • Each option requires different assumptions

5.3 Loss function approach

  • Define a measure of discrepancy between the data and the mathematical model
    • Find the values of \(\boldsymbol{\beta}\) that make \(\mathbf{X}\boldsymbol{\beta}\) “closest” to \(\mathbf{y}\)
    • Visual
  • Classic example \[\underset{\boldsymbol{\beta}}{\operatorname{argmin}}\sum_{i=1}^{n}(y_i-\mathbf{x}_{i}^{\prime}\boldsymbol{\beta})^2\] or in matrix form \[\underset{\boldsymbol{\beta}}{\operatorname{argmin}}(\mathbf{y} - \mathbf{X}\boldsymbol{\beta})^{\prime}(\mathbf{y} - \mathbf{X}\boldsymbol{\beta})\] which results in \[\hat{\boldsymbol{\beta}}=(\mathbf{X}^{\prime}\mathbf{X})^{-1}\mathbf{X}^{\prime}\mathbf{y}\]
  • Three ways to do it in program R
    • Using scalar calculus and algebra (kind of)
    y <- c(0.16,2.82,2.24)
x <- c(1,2,3)

y.bar <- mean(y)
x.bar <- mean(x)

# Estimate the slope parameter
beta1.hat <- sum((x-x.bar)*(y-y.bar))/sum((x-x.bar)^2)
beta1.hat
    ## [1] 1.04
    # Estimate the intercept parameter
beta0.hat <- y.bar - sum((x-x.bar)*(y-y.bar))/sum((x-x.bar)^2)*x.bar
beta0.hat
    ## [1] -0.34
    • Using matrix calculus and algebra
    y <- c(0.16,2.82,2.24)
X <- matrix(c(1,1,1,1,2,3),nrow=3,ncol=2,byrow=FALSE)
solve(t(X)%*%X)%*%t(X)%*%y
    ##       [,1]
## [1,] -0.34
## [2,]  1.04
    • Using modern (circa 1970’s) optimization techniques
    y <- c(0.16,2.82,2.24)
x <- c(1,2,3)

optim(par=c(0,0),method = c("Nelder-Mead"),fn=function(beta){sum((y-(beta[1]+beta[2]*x))^2)})
    ## $par
## [1] -0.3399977  1.0399687
## 
## $value
## [1] 1.7496
## 
## $counts
## function gradient 
##       61       NA 
## 
## $convergence
## [1] 0
## 
## $message
## NULL
    • Using modern and user friendly statistical computing software
    df <- data.frame(y = c(0.16,2.82,2.24),x = c(1,2,3))
lm(y~x,data=df)
    ## 
## Call:
## lm(formula = y ~ x, data = df)
## 
## Coefficients:
## (Intercept)            x  
##       -0.34         1.04
  • Live example