5 Day 5 (June 7)

5.1 Announcements

Tutoring for program R
- Dickens Hall room 108
- 12:30 - 1:30 Monday - Friday

5.2 Estimation

Three options to estimate $\boldsymbol{\beta}$
- Minimize a loss function
- Maximize a likelihood function
- Find the posterior distribution
- Each option requires different assumptions

5.3 Loss function approach

Define a measure of discrepancy between the data and the mathematical model
- Find the values of $\boldsymbol{\beta}$ that make $\mathbf{X}\boldsymbol{\beta}$ “closest” to $\mathbf{y}$
- Visual
Classic example $\underset{\boldsymbol{\beta}}{\operatorname{argmin}}\sum_{i=1}^{n}(y_i-\mathbf{x}_{i}^{\prime}\boldsymbol{\beta})^2$ or in matrix form $\underset{\boldsymbol{\beta}}{\operatorname{argmin}}(\mathbf{y} - \mathbf{X}\boldsymbol{\beta})^{\prime}(\mathbf{y} - \mathbf{X}\boldsymbol{\beta})$ which results in $\hat{\boldsymbol{\beta}}=(\mathbf{X}^{\prime}\mathbf{X})^{-1}\mathbf{X}^{\prime}\mathbf{y}$

Three ways to do it in program R

Using scalar calculus and algebra (kind of)

y <- c(0.16,2.82,2.24)
x <- c(1,2,3)

y.bar <- mean(y)
x.bar <- mean(x)

# Estimate the slope parameter
beta1.hat <- sum((x-x.bar)*(y-y.bar))/sum((x-x.bar)^2)
beta1.hat

## [1] 1.04

# Estimate the intercept parameter
beta0.hat <- y.bar - sum((x-x.bar)*(y-y.bar))/sum((x-x.bar)^2)*x.bar
beta0.hat

## [1] -0.34

Using matrix calculus and algebra

y <- c(0.16,2.82,2.24)
X <- matrix(c(1,1,1,1,2,3),nrow=3,ncol=2,byrow=FALSE)
solve(t(X)%*%X)%*%t(X)%*%y

##       [,1]
## [1,] -0.34
## [2,]  1.04

Using modern (circa 1970’s) optimization techniques

y <- c(0.16,2.82,2.24)
x <- c(1,2,3)

optim(par=c(0,0),method = c("Nelder-Mead"),fn=function(beta){sum((y-(beta[1]+beta[2]*x))^2)})

## $par
## [1] -0.3399977  1.0399687
## 
## $value
## [1] 1.7496
## 
## $counts
## function gradient 
##       61       NA 
## 
## $convergence
## [1] 0
## 
## $message
## NULL

Using modern and user friendly statistical computing software

df <- data.frame(y = c(0.16,2.82,2.24),x = c(1,2,3))
lm(y~x,data=df)

## 
## Call:
## lm(formula = y ~ x, data = df)
## 
## Coefficients:
## (Intercept)            x  
##       -0.34         1.04

Live example