Least Squares Estimates For The Linear Regression

The least squares estimates for a linear regression of the form $$\mathbf{Y} = \mathbf{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}.$$ is given by $$\boldsymbol{\hat{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y}$$

In order for $\hat{\beta}$ to exist, we need to make some assumptions.

Assumption: The $(n \times [p+1])$ matrix $\mathbf{X}$, is of full rank $p+1$.

This will mean that the matrix $\mathbf{X}^T\mathbf{X}$ is also of full rank $p+1$. As a consequence of this:

1. $\mathbf{X}^T\mathbf{X}$ is a positive definite matrix.

2. $(\mathbf{X}^T\mathbf{X})^{-1}$ exists and is unique.

First, we recall the steps for the calculus based solution for obtaining least squares estimates

• Construct the sum-of-squares function as:

  $$S(\boldsymbol \beta) = \sum_{i=1}^n(y_i-E(y_i))^2$$

• Differentiate with respect to each of the parameters $$\frac{\partial S(\boldsymbol{\beta})}{\partial \alpha}, \frac{\partial S(\boldsymbol{\beta})}{\partial\beta_1}, \ldots, \frac{\partial S(\boldsymbol{\beta})}{\partial\beta_{(p)}};$$ which can be compactly written as $$\frac{\partial S(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}= \left( \begin{array}{c}\frac{\partial S(\boldsymbol{\beta})}{\partial \alpha}\\ \frac{\partial S(\boldsymbol{\beta})}{\partial \beta_1}\\ \vdots \\ \frac{\partial S(\boldsymbol{\beta})}{\partial \beta_{(p)}} \end{array}\right)$$

• Set partial derivatives = 0 and solve for each of the parameters i.e. $$\frac{\partial S(\boldsymbol{\beta})}{\partial \alpha} = 0, \frac{\partial S(\boldsymbol{\beta})}{\partial \beta_j} = 0, \quad j=1,\ldots, p; \quad \mbox{ or } \quad \frac{\partial S(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}= \mathbf{0};$$ where $\mathbf{0}$ is length $p$ vector of $0$s.

• Check that you have found a minimum by computing second derivatives.

Let us first expand $S(\boldsymbol{\beta})$ $$\begin{aligned} S(\boldsymbol{\beta})&=(\boldsymbol{Y}-\mathbf{X}\boldsymbol{\beta})^T(\mathbf{Y}-\mathbf{X}\boldsymbol{\beta})\\ &=(\mathbf{Y}^T-\boldsymbol{\beta}^T\mathbf{X}^T)(\mathbf{Y}-\mathbf{X}\boldsymbol{\beta})\\ &=\mathbf{Y}^T\mathbf{Y}-\boldsymbol{\beta}^T\mathbf{X}^T\mathbf{Y}-\mathbf{Y}^T\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\beta}^T\mathbf{X}^T\mathbf{X}\boldsymbol{\beta}\\ &=\mathbf{Y}^T\mathbf{Y}-2\mathbf{Y}^T\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\beta}^T\mathbf{X}^T\mathbf{X}\boldsymbol{\beta} & \mbox{ as } \boldsymbol{\beta}^T\mathbf{X}^T\mathbf{Y} \mbox{is a scalar it is equal to its transpose } (\mathbf{Y}^T\mathbf{X}\boldsymbol{\beta})^T. \end{aligned}$$

Now, we calculate the vector differentiation, $\frac{\partial S(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}$.

$$\begin{aligned} \frac{\partial S(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} &= \frac{\partial} {\partial \boldsymbol{\beta}} ( \mathbf{Y}^T\mathbf{Y}-2\mathbf{Y}^T\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\beta}^T\mathbf{X}^T\mathbf{X}\boldsymbol{\beta}) \\ &=0-2\mathbf{Y}^T\mathbf{X}+2\boldsymbol{\beta}^T\mathbf{X}^T\mathbf{X} \end{aligned}$$ Equating the above to $\mathbf{0}$ gives

$$\begin{array}{rrll} &2\mathbf{Y}^T\mathbf{X}+2\boldsymbol{\beta}^T\mathbf{X}^T\mathbf{X} &=\mathbf{0}\\ \implies& \boldsymbol{\beta}^T\mathbf{X}^T\mathbf{X}&=\mathbf{Y}^T\mathbf{X}\\ \implies& \mathbf{X}^T\mathbf{X} \boldsymbol{\beta} &=\mathbf{X}^T\mathbf{Y}& \mbox{taking transpose on both sides}\\ \implies & \boldsymbol{\beta} &= (\mathbf{X}^T\mathbf{X})^{-1}(\mathbf{X}^T\mathbf{Y})& \mbox{pre-multiplying both sides by } (\mathbf{X}^T\mathbf{X})^{-1}\\ \end{array}$$ The last pre-multiplication is valid due to assumption 2 (i.e. $(\mathbf{X}^T\mathbf{X})^{-1}$ exists and is unique). Now we have the solution to $$\quad \frac{\partial S(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}= \mathbf{0} \implies \boldsymbol{\beta} = (\mathbf{X}^T\mathbf{X})^{-1}(\mathbf{X}^T\mathbf{Y}).$$ We should additionally check the second derivative matrix is positive definite to declare the above solution attaining the minimum of $S(\boldsymbol{\beta})$

$$\frac{\partial^2 S(\boldsymbol{\beta})}{\partial \boldsymbol{\beta} ~~\partial \boldsymbol{\beta^T}}= 2 (\mathbf{X}^T\mathbf{X})$$ which is always positive definite.

Residual Sum Of Squares

This choice of $\boldsymbol{\beta}$ is the least squares estimator. The resulting minimum value of $S(\boldsymbol{\beta})$ is called the residual sum of squares, denoted by $RSS$. Thus,

$$RSS=S(\boldsymbol{\hat{\beta}}) = \mathbf{Y}^T\mathbf{Y}-\mathbf{Y}^T\mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y}$$

i.e.

$$RSS=\mathbf{Y}^T\mathbf{Y}-\mathbf{Y}^T\mathbf{X}\boldsymbol{\hat{\beta}}.$$