5  Appendix

5.1 Calculate loglikelihood base on least square

In linear regression with fixed $x$, the least square estimator and maximum likelihood estimator are equivalent. Therefore it is very easy to derive the loglikelihood of a given linear regression model from its least square estimation. Assume we have $$y_i={\beta }_0+{\beta }_1{x}_{1i}+{ϵ}_i$$

$$\begin{array}{rl}L& =\prod _{i=1}^n\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp -\frac{(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2}{2{\sigma }^2}\\ \ell & =\sum _{i=1}^n-\frac{1}{2}\log (2\pi {\sigma }^2)-\frac{1}{2{\sigma }^2}\sum _{i=1}^n(e_i^2)\\ & =-\frac{n}{2}\log (2\pi )-\frac{n}{2}\log ({\sigma }^2)-\frac{1}{2{\sigma }^2}\sum _{i=1}^n(e_i^2).\end{array}$$

The least square estimator and the normal maximum likelihood estimator of ${\sigma }^2$ is $${\hat{\sigma }}^2=s^2=\frac{1}{n-2}\sum _{i=1}^n(e_i^2),$$

thus we have $$\begin{array}{rl}\ell & =-\frac{n}{2}\log (2\pi )-\frac{n}{2}\log (\frac{1}{n-2})-\frac{n}{2}\log (\sum _{i=1}^n(e_i^2))-\frac{n-2}{2}\\ & =-\frac{n}{2}\log (2\pi )+\frac{n}{2}\log (n-2)-\frac{n}{2}\log (\sum _{i=1}^n(e_i^2))-\frac{n}{2}+1\\ & =-\frac{n}{2}[\log (2\pi )-\log (n-2)+\log (\sum _{i=1}^n(e_i^2))+1-\frac{2}{n}]\end{array}$$

1. View(getAnywhere(lrtest.default)$objs[[1]]), line 3
2. View(getAnywhere(logLik.lm)$objs[[1]]), line 22
3. View(getAnywhere(lrtest.default)$objs[[1]]), line 94, 102

5.2 Sum of squares decomposition for $F$ test

Assume we have $p$ predictors, $n$ observations, ${\mathit{y}}^{\prime }=[y_1,\cdots ,y_n]$, ${\mathit{\beta }}^{\prime }=[{\beta }_0,{\beta }_1,\cdots ,{\beta }_p]$,

$$\begin{array}{rl}\mathit{X}& =\left[\begin{array}{ccccc}1& x_{11}& x_{12}& \cdots & x_{1p}\\ 1& x_{21}& x_{22}& \cdots & x_{2p}\\ ⋮& & & & \\ 1& x_{n1}& x_{n2}& \cdots & x_{np}\end{array}\right].\end{array}$$ Because $$\begin{array}{rl}\mathit{X}\hat{\mathit{\beta }}& =\mathit{y}\\ {\mathit{X}}^{\prime }\mathit{X}\hat{\mathit{\beta }}& ={\mathit{X}}^{\prime }\mathit{y}\\ {\mathit{X}}^{\prime }(\mathit{X}\hat{\mathit{\beta }}-\mathit{y})& =0\\ {\mathit{X}}^{\prime }\mathit{e}& =0\\ \left[\begin{array}{cccc}1& 1& \cdots & 1\\ x_{11}& x_{21}& \cdots & x_{n1}\\ x_{12}& x_{22}& \cdots & x_{n2}\\ ⋮& & & \\ x_{1p}& x_{2p}& \cdots & x_{np}\end{array}\right]\times \left[\begin{array}{c}{e}_1\\ e_2\\ ⋮\\ e_n\end{array}\right]& =\mathbf{0}\\ \left[\begin{array}{c}\sum _{i=1}^n{e}_i\\ \sum _{i=1}^n{x}_{i1}{e}_i\\ \sum _{i=1}^n{x}_{i2}{e}_i\\ ⋮\\ \sum _{i=1}^n{x}_{ip}{e}_i\end{array}\right]& =\mathbf{0}.\end{array}$$ thus we have $$\begin{array}{rl}2\sum _{i=1}^n(y_i-{\hat{y}}_i)({\hat{y}}_i-\overline{y})& =2\sum _{i=1}^n{e}_i({\hat{y}}_i-\overline{y})\\ & =\sum _{i=1}^n{\hat{y}}_i{e}_i-\sum _{i=1}^n\overline{y}{e}_i\\ & ={\hat{\beta }}_0\sum _{i=1}^n{e}_i+{\hat{\beta }}_1\sum _{i=1}^n{x}_{i1}{e}_i+\cdots +{\hat{\beta }}_1\sum _{i=1}^n{x}_{ip}{e}_i-\overline{y}\sum _{i=1}^n{e}_i\\ & =0.\end{array}$$

Reference: Multiple Linear Regression (MLR) Handouts by Yibi Huang