Simple linear regression with a slope and intercept term

\(y_i = \alpha + \beta x_i + \epsilon_i\), where \(\epsilon_i \sim N(0,\sigma^2)\).

\(\newline\) The method of Maximum Likelihood will be used to find the estimators for \(\hat{\alpha}\) and \(\hat{\beta}\).

\[ y_i \sim N(\alpha+\beta x_i,\sigma^2)\]

Likelihood and Log-likelihood function

We are assuming \(y_i|x_i \sim N(\alpha + \beta x_i, \sigma^2)\) and that our observations are independent and the probability density function of the normal distribution

\[\begin{eqnarray*} p(y_i) &=& \frac{1}{\sqrt{2\pi}\sigma}\mathrm{exp}(-\frac{1}{2\sigma^2}\{y_i-(\alpha+\beta x_i)\}^2)\\ \end{eqnarray*}\]

Hence \[\begin{eqnarray*} L(\alpha,\beta) &\propto& \prod_{i=1}^n \frac{1}{{\sqrt{2\pi}}\sigma}\mathrm{exp}(-\frac{1}{2\sigma^2}\{y_i-(\alpha+\beta x_i)\}^2)\\ &=&(2\pi)^\frac{-n}{2}\sigma^{-n}\mathrm{exp}(-\frac{1}{2\sigma^2}\sum_{i=1}^n\{y_i-(\alpha+\beta x_i)\}^2)\\ \\ l(\alpha,\beta)&=& -\frac{n}{2}\mathrm{log_e}(2\pi)-n\mathrm{log_e}\sigma-\frac{1}{2\sigma^2}\sum_{i=1}^n\{y_i-(\alpha+\beta x_i)\}^2\\ \end{eqnarray*}\]

\[\begin{eqnarray*} \sum_{i=1}^n\{y_i-(\alpha+\beta x_i)\}^2 &=&\sum_{i=1}^n(y_i^2-\alpha y_i-\beta x_iy_i-\alpha y_i+\alpha^2+\alpha\beta x_i-\beta x_iy_i+\alpha\beta x_i+\beta^2x_i^2)\\ &=&\sum_{i=1}^n(y_i^2-2\alpha y_i+\alpha^2-2\beta x_iy_i+2\alpha\beta x_i+\beta^2x_i^2)\\ \end{eqnarray*}\]

\[\begin{eqnarray*} \frac{\partial l}{\partial \alpha}&=&-\frac{1}{2\sigma^2}\sum_{i=1}^n(-2y_i+2\alpha+2\beta x_i) \\ \frac{\partial l}{\partial \beta}&=&-\frac{1}{2\sigma^2}\sum_{i=1}^n( -2x_iy_i+2\alpha x_i+2\beta x_i^2) \end{eqnarray*}\]

\[\frac{\partial l}{\partial \alpha}=\frac{\partial l}{\partial \beta}=0\]

\[\begin{eqnarray*} \sum_iy_i-n\alpha-\beta\sum_ix_i &=& 0\\ \sum_ix_iy_i-\alpha\sum_ix_i-\beta\sum_ix_i^2 &=& 0\\ \end{eqnarray*}\]

\[\begin{eqnarray*} -n\alpha &=& -\sum_iy_i+\beta\sum_ix_i\\ \hat{\alpha} &=& \bar{y}-\hat{\beta}\bar{x} \end{eqnarray*}\]

\[\begin{eqnarray*} -\beta\sum_ix_i^2-\alpha\sum_ix_i &=& -\sum_ix_iy_i\\ \beta\sum_i x_i^2+(\bar{y}-\beta\bar{x})\sum_ix_i &=& \sum_ix_iy_i\\ \beta\sum_i x_i^2+(\frac{\sum_i y_i}{n}-\beta\frac{\sum_ix_i}{n})\sum_ix_i &=& \sum_ix_iy_i\\ \beta\sum_i x_i^2+\frac{\sum_i y_i\sum_ix_i}{n}-\beta\frac{\sum_ix_i\sum_ix_i}{n}&=& \sum_ix_iy_i\\ \beta(\sum_ix_i^2-(\sum_ix_i)^2/n) &=& \sum_i x_iy_i-(\sum_ix_i\sum_iy_i)/n\\ \beta &=& \frac{\sum_i x_iy_i-(\sum_ix_i\sum_iy_i)/n}{\sum_ix_i^2-(\sum_ix_i)^2/n}\\ \hat{\beta} &=& \frac{S_{xy}}{S_{xx}} \end{eqnarray*}\]

\[ \frac{\partial^2 l}{\partial \alpha^2}=-\frac{1}{\sigma^2}\sum_{i=1}^n 1, \quad \frac{\partial^2 l}{\partial \beta^2}=-\frac{1}{\sigma^2}\sum_{i=1}^nx_i^2, \quad \frac{\partial^2 l}{\partial \alpha\beta}=-\frac{1}{\sigma^2}\sum_{i=1}^nx_i \]

\[\left( \begin{array}{cc} \frac{\partial^2 l}{\partial \alpha^2} &\frac{\partial^2 l}{\partial \alpha\beta}\\ \frac{\partial^2 l}{\partial \beta\alpha} & \frac{\partial^2 l}{\partial \beta^2}\\ \end{array} \right) = -\frac{1}{\sigma^2}\left( \begin{array}{cc} n & \sum_{i=1}^nx_i \\ \sum_{i=1}^nx_i & \sum_{i=1}^nx_i^2\\ \end{array} \right)\]

Matrix of second derivatives is negative definite, hence we have a maximum.