Chapter 2 Review of Linear Models

• General Linear Model (GLM): suppose $y = X\beta + \epsilon$ with $E(\epsilon) = 0$

  • distribution of $y$ is left unspecified
  • $E(y) \in \mathcal{C}(X)$

• Ordinary Least Squares Estimator (OLSE): $\hat y = P_X X = X(X'X)^-X'y$

• Orthogonal projection matrix $P_X$:

  • $P_X$ is symmetric, idempotent,
  • $P_XX = X$ and $X'P_X = P_X$
  • $rank(X)= rank(P_X) = tr(P_X)$
  • other properties:
    • $X'XA = X'XB \Leftrightarrow XA = XB$
    • $\forall (X'X)^- \Rightarrow X(X'X)^-X'X = X$, $X'X(X'X)^-X' = X'$
    • $A' = A$, $AGA = A \Rightarrow$ $AG'A = A$

• Estimable: if $C$ is any $q\times p$ matrix, the linear function of $\beta$ given by $C\beta$ is estimable if and only if $C = AX$ for some matrix $q\times n$ matrix A

  • OLSE of an estimable linear function $C\beta$ is $C(X'X)^-X'y$

• Normal equation: $X'Xb = X'y$

  • The OLSE of estimable $C\beta$ is $C\hat\beta$ where $\hat\beta$ is any solution for $b$ in the normal equations
  • $\hat\beta = (X'X)^-X'y$ is always a solution to the Normal Equations for any $(X'X)^-$
  • if $C\beta$ is estimable, then $C\hat\beta$ is the same for all solution $\hat\beta$ to the Normal equations and $C\hat\beta = AP_Xy$ where $C = AX$

• Gauss-Markov Model(GMM): suppose $y = X\beta + \epsilon$ with $E(\epsilon) = 0$ and $Var(\epsilon) = \sigma^2I$

• Gauss-Markov Theorem: the OLSE of an estimable function $C\beta$ is the BLUE of $C\beta$

  • an unbiased estimator of $\sigma^2$ under GMM is given by $\hat\sigma^2 = \frac{y'(I-P_X)y}{n-r}$

• Gauss-Markov Model with Normal Errors (GMMNE): suppose $y=X\beta+\epsilon$ with $\epsilon\sim N(0, \sigma^2I)$

  • GMMNE is useful for drawing statistical inferences regrading estimable $C\beta$
  • assume: 1. GMMNE 2. $C\beta$ is estimable 3. rand(C) = q and $d$ is a known $q\times 1$ vector. Then $H_0: C\beta = d$ is a testable hypothesis

• $C\hat\beta\sim N(C\beta, \sigma^2C(X'X)^-C')$ and $\hat\sigma^2\sim \frac{\sigma^2}{n-r}\chi_{n-r}^2$ are independent

• The F test statistic $$\begin{aligned} F & = (C\hat\beta-d)'[\widehat{Var}(C\hat\beta)]^{-1}(C\hat\beta - d)/q \\ & = \frac{(C\hat\beta-d)'[C(X'X)^-C']^{-1}(C\hat\beta-d)/q}{\hat\sigma^2} \\ & \sim F_{q, n-r}(\frac{(C\beta-d)'[C(X'X)^{-}C']^{-1}(C\beta-d)}{2\sigma^2}) \end{aligned}$$ Under the null hypothesis $H_0: C\beta = d$, the non-negative non-centrality parameter is 0.

• The t test statistic $$\begin{aligned} t &= \frac{c'\hat\beta-d}{\sqrt{\widehat{Var}(c'\beta)}} = \frac{c'\hat\beta-d}{\sqrt{\hat\sigma^2c'(X'X)^-c}} \\ & \sim t_{n-r}(\frac{c'\beta-d}{\sqrt{\sigma^2c'(X'X)^-c}}) \end{aligned}$$ Under the null hypothesis $H_0: C\beta = d$, the non-negative non-centrality parameter is 0.