Chapter 2 Review of Linear Models

• General Linear Model (GLM): suppose \(y = X\beta + \epsilon\) with \(E(\epsilon) = 0\)

  • distribution of \(y\) is left unspecified
  • \(E(y) \in \mathcal{C}(X)\)

• Ordinary Least Squares Estimator (OLSE): \(\hat y = P_X X = X(X'X)^-X'y\)

• Orthogonal projection matrix \(P_X\):

  • \(P_X\) is symmetric, idempotent,
  • \(P_XX = X\) and \(X'P_X = P_X\)
  • \(rank(X)= rank(P_X) = tr(P_X)\)
  • other properties:
    • \(X'XA = X'XB \Leftrightarrow XA = XB\)
    • \(\forall (X'X)^- \Rightarrow X(X'X)^-X'X = X\), \(X'X(X'X)^-X' = X'\)
    • \(A' = A\), \(AGA = A \Rightarrow\) \(AG'A = A\)

• Estimable: if \(C\) is any \(q\times p\) matrix, the linear function of \(\beta\) given by \(C\beta\) is estimable if and only if \(C = AX\) for some matrix \(q\times n\) matrix A

  • OLSE of an estimable linear function \(C\beta\) is \(C(X'X)^-X'y\)

• Normal equation: \(X'Xb = X'y\)

  • The OLSE of estimable \(C\beta\) is \(C\hat\beta\) where \(\hat\beta\) is any solution for \(b\) in the normal equations
  • \(\hat\beta = (X'X)^-X'y\) is always a solution to the Normal Equations for any \((X'X)^-\)
  • if \(C\beta\) is estimable, then \(C\hat\beta\) is the same for all solution \(\hat\beta\) to the Normal equations and \(C\hat\beta = AP_Xy\) where \(C = AX\)

• Gauss-Markov Model(GMM): suppose \(y = X\beta + \epsilon\) with \(E(\epsilon) = 0\) and \(Var(\epsilon) = \sigma^2I\)

• Gauss-Markov Theorem: the OLSE of an estimable function \(C\beta\) is the BLUE of \(C\beta\)

  • an unbiased estimator of \(\sigma^2\) under GMM is given by \(\hat\sigma^2 = \frac{y'(I-P_X)y}{n-r}\)

• Gauss-Markov Model with Normal Errors (GMMNE): suppose \(y=X\beta+\epsilon\) with \(\epsilon\sim N(0, \sigma^2I)\)

  • GMMNE is useful for drawing statistical inferences regrading estimable \(C\beta\)
  • assume: 1. GMMNE 2. \(C\beta\) is estimable 3. rand(C) = q and \(d\) is a known \(q\times 1\) vector. Then \(H_0: C\beta = d\) is a testable hypothesis

• \(C\hat\beta\sim N(C\beta, \sigma^2C(X'X)^-C')\) and \(\hat\sigma^2\sim \frac{\sigma^2}{n-r}\chi_{n-r}^2\) are independent

• The F test statistic \[ \begin{aligned} F & = (C\hat\beta-d)'[\widehat{Var}(C\hat\beta)]^{-1}(C\hat\beta - d)/q \\ & = \frac{(C\hat\beta-d)'[C(X'X)^-C']^{-1}(C\hat\beta-d)/q}{\hat\sigma^2} \\ & \sim F_{q, n-r}(\frac{(C\beta-d)'[C(X'X)^{-}C']^{-1}(C\beta-d)}{2\sigma^2}) \end{aligned} \] Under the null hypothesis \(H_0: C\beta = d\), the non-negative non-centrality parameter is 0.

• The t test statistic \[ \begin{aligned} t &= \frac{c'\hat\beta-d}{\sqrt{\widehat{Var}(c'\beta)}} = \frac{c'\hat\beta-d}{\sqrt{\hat\sigma^2c'(X'X)^-c}} \\ & \sim t_{n-r}(\frac{c'\beta-d}{\sqrt{\sigma^2c'(X'X)^-c}}) \end{aligned} \] Under the null hypothesis \(H_0: C\beta = d\), the non-negative non-centrality parameter is 0.