Chapter 6 Analysis of Variance (ANOVA)

• Notation: \(y = X\beta + \epsilon\), \(\epsilon \sim N(0, \sigma^2)\). Let \(X_1 = 1\), \(X_m = X\) and \(X_{m+1} = I\). Suppose \(C(X_1)\subset C(X_2) \cdots \subset C(X_{m-1}) \subset C(X_m)\). Let \(P_j = P_{X_j}\) and \(r_j = rank(X_j), \forall j = 1, \ldots, m+1\)

  • Total sum of squares: \(SSTo = \sum_{i=1}^n (y_i - \bar y_.)^2 = y'(I-P_1)y = \sum_{j=1}^m y'(P_{j+1} - P_j)y\)
  • \(SSE = y'(I- P_X)y\)
  • Sum of squares: \(SS(2|1) = y'(P_2 - P_1)y, \ldots, SS(m|m-1) = y'(P_{m+1} - P_m)y\)
  • \(rank(P_{j+1} - P_{j}) = tr(P_{j+1}) - tr(P_j) = r_{j+1} - r_j\)
  • zero cross-products: \((P_{j+1} - P_j)(P_{l+1} - P_{l}) = 0\)
  • because \((\frac{P_{j+1} - P_j}{\sigma^2})(\sigma^2I)\) is idempotent, \(\frac{y'(P_{j+1}-P_j)y}{\sigma^2} \sim \chi_{r_{j+1}-r_j}^2\left(\frac{\beta'X'(P_{j+1}-P_j)X\beta}{2\sigma^2} \right)\) for all \(j = 1, \ldots, m\)
  • Mean squares: \(MS(j+1\mid j) = \frac{SS(j+1\mid j)}{r_{j+1}-r_j}\)

• ANOVA F statistics: \[ F_j = \frac{MS(j+1\mid j)}{MSE} = \frac{y'(P_{j+1} - P_j)y)/(r_{j+1} - r_j)}{y'(I-P_X)y/(n-r)} \sim F_{r_{j+1} - r_j, n-r}\left( \frac{\beta'X'(P_{j+1} - P_{j})X\beta}{2\sigma^2}\right) \]

  • \(F_j\) can be used to test $H_{0j}: = 0 $ vs. \(H_{Aj}: \frac{\beta'X'(P_{j+1} - P_{j})X\beta}{2\sigma^2} \neq 0\)
  • \(\frac{\beta'X'(P_{j+1} - P_{j})X\beta}{2\sigma^2}\) \(\Leftrightarrow\) \((P_{j+1}-P_j)X\beta = 0\) \(\Leftrightarrow\) \(P_j E(y) = P_{j+1}E(y)\) \(\Leftrightarrow\) \(P_{j+1}E(y) \in C(X_j)\)
  • \(C^*_j = P_{j+1} - P_j\) is not full rank so \((P_{j+1}-P_j)X\beta = 0\) is not a testable hypothesis. We can write \(H_{0j}\) as a testable hypothesis by replacing \(C^*_j\) with any matrix \(C_j\) whose \(q = r_{j+1} - r_j\) rows form a basis for the row space of \(C^*_j\).