3.7 The hat matrix

Let’s hop back to the matrix form of the normal equations for a minute: $\boldsymbol{b} = (\boldsymbol X' \boldsymbol X)^{-1}\boldsymbol X' \boldsymbol y$

Now, if we want to get actual predictions out of this vector of coefficients we’re estimating, we do it with multiplication: $$\boldsymbol{Xb}$$. But according to our equations, $\hat{\boldsymbol y}=\boldsymbol{Xb} = \boldsymbol X((\boldsymbol X' \boldsymbol X)^{-1}\boldsymbol X' \boldsymbol y)$ or rather: $\hat{\boldsymbol y}= \boldsymbol{Xb} = (\boldsymbol X(\boldsymbol X' \boldsymbol X)^{-1}\boldsymbol X')\boldsymbol{y}$ So the vector of predicted values is achieved by multiplying $$\boldsymbol{y}$$ with a matrix. This is a projection matrix; it projects the $$\boldsymbol{y}$$ vector down onto the column space of $$\boldsymbol{X}$$ where our predictions must lie. Because it’s the matrix that “turns $$\boldsymbol{y}$$ into $$\hat{\boldsymbol y}$$,” it’s called the hat matrix, $$\boldsymbol H$$. The hat matrix has some other cool features too:

• Square
• Symmetric
• Idempotent: $$\boldsymbol H \boldsymbol H = \boldsymbol H$$. This makes sense since it’s a projection matrix: once you project down into $$Col(\boldsymbol X)$$, you’re already there, so projecting again won’t do anything. So $$\boldsymbol{Hc} = \boldsymbol{HHc}$$ for any $$\boldsymbol c$$!