5.7 Ridge Regression

When we have the Collinearity problem, we could use the Ridge regression.

The main problem with multicollinearity is that \(\mathbf{X'X}\) is “ill-conditioned”. The idea for ridge regression: adding a constant to the diagonal of \(\mathbf{X'X}\) improves the conditioning

\[ \mathbf{X'X} + c\mathbf{I} (c>0) \]

The choice of c is hard. The estimator

\[ \mathbf{b}^R = (\mathbf{X'X}+c\mathbf{I})^{-1}\mathbf{X'y} \]

is biased.

• It has smaller variance than the OLS estimator; as c increases, the bias increases but the variance decreases.
• Always exists some value of c for which the ridge regression estimator has a smaller total MSE than the OLS
• The optimal c varies with application and data set.
• To find the “optimal” \(c\) we could use “ridge trace”.

We plot the values of the \(p - 1\) parameter estimates for different values of c, simultaneously.

• Typically, as c increases toward 1 the coefficients decreases to 0.
• The values of the VIF tend to decrease rapidly as c gets bigger than 0. The VIF values begin to change slowly as \(c \to 1\).
• Then we can examine the ridge trace and VIF values and chooses the smallest value of c where the regression coefficients first become stable in the ridge trace and the VIF values have become sufficiently small (which is very subjective).
• Typically, this procedure is applied to the standardized regression model.