12.3 Smoothing splines

These estimators use a regularized regression over the natural spline basis: placing knots at all points \(x_1,...x_n\)

For the case of cubic splines, the coefficients are the minimization of

\[ ||y - G\beta||^2_2 + \lambda \beta^T \Omega \beta \]

where \(\Omega \in R^{n \times n}\) is the penalty matrix

\[ \Omega_{ij} = \int g''_i(t) g''_j(t) dt, \]

and i,j = 1,..,n

and \(\lambda \beta^T \Omega \beta\) is the regularization term used to shrink the components of \(\hat{\beta}\) towards 0. \(\lambda > 0\) is the tuning parameter (or smoothing parameter). Higher value of \(\lambda\), faster shrinkage (shrinking away basis functions)

Note
smoothing splines have similar fits as kernel regression.

Smoothing splines kernel regression
tuning parameter smoothing parameter \(\lambda\) bandwidth h