16.2 Parameter Estimation

• Definition: Parameter estimation, represented by \(\hat{\beta}\), focuses on estimating the relationship between \(y\) and \(x\).

• Goal: The aim is consistency, ensuring that models perform well on the training data:

  \[ E[\hat{f}] = f \]

• Challenges:

  • High-dimensional spaces can lead to covariance among variables and multicollinearity.
  • This leads to the bias-variance tradeoff (Hastie et al. 2009).

References

Hastie, Trevor, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. 2009. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Vol. 2. Springer.