16.2 Parameter Estimation

  • Definition: Parameter estimation, represented by \(\hat{\beta}\), focuses on estimating the relationship between \(y\) and \(x\).

  • Goal: The aim is consistency, ensuring that models perform well on the training data:

    \[ E[\hat{f}] = f \]

  • Challenges:

    • High-dimensional spaces can lead to covariance among variables and multicollinearity.
    • This leads to the bias-variance tradeoff (Hastie et al. 2009).


Hastie, Trevor, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. 2009. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Vol. 2. Springer.