17.2 Parameter Estimation

• Definition: Parameter estimation, represented by $\hat{\beta}$, focuses on estimating the relationship between $y$ and $x$.

• Goal: The aim is consistency, ensuring that models perform well on the training data:

  $$E[\hat{f}] = f$$

• Challenges:

  • High-dimensional spaces can lead to covariance among variables and multicollinearity.
  • This leads to the bias-variance tradeoff (Hastie et al. 2009).

References

Hastie, Trevor, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. 2009. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Vol. 2. Springer.