- Some ML methods are less flexible (shape of f), e.g., linear regression can only generate linear functions (Example: Prediction: Linear model (Visualization))
- James et al. (2013, 25), Fig. 2.7. provides an overview
- Q: Why would we ever choose to use a more restrictive method instead of a very flexible approach?
- Q: Is high flexibility always desirable?
- Debate around interpretable machine learning
- Low flexibility: Better interpretation if aim is inference (association between predictor and outcome) BUT potentially worse prediction
- High flexibility: Can also yield worse predictions because of overfitting (counterintuitive!)
James, Gareth, Daniela Witten, Trevor Hastie, and Robert Tibshirani. 2013. An Introduction to Statistical Learning: With Applications in R. Springer Texts in Statistics. Springer.