9.8 Resampling methods (7): Some caveats

• Beware: Conceptual confusion
  • Cross-validation four meanings in political science (Neunhoeffer and Sternberg 2019, 102)
    • 1. Validating new measures or instruments
    • Resampling:
      • 2. Robustness check of coefficents across data subsets
      • 3. procedure to obtain estimate of true error (unseen data)
      • 4. procedure of model tuning (e.g., parameter tuning, feature selection, up-/down-sampling of imbalanced data prior to training)
• Cross-validation for tuning vs. test error rate
  • “A problematic use of cross-validation occurs when a single cross-validation procedure is used for model tuning and to estimate true error at the same time […]. Ignoring this can lead to serious misreporting of performance measures. If the goal of cross-validation is to obtain an estimate of true error, every step involved in training the model (including (hyper-) parameter tuning, feature selection or up-/down-sampling) has to be performed on each of the training folds of the cross-validation procedure. Hastie, Tibshirani, and Friedman (2011, 245) refer to this problem as the wrong way of doing cross-validation. We take down-sampling of imbalanced data as a simple example of this problem.” (Neunhoeffer and Sternberg 2019, 102)

References

Neunhoeffer, Marcel, and Sebastian Sternberg. 2019. “How Cross-Validation Can Go Wrong and What to Do about It.” Polit. Anal. 27 (1): 101–6.