Mallows’s C Statistic |
Information Criterion |
Subset Selection |
Model Complexity vs Fit |
Balances fit and simplicity. |
Akaike Information Criterion (AIC) |
Information Criterion |
Model Selection |
Minimizes AIC |
Penalizes model complexity. |
Bayesian Information Criterion (BIC) |
Information Criterion |
Model Selection |
Minimizes BIC |
Stronger penalty for complexity. |
Hannan-Quinn Criterion (HQC) |
Information Criterion |
Model Selection |
Minimizes HQC |
Combines AIC and BIC features. |
Minimum Description Length (MDL) |
Information Criterion |
Model Selection |
Data + Model Encoding Costs |
Focuses on encoding efficiency. |
Prediction Error Sum of Squares (PRESS) |
Error-Based |
Cross-Validation |
Minimizes Prediction Error |
Measures predictive accuracy. |
Best Subsets Algorithm |
Exhaustive Search |
Subset Selection |
Best Fit Across Subsets |
Considers all variable combinations. |
Forward Selection |
Stepwise |
Add Variables |
Significance Testing |
Adds variables one at a time. |
Backward Elimination |
Stepwise |
Remove Variables |
Significance Testing |
Removes variables iteratively. |
Stepwise (Both Directions) |
Stepwise |
Add/Remove Variables |
Significance Testing |
Combines forward and backward methods. |
Branch-and-Bound Algorithm |
Optimized Search |
Subset Selection |
Efficient Subset Search |
Avoids exhaustive search. |
Recursive Feature Elimination (RFE) |
Iterative Optimization |
Feature Removal |
Model Performance |
Removes least important predictors. |
Genetic Algorithms |
Heuristic Search |
Evolutionary Process |
Fitness Function |
Mimics natural selection for subsets. |