15.4 Summary Table

Comparison of Variable Selection Methods

Method	Type	Approach	Key Criterion	Notes
Mallows’s C Statistic	Information Criterion	Subset Selection	Model Complexity vs Fit	Balances fit and simplicity.
Akaike Information Criterion (AIC)	Information Criterion	Model Selection	Minimizes AIC	Penalizes model complexity.
Bayesian Information Criterion (BIC)	Information Criterion	Model Selection	Minimizes BIC	Stronger penalty for complexity.
Hannan-Quinn Criterion (HQC)	Information Criterion	Model Selection	Minimizes HQC	Combines AIC and BIC features.
Minimum Description Length (MDL)	Information Criterion	Model Selection	Data + Model Encoding Costs	Focuses on encoding efficiency.
Prediction Error Sum of Squares (PRESS)	Error-Based	Cross-Validation	Minimizes Prediction Error	Measures predictive accuracy.
Best Subsets Algorithm	Exhaustive Search	Subset Selection	Best Fit Across Subsets	Considers all variable combinations.
Forward Selection	Stepwise	Add Variables	Significance Testing	Adds variables one at a time.
Backward Elimination	Stepwise	Remove Variables	Significance Testing	Removes variables iteratively.
Stepwise (Both Directions)	Stepwise	Add/Remove Variables	Significance Testing	Combines forward and backward methods.
Branch-and-Bound Algorithm	Optimized Search	Subset Selection	Efficient Subset Search	Avoids exhaustive search.
Recursive Feature Elimination (RFE)	Iterative Optimization	Feature Removal	Model Performance	Removes least important predictors.
Genetic Algorithms	Heuristic Search	Evolutionary Process	Fitness Function	Mimics natural selection for subsets.