10.3 Multiple Models Comparison
Multiple model comparison is also called Cross Model Validation. Here the model refers to completely different algorithms. The idea is to use multiple models constructed from the same training dataset and validated using the same verification dataset to find out the performance of the different models.
We have already used the technique to compare our decision tree models and random forest models. Cross-model verification has a broader meaning that refers to the comparison between different models produced by the different algorithms or completely different approaches such as decision tree against random forest or decision tree against Support Vector Machine (SVM).
To demonstrate cross model validation, let us produce a few more models with completely different algorithms with the same predictors as much as possible. Let us use Sex, Fare_pp, Pclass, Title, Age_group, Group_size, Ticket_class, Embarked as predictors.
Regression Model for Titanic
Logistic regression is a classification, not a regression algorithm. It predicts the probability of occurrence of an event by fitting data to a logit function. Hence, it is also known as logit regression (VIDHYA 2015). Since it predicts the probability, its output values lie between 0 and 1, we can simply separate (or normalise) them by setting a threshold like (> 0.5).
LR_Model <- glm(formula = Survived ~ Pclass + Title + Sex + Age_group + Group_size + Ticket_class + Fare_pp + Embarked, family = binomial, data = trainData) #summary(LR_Model_CV) ### Validate on trainData Valid_trainData <- predict(LR_Model, newdata = trainData, type = "response") #prediction threshold Valid_trainData <- ifelse(Valid_trainData > 0.5, 1, 0) # set binary #produce confusion matrix confusion_Mat<- confusionMatrix(as.factor(trainData$Survived),as.factor(Valid_trainData)) # accuracy on traindata Regression_Acc_Train <- round(confusion_Mat$overall["Accuracy"]*100,2) paste('Model Train Accuracy =', Regression_Acc_Train)
##  "Model Train Accuracy = 84.97"
### Validate on validData validData_Survived_predicted <- predict(LR_Model, newdata = validData, type = "response") validData_Survived_predicted <- ifelse(validData_Survived_predicted > 0.5, 1, 0) # set binary prediction threshold conMat<- confusionMatrix(as.factor(validData$Survived),as.factor(validData_Survived_predicted)) Regression_Acc_Valid <-round(conMat$overall["Accuracy"]*100,2) paste('Model Valid Accuracy =', Regression_Acc_Valid)
##  "Model Valid Accuracy = 79.89"
## Area under the curve: 0.832
#predict on test test$Survived <- predict(LR_Model, newdata = test, type = "response") test$Survived <- ifelse(test$Survived > 0.5, 1, 0) # set binary prediction threshold submit <- data.frame(PassengerId = test$PassengerId, Survived = as.factor(test$Survived)) write.csv(submit, file = "./data/LG_model1_CV.CSV", row.names = FALSE) # Kaggle test accuracy score:0.76555 # record accuracy Regr_Acc <- c(Regression_Acc_Train, Regression_Acc_Valid, 0.76555) acc_names <- c("Train Accu", "Valid Accu", "Test Accu") names(Regr_Acc) <- acc_names Regr_Acc
## Train Accu Valid Accu Test Accu ## 84.97000 79.89000 0.76555
Support Vector Machine Model for Titanic
Let us also consider a support vector machine (SVM) model (Cortes and Vapnik 1995). We use the C-classification mode. Again, we fit a model with the same set of attributes as in the logistic regression model. We use function
svm() from the
e1071 package (https://cran.r-project.org/web/packages/e1071/e1071.pdf).
We could try to tune the two parameters of the SVM model
cost, find and select the best parameters (see exercise).
We then use the best model to make predictions. The results of the model are collected for comparison.
#load library library(e1071) # fit the model using default parameters SVM_model<- svm(Survived ~ Pclass + Title + Sex + Age_group + Group_size + Ticket_class + Fare_pp + Deck + HasCabinNum + Embarked, data=trainData, kernel = 'radial', type="C-classification") #summary(SVM_model) ### Validate on trainData Valid_trainData <- predict(SVM_model, trainData) #produce confusion matrix confusion_Mat<- confusionMatrix(as.factor(trainData$Survived), as.factor(Valid_trainData)) # output accuracy AVM_Acc_Train <- round(confusion_Mat$overall["Accuracy"]*100,4) paste('Model Train Accuracy =', AVM_Acc_Train)
##  "Model Train Accuracy = 84.4101"
### Validate on validData validData_Survived_predicted <- predict(SVM_model, validData) #produce confusion matrix conMat<- confusionMatrix(as.factor(validData$Survived), as.factor(validData_Survived_predicted)) # output accuracy AVM_Acc_Valid <- round(conMat$overall["Accuracy"]*100,4) paste('Model Valid Accuracy =', AVM_Acc_Valid)
##  "Model Valid Accuracy = 78.2123"
### make prediction on test # SVM failed to produce a prediction on test because test has Survived col and it has value NA. A work around is assign it with a num like 1. test$Survived <-1 # predict results on test Survived <- predict(SVM_model, test) solution <- data.frame(PassengerId=test$PassengerId, Survived =Survived) write.csv(solution, file = './data/svm_predicton.csv', row.names = F) # prediction accuracy on test SVM_Acc <- c(AVM_Acc_Train, AVM_Acc_Valid, 0.78947) names(SVM_Acc) <- acc_names # print out SVM_Acc
## Train Accu Valid Accu Test Accu ## 84.41010 78.21230 0.78947
Neural Network Models
Neural networks are a rapidly developing paradigm for information processing based loosely on how neurons in the brain process information. A neural network consists of multiple layers of nodes, where each node performs a unit of computation and passes the result onto the next node. Multiple nodes can pass inputs to a single node and vice versa.
The neural network also contains a set of weights, which can be refined over time as the network learns from sample data. The weights are used to describe and refine the connection strengths between nodes.
Neural Network with one hidden layer utilizing all features.
# load library library(nnet) # train the model xTrain = train[ , c("Survived", "Pclass","Title", "Sex","Age_group","Group_size", "Ticket_class", "Fare_pp", "Deck", "HasCabinNum", "Embarked")] NN_model1 <- nnet(Survived ~ ., data = xTrain, size=10, maxit=500, trace=FALSE) #How do we do on the training data? nn_pred_train_class = predict(NN_model1, xTrain, type="class" ) # yields "0", "1" nn_train_pred = as.numeric(nn_pred_train_class ) #transform to 0, 1 confusion_Mat<-confusionMatrix(as.factor(nn_train_pred), train$Survived) # output accuracy NN_Acc_Train <- round(confusion_Mat$overall["Accuracy"]*100,4) paste('Model Train Accuracy =', NN_Acc_Train)
##  "Model Train Accuracy = 89.2256"
#How do we do on the valid data? nn_pred_valid_class = predict(NN_model1, validData, type="class" ) # yields "0", "1" nn_valid_pred = as.numeric(nn_pred_valid_class ) #transform to 0, 1 confusion_Mat<-confusionMatrix(as.factor(nn_valid_pred), validData$Survived) # output accuracy NN_Acc_Valid <- round(confusion_Mat$overall["Accuracy"]*100,4) paste('Model valid Accuracy =', NN_Acc_Valid)
##  "Model valid Accuracy = 87.7095"
#make a prediction on test data nn_pred_test_class = predict(NN_model1, test, type="class" ) # yields "0", "1" nn_pred_test = as.numeric(nn_pred_test_class ) #transform to 0, 1 solution <- data.frame(PassengerId=test$PassengerId, Survived = nn_pred_test) write.csv(solution, file = './data/NN_predicton.csv', row.names = F) ### # 0.8934,0.8547, 0.71052 NN_Acc <- c(NN_Acc_Train, NN_Acc_Valid, 0.71052) names(NN_Acc) <- acc_names NN_Acc
## Train Accu Valid Accu Test Accu ## 89.22560 87.70950 0.71052
Comparision among Different Models
Let us compare the different models we have produced and see which one has a better prediction accuracy on the test dataset. We will use our best prediction accuracy on the test dataset for decision tree and random forest models.
library(tidyr) Model <- c("Regression","SVM","NN", "Decision tree", "Random Forest") Train <- c(Regression_Acc_Train, AVM_Acc_Train, NN_Acc_Train, 82.72, 83.16) Valid <- c(Regression_Acc_Valid, AVM_Acc_Valid, NN_Acc_Valid, 81.01, 92) Test <- c(76.56, 78.95, 71.05, 77.75, 78.95) df1 <- data.frame(Model, Train, Valid, Test) knitr::kable(df1, longtable = TRUE, booktabs = TRUE, digits = 2, col.names =c("Models", "Accuracy on Train", "Accuracy on Valid","Accuracy on Test"), caption = 'The Comparision among 3 Machine Learning Models' )
|Models||Accuracy on Train||Accuracy on Valid||Accuracy on Test|
From the above table and plot, we can see that multiple models cross-validation does not provide a conclusive answer on which model to use for real applications or production. Ideally, we would choose the model that has higher accuracy on trainData and validData. From the table 10.2, we should choose model NN since it has the highest train accuracy (91.13%) and the second highest validation accuracy (87.71%), however, it has the lowest test accuracy (71.05%). Another possible logic would be to choose the highest validation accuracy and ignore the train accuracy, in this case, we would choose Random Forest model since it has the highest validation accuracy (92.00%), and the highest test accuracy among the models (78.95%). However, the SVM model also has 78.95% test accuracy but its validation accuracy is the lowest.
It reveals an unpleasant fact - there is no model which be a certainty that its performance on unseen data can be ensured by the cross validation. The CV can only be used to spot and discover problems but not solutions.
Cortes, Corinna, and Vladimir Vapnik. 1995. “Support-Vector Networks.” Machine Learning 20. https://link.springer.com/article/10.1007/BF00994018.
VIDHYA, ANALYTICS. 2015. “Simple Guide to Logistic Regression in R and Python.” https://www.analyticsvidhya.com/blog/2015/11/beginners-guide-on-logistic-regression-in-r/.