Chapter 5 Deployment and final words

For the modeling part, the best accuracy found is 77% and comes from the KNN model. Concerning the ROC metric and more precisely the AUC, the best model was the random forest model with a score of 85%. Even if these results open the door to a first interesting analysis tool for a bank employee, they do not make it possible to automate this classification task as there is room for accuracy improvement as well as specificity and sensitivity scores.

Regarding the dataset level analysis in the chapter of DALEX, The LDA model as well as the KNN model seem to have the most interesting residuals. They are closer to zero and therefore guarantee a lower volatility of the predictions. Globally, we do not observe any pattern in the residuals of our models.

In our models, certain variables play a crucial role in risk prediction. We found some explanations for some variables:

• chk_acct: The more money the customers have in their account, the less at risk they are.
• history: Payment history does not explain the risk. For our models, there is no correlation between being a bad payer and being classified as a risk.
• Log1pDurationstd: The longer the term of the credit, the more likely the customer is classified as being at risk.
• sav_acct: The more money the client has in his savings account, the less at risk he will be.
• Log1pAmountstd: We note that the amount of money loaned does not explain whether the client is at risk or not.
• education: The more educated is the customer, the less likely he will be considered as a good risk credit.

Among these variables, education and history do not seem to correspond to reality.

When using one of the prediction models to analyse the credit application, a bank employee could judge the customers according to their reliability. The employee will receive a first answer “good” or “bad.”

In the real life, it would be interesting to use either the KNN model, the naive bayes model or the random forest model. Nevertheless, as it is more important to predict bad risk customers, naive bayes model is better. Moreover, by removing variables whose results do not reflect reality (education and history), we could have better results with our three models. This way, we could choose the best one.

However, our models have not been able to achieve sufficient sensitivity to exclude good customers to allow manual analysis of remaining customers who would be potentially bad risk credits. Therefore, it would be necessary to double-check the results of our models. Nonetheless, it would still be important to analyse the money on the customers’accounts. If the term of the credit is important, the client should have sufficient financial resources to get a loan of money.