11.3 Tuning Model’s Parameters

Tuning model parameters is a parameter optimization problem (Dalpiaz 2021). Depending on the models, the adjustable parameters can be different completely. For example, the decision tree has two adjustable parameters: complexity parameter (CP) and tune length (TL). CP tells the algorithm to stop when the measure (generally is accuracy) does not improve by this factor. TL tells how many instances to use for training. SVM models, as another example, also have two adjustable parameters cost and gamma. The cost, is a parameter that controls the trade-off between the classification of training points and a smooth decision boundary. It suggests the model chooses data points as a support vector. If the value of cost is large, then the model choose more data points as a support vector and we get a higher variance and lower bias, which may lead to the problem of overfitting; If the value of cost is small, then the model will choose fewer data points as a support vector and get a lower variance and high bias. Gamma defines how far the influence of a single training example reaches. If the value of Gamma is high, then the decision boundary will depend on the points close to the decision boundary and the nearer points carry more weights than far away points due to which the decision boundary becomes more wiggly. If the value of Gamma is low, then the far-away points carry more weights than the nearer points and thus the decision boundary becomes more like a straight line.

We will continue use RF model as an example to demonstrate the parameter tuning process. RF has many parameters that can be adjusted but the two main tuning parameters are mtry and ntree.

  • mtry: Number of variables randomly selected as testing conditions at each split of decision trees. default value is sqr(col). Increasing mtry generally improves the performance of the model as each node has a higher number of options to be considered. However, this is not necessarily true as this decreases the diversity of individual trees. At the same time, it will decrease the speed. Hence, it needs to strike the right balance.

  • ntree: Number of trees to grow. the default value is 500. A higher number of trees give you better performance but makes your code slower. You should choose as high a value as your processor can handle because this makes your predictions stronger and more stable.

In the rest of the section, we demonstrate the process of using CV to fine-tune RF model’s parameters mtry and ntree. In general, different optimization strategies can be used to find a model’s optimal parameters. The two most commonly used methods for RF are Random search and Grid search.

Random Search. Define a search space as a bounded domain of parameter values and randomly sample points in that domain.

Grid Search. Define a search space as a grid of parameter values and evaluate every position in the grid.

Let us try them one at a time.

References

Dalpiaz, David. 2021. Tune Machine Learning Algorithms in R. otexts. https://machinelearningmastery.com/tune-machine-learning-algorithms-in-r/.


  1. Not all machine learning algorithms are available in caret for tuning. The choice of parameters was decided by the developers of the package. Only those parameters that have a large effect are available for tuning in caret. For the RF method, only mtry parameter is available in caret for tuning. The reason is its effect on the final accuracy and that it must be found empirically for a dataset↩︎

  2. These are generally used ntree values. For demonstration purposes we only choose these values, you can try more different values.↩︎