Hyperparameter Tuning (GridSearchCV)

The settings you choose yourself

A model learns most things on its own — a line’s slope, a tree’s questions. But some settings you must choose before training: the k in KNN, the max_depth of a tree, the alpha of Ridge, the learning_rate of boosting. These chosen-in-advance settings are called hyperparameters.

So far you have tweaked these by hand, re-running and eyeballing the score. That is slow and easy to fool yourself with. Hyperparameter tuning automates the search.

Grid search: try every combination

GridSearchCV takes a grid of values you want to try and tests every combination, scoring each with cross-validation (so the result is trustworthy, not a lucky split). It then hands you the best combination. The “CV” in the name is exactly the cross-validation you already met.

Step by step, for a KNN where we want to try k = 1, 3, 5, 7:

1. List the values to try: { 'n_neighbors': [1, 3, 5, 7] }.
2. GridSearchCV trains a KNN for each value of k.
3. Each one is scored with cross-validation (e.g. 5 folds), giving a fair average.
4. It compares the averages and keeps the best k.
5. It reports the winning settings and best score.

from sklearn.model_selection import GridSearchCV
from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)

grid = { 'n_neighbors': [1, 3, 5, 7, 9] }
search = GridSearchCV(KNeighborsClassifier(), grid, cv=5)
search.fit(X, y)

print('Best k:    ', search.best_params_)
print('Best score:', round(search.best_score_, 3))

Searching two hyperparameters at once

Grid search really earns its keep with several hyperparameters, where hand-tuning combinations becomes hopeless. For a tree you might tune depth and the split criterion together:

from sklearn.model_selection import GridSearchCV
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)

grid = {
    'max_depth': [2, 3, 4, 5],
    'criterion': ['gini', 'entropy'],
}
search = GridSearchCV(DecisionTreeClassifier(random_state=0), grid, cv=5)
search.fit(X, y)

print('Best settings:', search.best_params_)
print('Best score:   ', round(search.best_score_, 3))

When the grid is too big: RandomizedSearchCV

Grid search tries every combination, so the count explodes fast: 5 settings × 4 settings × 3 settings = 60 combinations, each cross-validated. When the grid is huge, RandomizedSearchCV samples a fixed number of random combinations instead of all of them — usually finding a near-best answer in a fraction of the time.