Overfitting vs Underfitting

The two ways learning goes wrong

A good model learns the real pattern and works on new data. Two failures are common:

• Underfitting — the model is too simple. It misses the pattern and does badly on everything, even the training data.
• Overfitting — the model is too complex. It memorises the training data (including noise) and does badly on new data.

A study analogy

Underfitting is barely studying — you fail the practice and the exam. Overfitting is memorising the practice answers word for word — you ace the practice but flop on the real exam with new questions.

How to spot them: compare two scores

The trick is to look at the training score and the test score together.

Let us see the “High, Low” row happen for real. The code below loads a built-in flower dataset (load_iris), splits it, then trains a decision tree with no depth limit so it is free to memorise — and prints its score on the training data and on the hidden test data so we can compare the two.

# An overfitting decision tree, shown by its two scores
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)

deep = DecisionTreeClassifier()          # no depth limit -> can memorise
deep.fit(Xtr, ytr)
print('Train score:', round(deep.score(Xtr, ytr), 2))
print('Test score: ', round(deep.score(Xte, yte), 2))

Simple fixes

• Overfitting: get more data, use a simpler model (e.g. limit tree depth), or remove noisy features.
• Underfitting: use a more powerful model, add better features, or train longer.