17.2 Regularisation II: Early Stopping and Data Augmentation

Beyond penalizing weights or dropping out neurons, there are other, more procedural methods to prevent overfitting. Early stopping and data augmentation are two of the most practical and widely used techniques that don't require changing the loss function itself.

Early Stopping

Early stopping is perhaps the most intuitive form of regularization. The core idea is simple: stop training as soon as the model's performance on a validation set starts to degrade.

During training, we typically monitor two loss values: the training loss (on the data the model is learning from) and the validation loss (on a separate dataset the model doesn't see).

Initially, both training and validation loss will decrease.
The training loss will almost always continue to decrease as the model fits the training data more and more closely.
However, at a certain point, the validation loss will bottom out and start to increase. This is the moment of overfitting: the model has started learning noise from the training set that doesn't generalize to the validation set.

Early stopping simply means saving the model at the point where the validation loss is at its minimum, and then stopping the training process. This prevents the model from becoming overly specialized to the training data.

Conceptual Graph: Imagine a graph with 'Training Epochs' on the x-axis and 'Loss' on the y-axis.

The Training Loss line consistently goes down.

The Validation Loss line goes down, hits a minimum, and then starts to curve back up. The optimal point to stop training is at this minimum.

Pros:

Very simple to implement and computationally cheap.
Requires no modification of the model architecture or loss function.

Cons:

It requires a dedicated validation set, reducing the amount of data available for training.
It can sometimes stop training prematurely if the validation loss has a noisy trajectory with multiple dips.

Data Augmentation

Another highly effective way to reduce overfitting is to simply get more data. A model trained on more data is forced to learn more robust and general features. However, collecting and labeling new data can be expensive and time-consuming.

Data augmentation is a technique to artificially create new training data from existing data. This is done by applying a series of random (but realistic) transformations to the training samples.

For image data, common augmentation techniques include:

Flipping: Horizontally or vertically flipping the image.
Rotation: Rotating the image by a small angle.
Scaling: Zooming in or out.
Cropping: Randomly cropping a section of the image.
Translation: Shifting the image horizontally or vertically.
Color Jitter: Adjusting the brightness, contrast, or saturation.

By generating these new samples during training, the model is exposed to a much wider variety of data. It learns to be invariant to changes in position, orientation, and lighting, which are characteristics of a robust model. For example, a cat is still a cat whether it's on the left or right side of an image, or if the image is slightly darker.

Pros:

Can significantly improve model generalization and reduce overfitting.
Relatively easy to implement with modern deep learning libraries.

Cons:

The transformations must be realistic for the domain. (e.g., flipping a '6' upside down to get a '9' would be incorrect for digit recognition).
Can increase training time due to the overhead of performing transformations.

Conclusion

Early stopping and data augmentation are essential tools in the deep learning practitioner's toolkit. They are often used in combination with other regularization methods like L2 regularization and dropout to build robust models that perform well on unseen data.