17.1 Regularisation I: L1, L2, and Dropout
A common problem in machine learning is overfitting. This occurs when a model learns the training data too well, including its noise and random fluctuations, to the point that it performs poorly on new, unseen data. Regularization is a set of techniques used to combat overfitting by discouraging complex models.
The Problem of Complexity
A "complex" model often has very large weights. Large weights can cause the model's output to change drastically with small changes in the input, leading to erratic behavior. Regularization works by adding a penalty term to the loss function that penalizes large weights.
Original Loss: \( L = \text{Error}(y, \hat{y}) \)
Regularized Loss: \( L_{reg} = \text{Error}(y, \hat{y}) + \lambda \cdot R(W) \)
Here, \( R(W) \) is the regularization term (a function of the weights \( W \)), and \( \lambda \) is a hyperparameter that controls the strength of the penalty. A larger \( \lambda \) results in smaller weights and a simpler model.
L1 and L2 Regularization
L1 and L2 are the two most common types of regularization. They differ in how they penalize the weights.
- L2 Regularization (Ridge): The penalty is the sum of the squared values of the weights. $$ R(W) = \sum_{i} w_i^2 $$ L2 encourages weights to be small and distributed. It's the most common form of regularization.
- L1 Regularization (Lasso): The penalty is the sum of the absolute values of the weights. $$ R(W) = \sum_{i} |w_i| $$ L1 has a fascinating property: it can force some weights to be exactly zero, effectively performing feature selection by removing unimportant features from the model.
Interactive L1 vs. L2 Regularization
This visualization shows a simple linear regression model fitting a set of data points. You can adjust the regularization strength (\(\lambda\)) and see how L1 and L2 penalties affect the model's weights. Notice how L1 can shrink one of the weights to zero.
Weight 1 (Slope): {{weights.w1.toFixed(3)}}
Weight 2 (Quadratic): {{weights.w2.toFixed(3)}}
Dropout
Dropout is a completely different but very powerful regularization technique specific to neural networks. During training, for each forward pass, dropout randomly "drops out" (sets to zero) a fraction of the neurons in a layer.
This prevents neurons from co-adapting too much. It forces the network to learn more robust features that are useful in conjunction with many different random subsets of the other neurons. It's like forcing a team to work effectively even when some members are randomly absent, making each individual member more capable and less reliant on others.
During testing/inference, dropout is turned off, and the full network is used. To compensate for the fact that more neurons are active than during training, the outputs of the layer are scaled down by the dropout rate.
Dropout Visualization
The network below shows a simple feedforward layer. Click the "Apply Dropout" button to see how neurons are randomly deactivated at each step. The dropout rate determines the probability of a neuron being dropped.