15.3 Aspects of Neural Networks I: Hyperparameters

What are Hyperparameters?

In machine learning, a hyperparameter is a parameter whose value is set before the learning process begins. These are the "knobs" we can turn to configure the model's architecture and the training process. They are distinct from the model's internal parameters (like weights and biases), which are learned from the data.

Choosing the right hyperparameters is crucial for training a good model and is often one of the most challenging parts of building a neural network.

Key Hyperparameters in Deep Learning

1. Learning Rate (\(\eta\))

This is arguably the most important hyperparameter. It controls how much we adjust the weights in response to the estimated error each time they are updated.

Too low: Training will be very slow, and the model might get stuck in a local minimum.
Too high: The training process can diverge, with the loss increasing instead of decreasing. The model may "overshoot" the minimum.

Techniques like learning rate schedules are often used, where the learning rate is gradually decreased during training.

2. Number of Hidden Layers and Units

These hyperparameters define the architecture of the network.

Number of Layers: A deeper network (more layers) can learn more complex functions and hierarchies of features. However, it is also more prone to overfitting and can be harder to train (e.g., due to vanishing gradients).
Number of Units per Layer: A wider layer (more units) can learn more features at that level. Too few units can lead to underfitting, while too many can lead to overfitting and increased computational cost.

A common practice is to start with a relatively small network and gradually increase its size.

3. Regularization

Regularization techniques are used to prevent overfitting, where the model performs well on the training data but poorly on unseen data.

L1 and L2 Regularization: These add a penalty to the loss function based on the magnitude of the weights. L2 regularization (also known as weight decay) is more common. It encourages the network to use smaller weights.
\(L_{reg} = L_{original} + \lambda \sum_i w_i^2\) (for L2)
Dropout: During training, dropout randomly sets a fraction of neuron outputs to zero at each update step. This forces the network to learn more robust features that are not dependent on any single neuron.

4. Batch Size

This is the number of training examples used in one iteration (i.e., one forward and backward pass).

Small batch size: Can offer a regularizing effect and lower memory usage, but the gradient estimates are noisy.
Large batch size: Provides a more accurate estimate of the gradient, but can be computationally expensive and may converge to sharp minima, which can have poorer generalization.

Hyperparameter Tuning

Finding the best combination of hyperparameters is often an empirical process. Common methods include:

Grid Search: Systematically tries all combinations of a predefined set of hyperparameter values.
Random Search: Randomly samples from the hyperparameter space. It is often more efficient than grid search.
Bayesian Optimization: A more advanced method that builds a probabilistic model of the function mapping from hyperparameter values to the objective evaluated on a validation set.