15.4 Aspects of Neural Networks II: Initialization

The Importance of Initialization

The way we initialize the weights of a neural network can have a significant impact on the training process. A good initialization can speed up convergence and improve the final performance of the model. A poor initialization can lead to slow training or prevent the network from learning at all.

If all weights are initialized to zero, every neuron in a layer will learn the same features. This symmetry prevents the network from learning anything useful. If weights are initialized to be too large, the outputs of the activation functions can saturate, leading to vanishing or exploding gradients.

Common Initialization Strategies

The goal of a good initialization strategy is to set the initial weights such that the variance of the outputs of each layer is approximately equal to the variance of its inputs. This helps to keep the signal flowing properly through the network.

1. LeCun Initialization

One of the earlier and effective initialization schemes. It suggests drawing weights from a distribution with a standard deviation of:

\(\sigma = \sqrt{\frac{1}{n_{in}}}\)

where \(n_{in}\) is the number of input units to the layer (the "fan-in"). This is often used with activation functions like Tanh.

2. Xavier/Glorot Initialization

This is a very popular initialization method that works well with Sigmoid and Tanh activation functions. It aims to keep the variance of activations and backpropagated gradients the same across layers.

It draws weights from a distribution with a standard deviation of:

\(\sigma = \sqrt{\frac{2}{n_{in} + n_{out}}}\)

where \(n_{in}\) is the fan-in and \(n_{out}\) is the number of output units from the layer (the "fan-out").

3. He Initialization

This method is specifically designed for the ReLU activation function and its variants. Since ReLU sets half of the inputs to zero, it changes the variance of the outputs. He initialization accounts for this.

It draws weights from a distribution with a standard deviation of:

\(\sigma = \sqrt{\frac{2}{n_{in}}}\)

This is the recommended initialization for networks that use ReLU, which is the most common case in modern deep learning.

Bias Initialization

The biases are typically initialized to zero. However, in some cases, it can be beneficial to initialize them to a small constant value (e.g., 0.01), particularly when using ReLU units, to ensure that all neurons are active initially.

Normalization Layers

While good initialization is crucial, its effects can diminish as training progresses. Batch Normalization is a technique that helps to address this. It normalizes the inputs to each layer to have a mean of zero and a standard deviation of one.

Batch Normalization is added as a layer in the network. It helps to stabilize and speed up training, reduces the dependence on initialization, and can also act as a form of regularization. It has become a standard component in many deep learning architectures.