15.2 Deep Learning Fundamentals

Training a Neural Network

The process of "learning" in a neural network involves adjusting the weights and biases so that the network's output gets closer to the desired output. This is an optimization problem, where we want to find the parameters that minimize a "loss" or "cost" function.

The Loss Function

A loss function measures how far the network's prediction is from the true target. The choice of loss function depends on the task:

Mean Squared Error (MSE): Commonly used for regression tasks. It measures the average squared difference between the predicted and actual values.
\(L = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2\)
Cross-Entropy Loss: Used for classification tasks. It measures the performance of a classification model whose output is a probability value between 0 and 1.
\(L = -\frac{1}{n} \sum_{i=1}^{n} [y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i)]\)

Gradient Descent

To minimize the loss, we use an algorithm called Gradient Descent. The idea is to calculate the gradient of the loss function with respect to each weight and bias in the network. The gradient is a vector that points in the direction of the steepest increase of the loss function.

We then take a small step in the opposite direction of the gradient to move towards a lower loss. The size of this step is controlled by the learning rate, \(\eta\).

For a weight \(w\), the update rule is:

\(w_{new} = w_{old} - \eta \frac{\partial L}{\partial w}\)

This process is repeated iteratively until the loss converges to a minimum.

Backpropagation

Calculating the gradients for all the parameters in a deep network can be complex. Backpropagation is a clever algorithm that makes this process efficient.

It works by first performing a "forward pass," where an input is fed through the network to compute the output and the loss. Then, it performs a "backward pass," where the gradient of the loss is propagated backward from the output layer to the input layer.

The chain rule from calculus is used to calculate the gradient of the loss with respect to the weights of each layer. For a simple network with one hidden layer, the gradient for a weight in the first layer depends on the gradients from the second layer.

Backpropagation allows us to compute all the necessary gradients in a single pass, making it feasible to train very deep networks.

Optimization Algorithms

While standard gradient descent works, several more advanced optimization algorithms are commonly used to speed up training and improve performance.

Stochastic Gradient Descent (SGD): Instead of using the entire dataset to compute the gradient at each step, SGD uses a single training example (or a small mini-batch). This makes the updates much faster and can help escape local minima.
Momentum: This method helps accelerate SGD in the relevant direction and dampens oscillations. It adds a fraction of the previous update vector to the current one.
Adam (Adaptive Moment Estimation): This is one of the most popular optimizers. It combines the ideas of momentum and another technique called RMSprop. It computes adaptive learning rates for each parameter, making it very effective in practice.