16.1 Backpropagation Algorithm

The Engine of Learning

Backpropagation, short for "backward propagation of errors," is the algorithm that allows us to efficiently train deep neural networks. It's the workhorse that powers modern deep learning, from image recognition to large language models.

The core idea is to calculate the gradient of the loss function with respect to each weight and bias in the network. This tells us how a small change in each parameter affects the overall error. We can then use this information to update the parameters in a way that reduces the error.

The Two-Pass Process

Backpropagation consists of two main passes through the network:

The Forward Pass: An input is fed into the network, and its value is propagated forward from layer to layer. At each neuron, we calculate the weighted sum of its inputs and apply the activation function. This continues until we get the final output, which is then used to calculate the loss.
The Backward Pass: This is where the magic happens. We start at the output layer and propagate the error backward. At each layer, we use the chain rule from calculus to compute the gradient of the loss with respect to the layer's weights and biases. This gradient is then passed to the previous layer, and the process continues until we reach the input layer.

A Glimpse at the Math

Let's consider a single neuron. Its output is $a = f(z)$, where $z = \sum w_i x_i + b$ is the weighted input and $f$ is the activation function. The loss is $L$.

To update a weight $w_i$, we need the gradient $\frac{\partial L}{\partial w_i}$. Using the chain rule, we can break this down:

$\frac{\partial L}{\partial w_i} = \frac{\partial L}{\partial a} \frac{\partial a}{\partial z} \frac{\partial z}{\partial w_i}$

$\frac{\partial L}{\partial a}$ is the gradient from the next layer (this is what's being "backpropagated").
$\frac{\partial a}{\partial z} = f'(z)$ is the derivative of the activation function.
$\frac{\partial z}{\partial w_i} = x_i$ is the input corresponding to the weight.

By calculating these components at each neuron, we can efficiently compute the gradients for the entire network.

Visualizing Backpropagation

This visualization shows a simple neural network. You can adjust the input and target values. The forward pass calculates the output and error. The backward pass then visualizes the flow of gradients (represented by the thickness and color of the lines) back through the network.

Input (x): Target (y):

Predicted Output: {{output.toFixed(4)}}

Loss (MSE): {{loss.toFixed(4)}}

16.1 The Backpropagation Algorithm

Backpropagation, short for "backward propagation of errors," is the cornerstone of training artificial neural networks. It's an efficient algorithm for computing the gradients of the loss function with respect to the network's weights and biases. These gradients are then used by an optimization algorithm, like gradient descent, to update the parameters and minimize the loss.

The Intuition: Chain Rule in Action

Imagine a simple neural network with one input, one hidden layer with one neuron, and one output. The output is a function of the input, weights, and biases.

Forward Pass:
1. Hidden neuron's pre-activation: $ z_h = w_1 \cdot x + b_1 $
2. Hidden neuron's activation: $ a_h = \sigma(z_h) $ (where $ \sigma $ is an activation function like sigmoid)
3. Output neuron's pre-activation: $ z_o = w_2 \cdot a_h + b_2 $
4. Final output: $ \hat{y} = \sigma(z_o) $

The loss $ L $ is a function of the predicted output $ \hat{y} $ and the true target $ y $, for example, Mean Squared Error: $ L = \frac{1}{2} (\hat{y} - y)^2 $.

To update a weight like $ w_2 $, we need the gradient $ \frac{\partial L}{\partial w_2} $. Using the chain rule, we can break this down: $$ \frac{\partial L}{\partial w_2} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z_o} \cdot \frac{\partial z_o}{\partial w_2} $$

This process is repeated "backwards" from the output layer to the input layer for all weights and biases, efficiently reusing computed gradients.

Interactive Backpropagation Visualization

This visualization demonstrates the forward and backward pass on a simple 2-2-1 neural network. Adjust the input and target values, then run the forward pass to see the calculated output and loss. The backward pass is then visualized by showing the magnitude of the gradient for each weight—thicker lines indicate a larger gradient, meaning the weight has a greater impact on the current error. Red indicates a negative gradient (weight should increase) and blue a positive one (weight should decrease).

Input (x): Target (y):

Predicted Output: {{output.toFixed(4)}}

Loss (MSE): {{loss.toFixed(4)}}

The Steps of Backpropagation

Forward Pass: Feed an input through the network to compute the output $ \hat{y} $ and the loss $ L(y, \hat{y}) $. Store all intermediate values (activations and pre-activations).
Backward Pass (Output Layer): Compute the gradient of the loss with respect to the output layer's pre-activation, $ \frac{\partial L}{\partial z_o} $. This is the starting point.
Backward Pass (Hidden Layers): Propagate the error backward. For each layer, compute the gradient of the loss with respect to its parameters (weights and biases) and with respect to its inputs. The gradient with respect to the inputs of the current layer becomes the gradient with respect to the outputs of the previous layer.
Parameter Update: Use the computed gradients to update all weights and biases in the network, typically using an optimization algorithm. $ w_{new} = w_{old} - \eta \frac{\partial L}{\partial w_{old}} $, where $ \eta $ is the learning rate.

Why It's Important

Without backpropagation, training deep neural networks would be computationally intractable. It provides a systematic and efficient way to distribute the error signal throughout the network, allowing even very deep and complex models to learn from data. It is the engine that powers modern deep learning.

Chain Rule Step-by-Step (Per Weight)

Inspect the decomposition of $\partial L / \partial w_i$ via the chain rule. Each component updates live after running the forward/backward pass above. This helps connect symbolic derivatives to concrete numerical values.

Weight	Expression	Chain Expansion	Numerical Factors	Value
`{{row.weight}}`	`{{row.symbolic}}`	`{{row.expansion}}`	`{{row.numericFactors}}`	{{row.value \| number:6}}