16.2 Optimisation of Neural Networks I: Gradient Descent Variants

Once we have the gradients computed via backpropagation, we need a way to use them to update the network's weights. The method used to update the parameters is called an optimization algorithm. The foundational algorithm is Gradient Descent, but several variants exist to improve its speed and performance, especially on large datasets.

The Core Idea: Walking Downhill

Imagine the loss function as a hilly landscape. Our goal is to find the lowest point (the minimum loss). Gradient Descent does this by taking steps in the direction of the steepest descent, which is the negative of the gradient. The basic update rule is: $$ W_{new} = W_{old} - \eta \nabla_W L $$ where $ W $ represents the weights, $ \eta $ is the learning rate (step size), and $ \nabla_W L $ is the gradient of the loss $ L $ with respect to the weights.

The key difference between the variants lies in how much data we use to compute the gradient at each step.

The Variants

Batch Gradient Descent: This is the "vanilla" version. To take a single step, we calculate the gradients for the entire training dataset.
- Pros: Guarantees convergence to the global minimum for convex loss surfaces and a stable convergence path.
- Cons: Extremely slow and memory-intensive for large datasets, as the entire dataset must be processed for every single update. It's computationally intractable for modern deep learning.
Stochastic Gradient Descent (SGD): In this approach, we perform an update for each individual training example. We shuffle the dataset and update the weights after every sample.
- Pros: Much faster updates, allowing for rapid iteration. The noisy updates can help escape shallow local minima.
- Cons: The convergence path is very noisy and can oscillate heavily. It never truly "settles" at the minimum but bounces around it.
Mini-batch Gradient Descent: This is the compromise and the most common method used in practice. We update the weights after processing a small batch (e.g., 32, 64, 128 samples) of training data.
- Pros: Offers the best of both worlds. It provides a more stable convergence than SGD while being far more computationally efficient than Batch GD. It also takes advantage of modern hardware (GPUs) that are optimized for parallel matrix operations.
- Cons: Introduces a new hyperparameter (the batch size) that needs to be tuned.

Interactive Comparison of Gradient Descent Variants

The visualization below shows a simplified 2D loss surface (like a contour map of a valley). The goal is to reach the center (the minimum loss). You can select an optimization algorithm and see how it navigates the landscape.

Optimizer: Learning Rate:

Iteration: {{iteration}}

Side-by-Side Optimizer Convergence

This tool runs multiple optimizers simultaneously on the same convex quadratic surface $f(x,y)=x^2+y^2$ so you can directly compare their trajectory and loss vs. iteration curves. Toggle optimizers, adjust the learning rate (shared where applicable), and restart to observe differences in speed, smoothness, and stability.

{{opt.name}} LR: Iterations:

Current Iteration: {{multiIter}} | Active Optimizers: {{activeOptimizerNames()}}

Learning Rate Scheduling

Different learning rate schedules can dramatically change convergence. Experiment with schedules to see how the path and loss curve adjust. Schedules implemented: Step Decay, Exponential Decay, Cosine Annealing, and Cyclical (Triangular).

Schedule: Base LR: Step Every: Gamma: Decay k: Tmax: Cycle: Min LR:

Iteration: {{lrIter}} / {{lrMaxIter}} | Current LR: {{currentScheduledLr | number:5}}

Learning Rate Sensitivity

This plot sweeps a range of learning rates for plain Gradient Descent on $f(x,y)=x^2+y^2$ (starting from the same initial point) and records the number of iterations needed to reach a small loss threshold. Extremely large learning rates may diverge (marked in red). Use it to pick a reasonable learning rate region.

Min LR: Max LR: Samples: Max Iter:

Conclusion

Choosing the right optimization strategy is crucial for training neural networks effectively. While Batch Gradient Descent provides a theoretical baseline, Mini-batch Gradient Descent is the de facto standard for its balance of efficiency and stability. The noisy nature of SGD, while sometimes beneficial, is often less reliable for achieving precise convergence.