16.3 Optimisation of Neural Networks II: Momentum, RMSprop, Adam

While Mini-batch Gradient Descent is a solid default, the optimization landscape of deep neural networks can be complex, containing ravines, saddle points, and local minima. Advanced optimization algorithms have been developed to navigate these challenges more effectively, leading to faster convergence and better final models.

The Need for Speed and Stability

Standard gradient descent can oscillate in directions of high curvature and move slowly in directions of low curvature. The following algorithms introduce mechanisms to dampen oscillations and accelerate progress.

The Algorithms

  • Momentum: This method helps accelerate SGD in the relevant direction and dampens oscillations. It does this by adding a fraction \( \gamma \) of the previous update vector to the current one. Imagine a ball rolling down a hill; it accumulates momentum and doesn't get stuck in small bumps.

    Update Rule:

    $$ v_t = \gamma v_{t-1} + \eta \nabla_W L $$ $$ W = W - v_t $$

    Here, \( v_t \) is the velocity at time \( t \), and \( \gamma \) (e.g., 0.9) is the momentum term.

  • RMSprop (Root Mean Square Propagation): This algorithm adapts the learning rate for each parameter separately. It divides the learning rate by an exponentially decaying average of squared gradients. This helps to decrease the learning rate for parameters with large gradients (to prevent oscillations) and increase it for parameters with small gradients.

    Update Rule:

    $$ E[g^2]_t = 0.9 E[g^2]_{t-1} + 0.1 g_t^2 $$ $$ W = W - \frac{\eta}{\sqrt{E[g^2]_t + \epsilon}} g_t $$

    Here, \( g_t \) is the gradient at time \( t \), and \( \epsilon \) is a small smoothing term to prevent division by zero.

  • Adam (Adaptive Moment Estimation): Adam is arguably the most popular and effective optimization algorithm today. It combines the ideas of both Momentum and RMSprop. It keeps an exponentially decaying average of past gradients (like momentum) and past squared gradients (like RMSprop).

    Update Rule (simplified):

    $$ m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t $$ $$ v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2 $$ $$ W = W - \frac{\eta}{\sqrt{v_t} + \epsilon} m_t $$

    Adam uses bias-corrected estimates of \( m_t \) and \( v_t \) and default values for \( \beta_1 \) (0.9) and \( \beta_2 \) (0.999) that work well in most cases.

Interactive Comparison of Advanced Optimizers

This visualization compares how different optimizers navigate a challenging loss surface with a narrow ravine. Observe how Momentum "overshoots," how RMSprop adapts, and how Adam often finds the most direct path.

Iteration: {{iteration}}

Conclusion

Adam is often the best choice as a default optimizer for deep learning models because it combines the best properties of other algorithms. However, understanding Momentum and RMSprop provides crucial insight into how adaptive learning rates work and why they are so effective.