Adam Optimizer Explorer (vs SGD) next

Adam Optimizer Explorer 2D Quadratic

Visual comparison of Adam vs SGD on an anisotropic noisy quadratic surface. Adjust learning rate, momentum (β1), variance decay (β2), and noise to see trajectory and loss curves.

Parameters

\( \begin{aligned} m_t &= \beta_1 m_{t-1} + (1-\beta_1) g_t,\\ v_t &= \beta_2 v_{t-1} + (1-\beta_2) g_t^{2},\\ \hat m_t &= \frac{m_t}{1-\beta_1^{t}},\quad \hat v_t = \frac{v_t}{1-\beta_2^{t}},\\ heta_{t+1} &= \theta_t - \alpha \frac{\hat m_t}{\sqrt{\hat v_t}+\varepsilon} \end{aligned} \)

ITERATIONS {{params.steps}} LEARNING RATE α {{params.lr | number:4}} β₁ (momentum) {{params.b1 | number:3}} β₂ (variance) {{params.b2 | number:3}} EPSILON ε {{params.eps | number:7}} NOISE STD {{params.noise | number:3}} CURVATURE RATIO λ₂/λ₁ {{params.curvRatio | number:2}}

Tip: Large curvature ratio makes one direction steep → plain SGD oscillates; Adam adapts step sizes more effectively.

Trajectory

Adam Path SGD Path Start End

Loss Curves

Final Adam Loss: {{adam.loss[adam.loss.length-1] | number:5}}

Final SGD Loss: {{sgd.loss[sgd.loss.length-1] | number:5}}

Adam Steps: {{adam.x.length}}

SGD Steps: {{sgd.x.length}}

Parameters

Trajectory

Loss Curves

Explanation