index464 – Adam vs SGD (Text-Aware) next

Adam vs SGD (Text-Aware) index464

We derive a synthetic 2D quadratic landscape whose curvature ratio is influenced by the vocabulary size and entropy of your text from index461. Larger vocab + higher entropy → higher anisotropy (larger λ₂/λ₁). Observe how Adam adapts while SGD may oscillate along the stiff axis. Adjust β₁, β₂, learning rate, and noise to explore stability.

Editable Text & Parameters

EDIT TEXT (shared)

Words: {{meta.total}}

Unique: {{meta.vocab}}

H(p): {{meta.entropy | number:2}}

PPL: {{meta.perplexity | number:1}}

Parameters

\( \begin{aligned} m_t &= \beta_1 m_{t-1} + (1-\beta_1) g_t,\\ v_t &= \beta_2 v_{t-1} + (1-\beta_2) g_t^{2},\\ \hat m_t &= \frac{m_t}{1-\beta_1^{t}},\quad \hat v_t = \frac{v_t}{1-\beta_2^{t}},\\ heta_{t+1} &= \theta_t - \alpha \frac{\hat m_t}{\sqrt{\hat v_t}+\varepsilon} \end{aligned} \)

VOCAB SIZE |V| (from text) {{meta.vocab}} TEXT ENTROPY H(p) {{meta.entropy | number:2}} CURVATURE RATIO λ₂/λ₁ {{params.curvRatio | number:1}} ITERATIONS {{params.steps}} LEARNING RATE α {{params.lr | number:4}} β₁ {{params.b1 | number:2}} β₂ {{params.b2 | number:3}} EPSILON ε {{params.eps | number:8}} NOISE STD {{params.noise | number:3}}

Adam Final: {{adam.loss[adam.loss.length-1] | number:4}}

SGD Final: {{sgd.loss[sgd.loss.length-1] | number:4}}

Steps: {{adam.x.length}}

Heuristic: initial λ₂/λ₁ set to clamp( 2 + (|V| * H / 150), 2, 60 ). Modify directly with slider to exaggerate anisotropy.

Trajectory

AdamSGDStartEnd

Editable Text & Parameters

Parameters

Trajectory

Loss Curves