Optimizer Comparison: Momentum vs RMSProp vs Adam vs AdamW

This repository demonstrates and compares the behavior of four widely used optimization algorithms in machine learning:

• Momentum (SGD with Momentum)
• RMSProp
• Adam
• AdamW

The goal of this project is to visualize how these optimizers update parameters while minimizing a simple objective function.

This experiment provides an intuitive understanding of optimizer dynamics and highlights why modern deep learning models often prefer Adam or AdamW.

---

Objective Function

We minimize the function:

f(x) = x²

This is a convex function whose global minimum occurs at:

x = 0

The gradient of the function is:

df/dx = 2x

Each optimizer starts at:

x₀ = 8

and iteratively updates the parameter to reach the minimum.

---

Why This Experiment?

Optimization algorithms are at the core of training neural networks.

Different optimizers behave differently in terms of:

• convergence speed
• stability
• oscillation
• sensitivity to learning rate

This project visualizes those differences using:

1. Parameter trajectory plots
2. Loss convergence plots

---

Implemented Optimizers

1. Momentum (SGD with Momentum)

Momentum improves gradient descent by adding a velocity term that accumulates past gradients.

Standard gradient descent:

θ = θ − α ∇J(θ)

Momentum introduces velocity:

vₜ = β vₜ₋₁ + (1 − β) gₜ

Parameter update:

θ = θ − α vₜ

Where:

θ → parameter
α → learning rate
β → momentum coefficient (typically 0.9)
gₜ → gradient at time t

Momentum helps:

• accelerate training
• reduce zig-zag movement in valleys
• smooth gradient updates

However, it can overshoot the minimum, causing oscillations.

---

2. RMSProp

RMSProp adapts the learning rate based on the magnitude of recent gradients.

It maintains an exponentially weighted average of squared gradients.

Second moment estimate:

sₜ = β sₜ₋₁ + (1 − β) gₜ²

Parameter update:

θ = θ − α * gₜ / ( √sₜ + ε )

Where:

sₜ → moving average of squared gradients
ε → small constant for numerical stability

RMSProp helps by:

• scaling updates based on gradient magnitude
• preventing extremely large updates
• stabilizing training in deep networks

---

3. Adam (Adaptive Moment Estimation)

Adam combines the advantages of Momentum and RMSProp.

It tracks both:

• first moment (mean of gradients)
• second moment (variance of gradients)

First moment estimate:

mₜ = β₁ mₜ₋₁ + (1 − β₁) gₜ

Second moment estimate:

vₜ = β₂ vₜ₋₁ + (1 − β₂) gₜ²

Because these estimates start at zero, bias correction is applied:

m̂ₜ = mₜ / (1 − β₁ᵗ)

v̂ₜ = vₜ / (1 − β₂ᵗ)

Final update rule:

θ = θ − α * m̂ₜ / ( √v̂ₜ + ε )

Typical hyperparameters:

β₁ = 0.9
β₂ = 0.999
ε = 1e−8

Adam is widely used because it:

• converges quickly
• adapts learning rates automatically
• works well with sparse gradients

---

4. AdamW

AdamW is a modification of Adam that decouples weight decay from gradient updates.

In Adam, weight decay is implicitly applied through gradients.

AdamW applies weight decay directly:

θ = θ − α * m̂ₜ / ( √v̂ₜ + ε )

Then weight decay:

θ = θ − α * λ θ

Where:

λ → weight decay coefficient

AdamW improves generalization and is used in many modern architectures, including:

• Transformers
• BERT
• GPT models

---

Visualization

Two graphs are generated:

1. Parameter Convergence

Shows how the parameter x moves toward the optimal value (0).

Observations:

• Momentum overshoots and oscillates
• RMSProp moves slowly due to adaptive scaling
• Adam converges smoothly
• AdamW behaves similarly to Adam but includes regularization

---

2. Loss Convergence

Shows how the loss:

L(x) = x²

decreases over iterations.

Faster decrease indicates faster convergence.

---

Example Output

Two plots are generated:

Optimizer Parameter Convergence
Loss Convergence Comparison

These help visualize optimizer dynamics.

---

Project Structure

optimizer-comparison/
│
├── optimizer_comparison.py
├── README.md
└── results/

---

How to Run

Install dependencies:

pip install numpy matplotlib

Run the script:

python optimizer_comparison.py

The script will generate two plots comparing optimizer performance.

---

Learning Outcomes

This project demonstrates:

• the behavior of different gradient-based optimizers
• the importance of adaptive learning rates
• the role of momentum in optimization
• how weight decay affects training

Understanding these algorithms is essential for training deep neural networks effectively.

---

References

Kingma & Ba (2014)
Adam: A Method for Stochastic Optimization

Tieleman & Hinton (2012)
RMSProp Optimization Algorithm

Loshchilov & Hutter (2019)
Decoupled Weight Decay Regularization (AdamW)

Goodfellow, Bengio, Courville
Deep Learning Book