LiyuanLucasLiu/RAdam

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

2549 stars · 330 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

RAdam is a Python implementation of Rectified Adam, a variant of the Adam optimizer that analytically reduces the large variance of adaptive learning rates in early training, removing the need for learning rate warmup. It accompanies the ICLR 2020 paper 'On the Variance of the Adaptive Learning Rate and Beyond'.

Use cases

• train transformer models without learning rate warmup
• stabilize Adam optimizer convergence in early training
• find an optimizer robust to learning rate choice
• replace warmup schedules with a variance-rectified optimizer
• reproduce ICLR 2020 RAdam paper experiments

When to choose

• you experience divergence or poor convergence with vanilla Adam on transformers or NMT
• you want to avoid tuning warmup steps and eps
• you need a drop-in Adam variant for PyTorch training

When to avoid

• you need actively maintained optimizer code or new features
• you rely on optimizers already built into modern frameworks
• you need guaranteed convergence without any learning rate tuning

Facets

library · maturity maintenance

machine-learning llm-training machine-learning deep-learning large-language-models python optimizer adam radam learning-rate-warmup pytorch research-code

Member repositories

For agents

markdown · JSON · MCP: product_card(name="LiyuanLucasLiu/RAdam")