# LiyuanLucasLiu/RAdam

On the Variance of the Adaptive Learning Rate and Beyond

Repository: https://github.com/LiyuanLucasLiu/RAdam
Canonical: https://ross.abutalabs.com/products/radam
Homepage: https://arxiv.org/abs/1908.03265
Language: Python
License: Apache-2.0
License Family: permissive
Topics: optimizer, adam, adam-optimizer, warmup
Last push: 2021-07-31T17:55:32+00:00

## Health v2 (maintenance only)
Score: 32/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 0, release rhythm 35, longevity 100
- inputs: {"age_days": 2590, "days_push": 1859, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 2549, forks 330 (observed 2026-08-28T04:07:00.408623+00:00)

## What it is
RAdam is a Python implementation of Rectified Adam, a variant of the Adam optimizer that analytically reduces the large variance of adaptive learning rates in early training, removing the need for learning rate warmup. It accompanies the ICLR 2020 paper 'On the Variance of the Adaptive Learning Rate and Beyond'.

## Use cases
- train transformer models without learning rate warmup
- stabilize Adam optimizer convergence in early training
- find an optimizer robust to learning rate choice
- replace warmup schedules with a variance-rectified optimizer
- reproduce ICLR 2020 RAdam paper experiments

## When to choose
- you experience divergence or poor convergence with vanilla Adam on transformers or NMT
- you want to avoid tuning warmup steps and eps
- you need a drop-in Adam variant for PyTorch training

## When to avoid
- you need actively maintained optimizer code or new features
- you rely on optimizers already built into modern frameworks
- you need guaranteed convergence without any learning rate tuning

## Facets
- artifact type: library
- maturity: maintenance
- function: machine-learning, llm-training
- domain: machine-learning, deep-learning, large-language-models
- platform: python
- tags: optimizer, adam, radam, learning-rate-warmup, pytorch, research-code

## Member repositories
- LiyuanLucasLiu/RAdam (main) score 32

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:07:00.408623+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T02:23:57.889925+00:00, confidence not recorded.
  - readme: https://github.com/LiyuanLucasLiu/RAdam (fetched 2026-08-28T04:07:00.408623+00:00, sha ce67dcb8f654)
  - homepage: https://arxiv.org/abs/1908.03265 (fetched 2026-08-29T10:06:39.023820+00:00, sha 519b339d4488)
  - site_page: https://info.arxiv.org/about/donate.html (fetched 2026-08-29T10:06:39.032902+00:00, sha cca9c3a11c56)
  - site_page: https://info.arxiv.org/about/ourmembers.html (fetched 2026-08-29T10:06:39.036341+00:00, sha 47cbc55ff1de)
  - site_page: https://info.arxiv.org/about (fetched 2026-08-29T10:06:39.038285+00:00, sha a1f16f915a9a)
  - site_page: https://info.arxiv.org/labs/index.html (fetched 2026-08-29T10:06:39.034757+00:00, sha b14a8d05a0ec)
- Data as of 2026-08-30T08:39:29.467469+00:00.