Liuhong99/Sophia

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1001 stars · 56 forks observed · 2026-10-21

What it is AI-extracted, prompt v1, taxonomy v1, 2026-09-07, confidence 0.95

Official PyTorch implementation of Sophia-G, a scalable stochastic second-order optimizer for language model pre-training, from the paper by Liu et al. (2023). It includes the SophiaG optimizer class and GPT-2 training scripts based on nanoGPT and levanter.

Use cases

• speed up language model pretraining with a second-order optimizer
• replace AdamW with SophiaG in PyTorch training loops
• train GPT-2 models with the Sophia optimizer
• research second-order optimization methods for LLMs
• tune hyperparameters for SophiaG like rho and learning rate

When to choose

• you are pretraining or fine-tuning a transformer language model in PyTorch and want faster convergence than AdamW
• you are doing research on optimizer design for deep learning
• you want to reproduce the results of the Sophia paper

When to avoid

• you need a production-hardened optimizer with broad ecosystem support
• your model is not a language model or your framework is not PyTorch
• you cannot afford the periodic Hessian/gradient estimate computation overhead

Facets

library · maturity active

machine-learning llm-training deep-learning deep-learning large-language-models machine-learning python optimizer second-order-optimization pytorch language-model-pretraining research-code gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="Liuhong99/Sophia")