KellerJordan/modded-nanogpt resource

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

5707 stars · 869 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

A collaborative speedrun project that trains a GPT-2 (124M) scale language model to 3.28 validation loss on FineWeb in under 75 seconds on 8xH100 GPUs. It serves as both a record-setting training implementation and a learning resource showcasing modern LLM training techniques like the Muon optimizer, FP8, and architectural modernizations.

Use cases

• learn how to train a GPT-2 scale language model efficiently
• study modern LLM training optimizations like Muon and FP8
• benchmark GPU training speed for small language models
• find the fastest algorithm to train a 124M parameter transformer
• reproduce a fast GPT-2 replication on H100 GPUs
• learn modern transformer architecture tricks like QK-Norm and rotary embeddings

When to choose

• you want to learn state-of-the-art techniques for training small language models quickly
• you have access to 8xH100 GPUs and want to reproduce or compete in the speedrun
• you want a reference implementation of the Muon optimizer and modern GPT training tricks

When to avoid

• you need a production training framework with configuration, checkpointing, and multi-dataset support
• you lack high-end NVIDIA GPUs, since the code is heavily tuned for 8xH100 setups
• you want to train large models or fine-tune existing ones rather than a fixed 124M GPT-2 scale target

Facets

learning-resource · maturity active

llm-training benchmarking machine-learning deep-learning large-language-models gpu-computing tutorials python nanogpt speedrun muon-optimizer h100 training-efficiency pytorch fineweb gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="KellerJordan/modded-nanogpt")