vivekkalyanarangan30/llm_from_scratch resource

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1102 stars · 287 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A hands-on Python/PyTorch curriculum that builds large language models from scratch, covering transformer architecture, training, modern improvements like RoPE and MoE, and alignment via SFT, reward modeling, and PPO-based RLHF. It is structured as a step-by-step educational course with runnable code rather than a production library.

Use cases

• learn how transformers and self-attention work from first principles
• train a tiny LLM from scratch in PyTorch
• understand RLHF and PPO implementation
• implement mixture-of-experts layers
• learn modern LLM techniques like RoPE, RMSNorm, and KV cache
• study the full LLM pipeline from pretraining to alignment

When to choose

• you want to learn LLM internals by coding them yourself
• you need a structured curriculum covering pretraining through RLHF
• you prefer PyTorch-based educational code over high-level frameworks

When to avoid

• you need a production-ready training framework
• you want pretrained models or inference serving
• you need distributed multi-node training at scale

Facets

learning-resource · maturity active

llm-training deep-learning machine-learning large-language-models deep-learning machine-learning tutorials python cross-platform pytorch transformer rlhf mixture-of-experts hands-on-curriculum nano-gpt-style education gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="vivekkalyanarangan30/llm_from_scratch")