FareedKhan-dev/train-llm-from-scratch resource

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

9432 stars · 1306 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

An educational repository and tutorial that implements a transformer-based LLM entirely from scratch in plain PyTorch, covering the full pipeline from raw text data to an aligned, reasoning-style model. It walks through pretraining, SFT, reward modeling, and preference optimization (DPO, PPO, GRPO) without relying on libraries like transformers, trl, or peft.

Use cases

• learn how transformers work by implementing attention from scratch
• train a small LLM on a single GPU
• understand RLHF alignment algorithms like DPO and PPO
• build a next-token prediction model from raw text
• study the full LLM training pipeline end to end
• implement reward models and GRPO in plain PyTorch

When to choose

• you want to deeply understand LLM internals rather than use high-level libraries
• you are learning how pretraining, SFT, and RLHF fit together
• you need a from-scratch PyTorch reference implementation of transformer training
• you want to train a small (million to billion parameter) model on limited hardware

When to avoid

• you need a production-ready LLM training framework with optimizations and ecosystem support
• you want to fine-tune existing pretrained models quickly with Hugging Face tooling
• you need multi-node distributed training at scale
• you are looking for inference or deployment tooling rather than training education

Facets

learning-resource · maturity active

llm-training deep-learning machine-learning nlp developer-tools large-language-models deep-learning machine-learning tutorials artificial-intelligence python cross-platform transformer-from-scratch pytorch pretraining sft rlhf dpo ppo grpo reward-model educational single-gpu natural-language-processing gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="FareedKhan-dev/train-llm-from-scratch")