rasbt/LLMs-from-scratch resource

Health v2 · maintenance only

Flags: no_releases no_license

Adoption not part of the score

103857 stars · 15891 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

The official code repository for the book 'Build a Large Language Model (From Scratch)', containing step-by-step PyTorch implementations of a GPT-like LLM. It covers building, pretraining, and finetuning a small functional model for educational purposes, including loading pretrained weights for finetuning.

Use cases

• learn how large language models work internally by coding one from scratch
• implement a GPT-like transformer in PyTorch step by step
• understand attention mechanisms and tokenization hands-on
• pretrain a small language model for educational purposes
• finetune pretrained LLM weights like GPT-2
• study instruction tuning and classification finetuning of LLMs

When to choose

• you want a deep, code-first understanding of how LLMs like ChatGPT are built
• you are following the companion book and need its official source code
• you prefer educational, from-scratch implementations over using high-level libraries
• you want to learn pretraining and finetuning concepts with clear diagrams and notebooks

When to avoid

• you need a production-ready LLM framework or inference server
• you want to train large-scale foundation models efficiently
• you just need to use existing LLMs via APIs without understanding internals
• you need optimized, performant training code rather than pedagogical clarity

Facets

learning-resource · maturity active

llm-training machine-learning deep-learning nlp large-language-models deep-learning machine-learning tutorials python gpt pytorch from-scratch transformers pretraining finetuning instruction-tuning educational jupyter-notebooks book-companion natural-language-processing

Member repositories

For agents

markdown · JSON · MCP: product_card(name="rasbt/LLMs-from-scratch")