# rasbt/LLMs-from-scratch

Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

Repository: https://github.com/rasbt/LLMs-from-scratch
Canonical: https://ross.abutalabs.com/products/llms-from-scratch
Homepage: https://amzn.to/4fqvn0D
Language: Jupyter Notebook
License: NOASSERTION
License Family: other
Topics: gpt, large-language-models, llm, python, pytorch, ai, artificial-intelligence, language-model, deep-learning, machine-learning, from-scratch, generative-ai, transformers, attention-mechanism, finetuning, instruction-tuning, natural-language-processing, pretraining, tokenizer
Last push: 2026-08-26T22:54:48+00:00

## Health v2 (maintenance only)
Score: 73/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 99, release rhythm 35, longevity 81
- inputs: {"age_days": 1137, "days_push": 7, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases, no_license
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 103857, forks 15891 (observed 2026-08-28T04:12:23.732365+00:00)

## What it is
The official code repository for the book 'Build a Large Language Model (From Scratch)', containing step-by-step PyTorch implementations of a GPT-like LLM. It covers building, pretraining, and finetuning a small functional model for educational purposes, including loading pretrained weights for finetuning.

## Use cases
- learn how large language models work internally by coding one from scratch
- implement a GPT-like transformer in PyTorch step by step
- understand attention mechanisms and tokenization hands-on
- pretrain a small language model for educational purposes
- finetune pretrained LLM weights like GPT-2
- study instruction tuning and classification finetuning of LLMs

## When to choose
- you want a deep, code-first understanding of how LLMs like ChatGPT are built
- you are following the companion book and need its official source code
- you prefer educational, from-scratch implementations over using high-level libraries
- you want to learn pretraining and finetuning concepts with clear diagrams and notebooks

## When to avoid
- you need a production-ready LLM framework or inference server
- you want to train large-scale foundation models efficiently
- you just need to use existing LLMs via APIs without understanding internals
- you need optimized, performant training code rather than pedagogical clarity

## Facets
- artifact type: learning-resource
- maturity: active
- function: llm-training, machine-learning, deep-learning, nlp
- domain: large-language-models, deep-learning, machine-learning, tutorials
- platform: python
- tags: gpt, pytorch, from-scratch, transformers, pretraining, finetuning, instruction-tuning, educational, jupyter-notebooks, book-companion, natural-language-processing

## Member repositories
- rasbt/LLMs-from-scratch (main) score 73

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:12:23.732365+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T16:12:17.310688+00:00, confidence not recorded.
  - readme: https://github.com/rasbt/LLMs-from-scratch (fetched 2026-08-28T04:12:23.732365+00:00, sha 74b6c7459e88)
  - homepage: https://amzn.to/4fqvn0D (fetched 2026-08-28T17:32:51.271327+00:00, sha 44491a836726)
- Data as of 2026-08-30T08:39:29.467469+00:00.