Om-Alve/smolGPT

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1475 stars · 126 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A minimal pure-PyTorch implementation for training small GPT-style LLMs from scratch, featuring flash attention, RMSNorm, SwiGLU, RoPE, and modern sampling techniques. It includes a full training pipeline with dataset preprocessing, SentencePiece tokenizer training, and text generation sampling, aimed at education.

Use cases

• train a small GPT model from scratch
• learn how transformers and LLMs work internally
• generate text with a custom-trained language model
• train an LLM on the TinyStories dataset
• experiment with flash attention and RoPE in PyTorch
• build a custom tokenizer with SentencePiece

When to choose

• you want a readable, minimal codebase to learn LLM training internals
• you need to train a small GPT on a custom dataset with a modern GPU
• you want an educational alternative to heavyweight frameworks like nanoGPT

When to avoid

• you need production-grade LLM training at scale with distributed support
• you want a ready-made large pretrained model for real applications
• you need multi-GPU or multi-node training out of the box

Facets

library · maturity active

llm-training llm-inference machine-learning deep-learning large-language-models deep-learning machine-learning education tutorials python cross-platform pytorch gpt transformer flash-attention tinystories educational from-scratch sentencepiece rope text-generation tokenization gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="Om-Alve/smolGPT")