JonasGeiping/cramming

Health v2 · maintenance only

Adoption not part of the score

1366 stars · 105 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A research framework for pretraining BERT-type language models from scratch on a single consumer GPU in one day, replicating the paper 'Cramming: Training a Language Model on a Single GPU in One Day'. It provides a modified pretraining pipeline with performance close to BERT under tight compute budgets.

Use cases

• pretrain a BERT-like language model on a single GPU
• train a language model from scratch with limited compute
• replicate the cramming paper experiments
• study how scaling laws apply to low-compute LM training
• fine-tune a small masked language model for GLUE tasks
• experiment with efficient transformer pretraining recipes

When to choose

• you want to pretrain a small BERT-type model without a large compute budget
• you need a research baseline for low-compute language model training
• you want to study which training modifications help in constrained settings

When to avoid

• you need to train large generative LLMs at scale
• you want a production-ready pretrained model serving pipeline
• you need multi-node distributed training infrastructure

Facets

library · maturity maintenance

machine-learning llm-training deep-learning machine-learning deep-learning python bert pretraining single-gpu research-code pytorch masked-language-modeling scaling-laws natural-language-processing gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="JonasGeiping/cramming")