huggingface/nanoVLM resource

Health v2 · maintenance only

Adoption not part of the score

4997 stars · 507 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

nanoVLM is a minimal, readable pure-PyTorch repository (~750 lines) for training and finetuning small Vision-Language Models, inspired by nanoGPT. It combines a SigLIP vision backbone with a SmolLM2 language decoder and includes a simple training loop, primarily serving as an educational codebase.

Use cases

• train a small vision-language model from scratch
• learn how VLMs work by reading a minimal implementation
• finetune a multimodal model on custom image-text data
• understand how vision and language embeddings are combined
• reproduce a 222M-parameter VLM on a single GPU
• use as a starting codebase for multimodal model experiments

When to choose

• you want a readable, hackable educational codebase for VLM training
• you have a GPU and want to train a small multimodal model quickly
• you're learning how vision transformers and language decoders connect
• you prefer pure PyTorch over heavy training frameworks

When to avoid

• you need a production-ready or SOTA multimodal model
• you want polished support scripts, notebooks, and stable APIs (recent breaking changes broke some)
• you need multi-node distributed training out of the box without tweaking
• you want a pretrained VLM for inference rather than training

Facets

learning-resource · maturity active

llm-training machine-learning deep-learning large-language-models computer-vision deep-learning education python vision-language-model pytorch educational nanogpt-style finetuning multimodal gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="huggingface/nanoVLM")