TRI-ML/prismatic-vlms

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1009 stars · 1206 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Prismatic VLMs is a PyTorch-based codebase for training visually-conditioned language models (VLMs) with flexible vision backbones like CLIP, SigLIP, and DINOv2. It supports efficient scaling from 1B to 34B parameters using FSDP and Flash-Attention.

Use cases

• train a vision-language model from scratch
• fine-tune a VLM with custom vision backbones
• train multimodal LLMs on image-text data
• experiment with fused visual representations like DINOv2 + SigLIP
• scale VLM training to 34B parameters with FSDP
• instruct-tune a vision-language assistant

When to choose

• you need a research-grade, configurable codebase for training VLMs
• you want to swap or fuse different vision encoders (CLIP, SigLIP, DINOv2) via timm
• you need efficient large-scale training with FSDP and Flash-Attention
• you want to train on custom multimodal dataset mixtures

When to avoid

• you only need to run inference with an existing VLM
• you want a plug-and-play hosted API for multimodal models
• you lack multi-GPU resources for large-scale training
• you need a full evaluation suite rather than training (use their vlm-evaluation repo instead)

Facets

library · maturity active

llm-training machine-learning deep-learning large-language-models machine-learning computer-vision deep-learning python vision-language-models multimodal pytorch fsdp flash-attention model-training gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="TRI-ML/prismatic-vlms")