TinyLLaVA/TinyLLaVA_Factory

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1004 stars · 103 forks observed · 2026-09-03

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

TinyLLaVA Factory is an open-source modular PyTorch/HuggingFace codebase for training small-scale large multimodal models (LMMs) that combine vision encoders with small language models. It lets researchers customize vision towers, connectors, LLMs, and training recipes (frozen, full, LoRA/QLoRA) with minimal coding effort.

Use cases

• train a small vision-language model on custom image-text data
• fine-tune a multimodal model with LoRA or QLoRA
• build a LLaVA-style model with a small LLM like TinyLlama or Phi
• compare vision encoders like CLIP, SigLIP, and Dino for multimodal training
• reproduce small-scale LMM training results
• run a lightweight multimodal chatbot on limited GPU resources

When to choose

• you want to train or customize a small multimodal (vision+language) model
• you need reproducible LMM training with modular component swaps
• you lack resources for 7B+ models and want competitive 3B-class performance
• you want to experiment with different vision towers, connectors, and tuning recipes

When to avoid

• you need a production-ready inference server or end-user application
• you only want to run pretrained models without training
• you need video or audio multimodality out of the box
• you are not working in Python/PyTorch

Facets

framework · maturity active

machine-learning deep-learning llm-training image-processing large-language-models machine-learning computer-vision deep-learning python multimodal vision-language-models llava small-language-models pytorch huggingface lora-fine-tuning model-training gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="TinyLLaVA/TinyLLaVA_Factory")