NExT-GPT/NExT-GPT

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

3638 stars · 360 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

NExT-GPT is an end-to-end any-to-any multimodal large language model that accepts and generates arbitrary combinations of text, image, video, and audio. The repository provides training code, instruction-tuning datasets (MosIT), and model checkpoints built on a pretrained LLM with multimodal encoders and diffusion decoders.

Use cases

• build a chatbot that understands and generates text, images, video, and audio
• run an any-to-any multimodal LLM inference
• fine-tune a multimodal LLM with modality-switching instruction tuning
• research cross-modal understanding and generation in LLMs
• download pretrained multimodal LLM checkpoints
• extend an LLM to new input and output modalities

When to choose

• you need a single model handling any-to-any combinations of text, image, video, and audio
• you want a research-grade multimodal LLM with released weights and training data
• you are studying modality-switching instruction tuning or low-cost adaptor tuning

When to avoid

• you need a production-ready, commercially supported multimodal assistant
• you only need single-modality text generation from a standard LLM
• you lack GPU resources, since the 7B model plus diffusion decoders are heavy to run

Facets

library · maturity active

machine-learning llm-inference llm-training image-processing audio-processing video-processing chatbot agent-framework large-language-models artificial-intelligence deep-learning computer-vision python multimodal any-to-any instruction-tuning diffusion-decoders research-code icml-2024 natural-language-processing audio video ai-agents gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="NExT-GPT/NExT-GPT")