lucidrains/transfusion-pytorch

Health v2 · maintenance only

Adoption not part of the score

1408 stars · 76 forks observed · 2026-10-21

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A PyTorch implementation of Transfusion, MetaAI's approach to predicting the next token and diffusing images with a single multi-modal model. It substitutes diffusion with flow matching and supports extending to any number of modalities.

Use cases

• train a single model that generates both text and images
• implement next-token prediction combined with image diffusion
• experiment with flow matching for multimodal generation
• build a multimodal model handling text, images, and audio
• research multi-modality transformer architectures
• generate interleaved text and image sequences

When to avoid

• you need a production-ready, fully trained multimodal model out of the box
• you want a simple inference-only text-to-image tool without training
• you are not working in PyTorch
• you need a maintained product with extensive documentation and support

Facets

library · maturity active

machine-learning deep-learning llm-training transformers artificial-intelligence deep-learning machine-learning large-language-models python multimodal flow-matching diffusion text-to-image research-implementation pytorch

Member repositories

For agents

markdown · JSON · MCP: product_card(name="lucidrains/transfusion-pytorch")