invictus717/MetaTransformer

Health v2 · maintenance only

Adoption not part of the score

1647 stars · 116 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Meta-Transformer is a research framework for unified multimodal learning that maps inputs from 12 modalities (text, images, point clouds, audio, video, time series, tabular data, etc.) into a shared token space processed by a frozen encoder. It accompanies an arXiv paper and provides pretrained models and code for multimodal perception without paired multimodal training data.

Use cases

• process images, text, audio, and point clouds with one unified model
• multimodal learning without paired training data
• run perception across 12 modalities with a frozen transformer encoder
• benchmark a foundation model on multimodal tasks
• extract features from tabular, time-series, and graph data with a transformer
• research unified multimodal tokenization

When to choose

• you need a single model to handle many data modalities
• you lack paired multimodal training data
• you want to reproduce or extend the Meta-Transformer paper
• you need modality-agnostic feature extraction

When to avoid

• you need a production-ready, actively maintained multimodal API
• you only work with a single modality like text or images
• you need fine-grained state-of-the-art results on one specific task
• you require frequent updates or commercial support

Facets

library · maturity maintenance

machine-learning deep-learning transformers image-processing audio-processing video-processing nlp artificial-intelligence machine-learning computer-vision deep-learning python cross-platform multimodal foundation-model transformer research-code unified-tokenizer frozen-encoder natural-language-processing gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="invictus717/MetaTransformer")