facebookresearch/perception_models

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

2353 stars · 160 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Meta's Perception Models repository hosting state-of-the-art image, video, and audio encoders (Perception Encoder, PE) and a multimodal language model (Perception Language Model, PLM). It provides pretrained checkpoints, evaluation benchmarks, and integrations with Hugging Face transformers and timm.

Use cases

• encode images and videos into embeddings for retrieval
• run zero-shot image and video classification with CLIP-style models
• build video question answering with a multimodal language model
• evaluate multimodal models on PLM-VideoBench
• extract audio-visual embeddings for cross-modal search
• fine-tune vision-language models on custom video data

When to choose

• you need state-of-the-art visual or audiovisual embeddings for retrieval or classification
• you want an open multimodal LLM for video understanding tasks
• you want models already integrated into Hugging Face transformers or timm

When to avoid

• you need lightweight CPU-only inference without GPU resources
• you need a production application rather than research models and checkpoints
• you need text-only language modeling without vision or audio

Facets

library · maturity active

machine-learning deep-learning computer-vision image-processing video-processing audio-processing llm-inference transformers computer-vision deep-learning machine-learning artificial-intelligence python cross-platform clip perception-encoder perception-language-model multimodal embeddings video-understanding audiovisual pytorch model-checkpoints facebook-research video audio natural-language-processing gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="facebookresearch/perception_models")