PantoMatrix/PantoMatrix

Health v2 · maintenance only

Flags: no_releases no_license

Adoption not part of the score

1293 stars · 195 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

PantoMatrix is an open-source research project that generates 3D face and body animation from speech audio, including the EMAGE model for holistic co-speech gesture and facial motion. It takes speech audio as input and outputs motion parameters convertible to formats like ARKit blendshape weights and Vicon BVH skeleton files.

Use cases

• generate 3D avatar body gestures from speech audio
• create talking face animation from an audio clip
• convert speech to ARKit blendshape weights for iPhone avatars
• export co-speech motion as BVH skeleton files for Vicon or game engines
• research co-speech gesture and facial animation generation
• animate a virtual character to match spoken audio

When to choose

• you need speech-driven 3D face and body animation in one pipeline
• you want research-grade co-speech gesture models like EMAGE with pretrained weights
• you need motion output in standard formats like BVH or ARKit blendshapes

When to avoid

• you need a production-ready product with a license guarantee (no license is provided)
• you only need simple lip-sync for 2D video rather than 3D motion parameters
• you lack GPU resources or ML experience for research code

Facets

library · maturity active

machine-learning audio-processing animation sdk machine-learning computer-vision graphics artificial-intelligence python cross-platform co-speech-gesture talking-face-generation 3d-animation motion-generation research deep-learning blendshapes bvh gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="PantoMatrix/PantoMatrix")