# PantoMatrix/PantoMatrix

PantoMatrix: Generating Face and Body Animation from Speech

Repository: https://github.com/PantoMatrix/PantoMatrix
Canonical: https://ross.abutalabs.com/products/pantomatrix
Language: Python
License Family: other
Topics: co-speech-gesture, talking-face-generation, co-speech-facial
Last push: 2025-01-16T06:55:20+00:00

## Health v2 (maintenance only)
Score: 33/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 1, release rhythm 35, longevity 100
- inputs: {"age_days": 1514, "days_push": 594, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases, no_license
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1293, forks 195 (observed 2026-08-28T04:04:16.219272+00:00)

## What it is
PantoMatrix is an open-source research project that generates 3D face and body animation from speech audio, including the EMAGE model for holistic co-speech gesture and facial motion. It takes speech audio as input and outputs motion parameters convertible to formats like ARKit blendshape weights and Vicon BVH skeleton files.

## Use cases
- generate 3D avatar body gestures from speech audio
- create talking face animation from an audio clip
- convert speech to ARKit blendshape weights for iPhone avatars
- export co-speech motion as BVH skeleton files for Vicon or game engines
- research co-speech gesture and facial animation generation
- animate a virtual character to match spoken audio

## When to choose
- you need speech-driven 3D face and body animation in one pipeline
- you want research-grade co-speech gesture models like EMAGE with pretrained weights
- you need motion output in standard formats like BVH or ARKit blendshapes

## When to avoid
- you need a production-ready product with a license guarantee (no license is provided)
- you only need simple lip-sync for 2D video rather than 3D motion parameters
- you lack GPU resources or ML experience for research code

## Facets
- artifact type: library
- maturity: active
- function: machine-learning, audio-processing, animation, sdk
- domain: machine-learning, computer-vision, graphics, artificial-intelligence
- platform: python, cross-platform
- tags: co-speech-gesture, talking-face-generation, 3d-animation, motion-generation, research, deep-learning, blendshapes, bvh, gpu

## Member repositories
- PantoMatrix/PantoMatrix (main) score 33

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:04:16.219272+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T04:54:07.869253+00:00, confidence not recorded.
  - readme: https://github.com/PantoMatrix/PantoMatrix (fetched 2026-08-28T04:04:16.219272+00:00, sha ba959c1c12b2)
- Data as of 2026-08-30T08:39:29.467469+00:00.