bytedance/LatentSync
Taming Stable Diffusion for Lip Sync! observed · 2026-10-09
Health v2 · maintenance only
30/100
- Activity 19
- Release rhythm 35
- Longevity 48
Flags: no_releases
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-10-21. Adoption (stars, forks) is never an input.
- gap_med: n/a
- age_days: 678
- days_rel: n/a
- days_push: 487
- n_releases_24m: 0
Adoption not part of the score
6120 stars · 983 forks observed · 2026-10-09
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded
LatentSync is an end-to-end lip-sync framework from ByteDance based on audio-conditioned latent diffusion models, using Stable Diffusion to directly model audio-visual correlations without intermediate motion representations. It uses Whisper audio embeddings, cross-attention integration into U-Net, and SyncNet supervision (StableSyncNet) for accurate lip synchronization in videos.
Use cases
- sync lip movements in a video to new audio
- dub a video into another language with matching lips
- create talking avatar videos from a portrait
- generate lip-synced talking head videos
- retime a person's mouth to match a voiceover
- animate a face in a video to match speech audio
When to choose
- you need high-quality, research-grade lip sync on videos
- you want an open-source alternative to commercial lip-sync tools
- you have a GPU and want local, self-hosted inference
- you need multilingual lip sync including Chinese
When to avoid
- you lack a GPU or VRAM headroom for diffusion inference
- you need real-time lip sync on live streams
- you only need simple face swaps rather than audio-driven sync
- you need a production API with managed scaling
Facets
library · maturity active
video-processing machine-learning deep-learning speech-recognition image-processing deep-learning computer-vision artificial-intelligence python lip-sync latent-diffusion stable-diffusion talking-head audio-driven virtual-avatars video-generation syncnet whisper video audio linux gpu
6 sources
- readme: https://github.com/bytedance/LatentSync · fetched 2026-10-09 · 5e064aacc127
- homepage: https://arxiv.org/abs/2412.09262 · fetched 2026-08-29 · ba2c3f1c310f
- site_page: https://info.arxiv.org/about/donate.html · fetched 2026-08-29 · cca9c3a11c56
- site_page: https://info.arxiv.org/about/ourmembers.html · fetched 2026-08-29 · 47cbc55ff1de
- site_page: https://info.arxiv.org/about · fetched 2026-08-29 · a1f16f915a9a
- site_page: https://info.arxiv.org/labs/index.html · fetched 2026-08-29 · b14a8d05a0ec
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| bytedance/LatentSync | main | 30 |
For agents
markdown · JSON · MCP: product_card(name="bytedance/LatentSync")
Data as of 2026-10-19T08:05:51.212177+00:00 · Report a problem