bytedance/LatentSync
Taming Stable Diffusion for Lip Sync! observed · 2026-08-28
Health v2 · maintenance only
33/100
- Activity 27
- Release rhythm 35
- Longevity 45
Flags: no_releases
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-02. Adoption (stars, forks) is never an input.
- gap_med: n/a
- age_days: 630
- days_rel: n/a
- days_push: 439
- n_releases_24m: 0
Adoption not part of the score
6026 stars · 977 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded
LatentSync is an end-to-end lip-sync framework from ByteDance based on audio-conditioned latent diffusion models, using Stable Diffusion to directly model audio-visual correlations without intermediate motion representations. It uses Whisper audio embeddings, cross-attention integration into U-Net, and SyncNet supervision (StableSyncNet) for accurate lip synchronization in videos.
Use cases
- sync lip movements in a video to new audio
- dub a video into another language with matching lips
- create talking avatar videos from a portrait
- generate lip-synced talking head videos
- retime a person's mouth to match a voiceover
- animate a face in a video to match speech audio
When to choose
- you need high-quality, research-grade lip sync on videos
- you want an open-source alternative to commercial lip-sync tools
- you have a GPU and want local, self-hosted inference
- you need multilingual lip sync including Chinese
When to avoid
- you lack a GPU or VRAM headroom for diffusion inference
- you need real-time lip sync on live streams
- you only need simple face swaps rather than audio-driven sync
- you need a production API with managed scaling
Facets
library · maturity active
video-processing machine-learning deep-learning speech-recognition image-processing deep-learning computer-vision artificial-intelligence python lip-sync latent-diffusion stable-diffusion talking-head audio-driven virtual-avatars video-generation syncnet whisper video audio linux gpu
6 sources
- readme: https://github.com/bytedance/LatentSync · fetched 2026-08-28 · 5e064aacc127
- homepage: https://arxiv.org/abs/2412.09262 · fetched 2026-08-29 · ba2c3f1c310f
- site_page: https://info.arxiv.org/about/donate.html · fetched 2026-08-29 · cca9c3a11c56
- site_page: https://info.arxiv.org/about/ourmembers.html · fetched 2026-08-29 · 47cbc55ff1de
- site_page: https://info.arxiv.org/about · fetched 2026-08-29 · a1f16f915a9a
- site_page: https://info.arxiv.org/labs/index.html · fetched 2026-08-29 · b14a8d05a0ec
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| bytedance/LatentSync | main | 33 |
For agents
markdown · JSON · MCP: product_card(name="bytedance/LatentSync")
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem