CHAMELAION LipSync API From €0.33/mo · Free plan 9.1 Visit LatentSync
github.com
What it is
LatentSync is a free, open-source lip-sync method for Linux and self-hosted deployment. It uses audio-conditioned latent diffusion models: Whisper converts mel spectrograms into audio embeddings, which enter a U-Net through cross-attention layers. The project offers a Gradio application and command-line inference, plus checkpoints, training code and a data-processing pipeline. LatentSync 1.6 was trained on 512 × 512 videos to address blurry teeth and lips. Minimum inference memory is listed as 8 GB of VRAM for version 1.5 and 18 GB for version 1.6. More inference steps can improve visual quality but slow generation; increasing guidance scale can improve lip-sync accuracy while introducing distortion or jitter. Training configurations for the U-Net require 20–55 GB of VRAM, depending on configuration. The documented pipeline uses PySceneDetect for scene detection and InsightFace landmarks for face transformation. The repository lists Apache-2.0, while the model card lists openrail++.
Spec sheet
- What it does
- LatentSync is an end-to-end lip-sync system that generates mouth movements from audio with latent diffusion and no intermediate motion representation.Read 21 Oct 2026 · raw.githubusercontent.com
- Purpose
- LatentSync is an end-to-end lip-sync method based on audio-conditioned latent diffusion models.Read 8 Oct 2026 · github.com
- Intended users
- The README describes an efficient stage-two training configuration as suitable for consumer-grade GPUs such as the RTX 3090.Read 8 Oct 2026 · github.com
- License
- The source repository is licensed under Apache License 2.0.Read 21 Oct 2026 · raw.githubusercontent.com
- Audio conditioning
- Whisper converts mel-spectrograms into audio embeddings that enter the U-Net through cross-attention.Read 21 Oct 2026 · raw.githubusercontent.com
- Community complaint: GPU compatibility
- Open issues include requests for RTX 5090 support and reports that older PyTorch versions do not work with 50-series Nvidia cards.Read 21 Oct 2026 · github.com
- Community complaint: output quality
- Open issues include reports of red faces, blurry mouths, face misalignment and visible teeth artifacts.Read 21 Oct 2026 · github.com
- Community issue volume
- The GitHub issue tracker showed 211 open issues and 16 pull requests when opened.Read 21 Oct 2026 · github.com
- Data pipeline
- Its processing pipeline resamples video to 25 FPS and audio to 16 kHz, detects scenes, splits clips into 5–10 second segments, aligns faces and filters low-quality samples.Read 21 Oct 2026 · raw.githubusercontent.com
- Data processing tools
- The documented data pipeline uses PySceneDetect for scene detection and InsightFace landmarks to transform faces.Read 8 Oct 2026 · github.com
- Dependencies
- Pinned dependencies include PyTorch 2.5.1, Diffusers 0.32.2, Transformers 4.48.0, Gradio 5.24.0 and GPU ONNX Runtime 1.21.0.Read 21 Oct 2026 · github.com
- Developer extensibility
- The repository exposes scripts for preprocessing, inference, U-Net training and file-list generation, enabling custom training workflows.Read 21 Oct 2026 · raw.githubusercontent.com
- Efficient training tradeoff
- The 20 GB efficient Stage 2 configuration may slightly reduce visual quality and temporal consistency compared with the 30 GB configuration.Read 21 Oct 2026 · raw.githubusercontent.com
- Environment
- The setup script creates a Conda environment with Python 3.10.13, installs FFmpeg and requires the Linux package libgl1.Read 21 Oct 2026 · github.com
- Generation trade-off
- The README says more inference steps can improve visual quality but slow generation, while higher guidance scale can improve lip-sync accuracy but cause distortion or jitter.Read 8 Oct 2026 · github.com
- Hardware requirement
- The stated minimum inference VRAM is 8 GB for LatentSync 1.5 and 18 GB for LatentSync 1.6.Read 8 Oct 2026 · github.com
- Hosted API inputs
- Layer's schema requires at least one source-video file and one reference-audio file and supports seed and guidance-scale fields.Read 21 Oct 2026 · layer.ai
- Hosted API option
- Layer documents a POST inference endpoint that accepts a source video and reference audio through bearer-authenticated JSON requests.Read 21 Oct 2026 · layer.ai
- Hugging Face availability
- The model page provides Diffusers usage instructions and links to Google Colab and Kaggle notebooks.Read 21 Oct 2026 · huggingface.co
- Independent description
- ExplainGit describes LatentSync as a ByteDance research tool that replaces a video's lip movements to match a new audio track using Stable Diffusion and Whisper.Read 21 Oct 2026 · explaingit.com
- Independent setup assessment
- ExplainGit rates setup difficulty as hard and estimates more than one hour to reach a first run.Read 21 Oct 2026 · explaingit.com
- Inference interfaces
- The repository provides both a local Gradio application and a command-line inference script.Read 21 Oct 2026 · raw.githubusercontent.com
- Inference memory 1.5
- LatentSync 1.5 requires at least 8 GB of VRAM for inference.Read 21 Oct 2026 · raw.githubusercontent.com
- Inference memory 1.6
- LatentSync 1.6 requires at least 18 GB of VRAM for inference.Read 21 Oct 2026 · raw.githubusercontent.com
- Inference options
- The project provides a Gradio app and a command-line interface for inference.Read 8 Oct 2026 · github.com
- Limit: hardware
- Zenn identifies relatively high hardware requirements and recommends about 20 GB of VRAM.Read 21 Oct 2026 · zenn.dev
- Limit: hosted support
- Shotcraft lists no hosted API, SLA or vendor support for LatentSync.Read 21 Oct 2026 · shotcraft.video
- Limit: resolution detail
- Zenn reports detail loss associated with the 256×256 resolution of training data in the compared version.Read 21 Oct 2026 · zenn.dev
- Limit: setup
- Zenn characterizes installation as somewhat complex for users.Read 21 Oct 2026 · zenn.dev
- Limit: undocumented behavior
- Shotcraft says maximum video duration and supported aspect ratios are not established in the available documentation.Read 21 Oct 2026 · shotcraft.video
- Maker founding
- Dealroom lists ByteDance's launch date as March 2012.Read 21 Oct 2026 · app.dealroom.co
- Maker headquarters
- Dealroom places ByteDance's headquarters in Haidian, China.Read 21 Oct 2026 · app.dealroom.co
- Maker legal name
- Crunchbase identifies the legal entity as ByteDance Ltd.Read 21 Oct 2026 · crunchbase.com
- Maker size
- LinkedIn lists ByteDance as having more than 10,001 employees and says the company has over 110,000 employees globally.Read 21 Oct 2026 · linkedin.com
- Model downloads
- The setup script downloads the Whisper tiny checkpoint and LatentSync U-Net checkpoint from Hugging Face.Read 21 Oct 2026 · github.com
- Model hosting
- The Hugging Face model page says LatentSync 1.6 is not deployed by an Inference Provider.Read 8 Oct 2026 · huggingface.co
- Model-card license
- The ByteDance Hugging Face model page labels the model license openrail++.Read 21 Oct 2026 · huggingface.co
- Praise: accuracy
- A Zenn comparison calls LatentSync the most accurate open-source lip-sync technology in its 2025 comparison.Read 21 Oct 2026 · zenn.dev
- Praise: temporal quality
- The same comparison credits LatentSync with improved temporal consistency and reduced frame flicker.Read 21 Oct 2026 · zenn.dev
- Release history
- The repository has no GitHub Releases entries even though its README documents versions 1.5 and 1.6.Read 21 Oct 2026 · github.com
- Replicate availability
- Replicate exposes ByteDance LatentSync through its API and reports more than 178,000 runs on the opened page.Read 21 Oct 2026 · replicate.com
- Security and compliance
- The GitHub repository identifies its license as Apache-2.0; the Hugging Face model card identifies its license as openrail++.Read 8 Oct 2026 · github.com
- Stable Diffusion basis
- The framework uses Stable Diffusion to model complex audio-visual relationships directly.Read 21 Oct 2026 · raw.githubusercontent.com
- Temporal losses
- Training includes TREPA, LPIPS and SyncNet losses in pixel space.Read 21 Oct 2026 · raw.githubusercontent.com
- Training hardware
- The listed U-Net training configurations require 20–55 GB of VRAM, depending on the configuration.Read 8 Oct 2026 · github.com
- Training resources
- The project publishes inference code, checkpoints, a data-processing pipeline and training code.Read 21 Oct 2026 · raw.githubusercontent.com
- Training VRAM
- The documented training configurations require between 20 GB and 55 GB of VRAM, depending on stage and resolution.Read 21 Oct 2026 · raw.githubusercontent.com
- Version 1.5 news
- ByteDance says version 1.5 was released on 2025-03-14 with temporal layers, better Chinese-video performance and 20 GB Stage 2 training optimization.Read 21 Oct 2026 · raw.githubusercontent.com
- Version 1.6
- LatentSync 1.6 was trained on 512 × 512 resolution videos to address blurry teeth and lips.Read 8 Oct 2026 · huggingface.co
- Version 1.6 news
- ByteDance says LatentSync 1.6 was released on 2025-06-11 and trained at 512×512 resolution to reduce blurriness.Read 21 Oct 2026 · raw.githubusercontent.com
Plans and prices
Open-source release Free Inference code and checkpoints · data processing pipeline · training code Read 8 Oct 2026 · github.com
Strong points and limits
Strong points
- Free open-source release
- Provides Gradio and command-line inference
- Includes checkpoints, pipeline and training code
- Supports self-hosted Linux deployment
- Offers two documented inference versions
Limits
- LatentSync 1.6 needs 18 GB VRAM
- Training configurations require 20–55 GB VRAM
- More inference steps slow generation
- Higher guidance can cause distortion or jitter
Who it suits
It suits developers and creators able to run Linux, self-host models and provide the stated GPU memory. Consumer-grade GPU users are specifically mentioned for an efficient stage-two training configuration.
Verdict
LatentSync supplies an open implementation for audio-driven lip synchronization, with inference and training resources included. Hardware needs vary sharply by version and training configuration, so deployment planning matters.
Where Specifiction places it
- AI Video Lip Sync Tools#4 of 288.6
