Skip to content
SPECIFICTION
Get listed

Diff2Lip

github.com

WebLinuxSelf-hosted
Visit github.com
Spec score 7.9 #10 of 28 in AI Video Lip Sync Tools

What it is

Diff2Lip is a free, open-source Linux and self-hosted project that synchronizes speech with face videos using an audio-conditioned diffusion model. It accepts speech and face-video material and generates lip-synced video while aiming to preserve the identity shown in source frames. The repository includes inference scripts for audio-video pairs and for a single video. Cross mode drives a video with an audio source; reconstruction mode drives the video's first frame with audio. The project lists movies, education and virtual avatars as applications, with video conferencing described as a possible future use. Diff2Lip is not real time yet. Reported evaluations use the VoxCeleb2 and LRW datasets, and the repository identifies the work as a WACV 2024 paper. Setup instructions use Python 3.9, FFmpeg 5.0.1 and the requirements file. Inference supports one or more GPUs through NUM_GPUS, with values above one running distributed generation. A Google Colab notebook is linked for trying it.

Spec sheet

Purpose
Diff2Lip generates lip-synchronized videos from speech audio and face videos.Read 10 Oct 2026 · github.com
Integrations
The repository links Google Colab for trying the project; it does not list other product integrations.Read 10 Oct 2026 · github.com
License
The repository states that its text and code are under CC BY-NC 4.0, with attribution required and commercial use prohibited.Read 10 Oct 2026 · github.com
Access
The repository links a Google Colab notebook to try Diff2Lip.Read 10 Oct 2026 · github.com
Applications
The project lists movies, education, and virtual avatars as applications, and video conferencing as a possible future application.Read 4 Oct 2026 · soumik-kanad.github.io
Commercial use
The repository's license description says the work may not be used for commercial purposes.Read 4 Oct 2026 · github.com
Datasets
The paper says the model was trained on VoxCeleb2 and reports results on VoxCeleb2 and LRW.Read 10 Oct 2026 · arxiv.org
GPU requirement
The inference instructions include a NUM_GPUS setting and say values greater than one run distributed generation.Read 4 Oct 2026 · github.com
Identity preservation
The project says its generated video frames preserve identity without identity loss.Read 4 Oct 2026 · soumik-kanad.github.io
Inference
The repository includes scripts for lip-sync inference on audio-video pairs and on a single video.Read 4 Oct 2026 · github.com
Inference inputs
The repository includes inference scripts for audio-video pairs and supports running on a single video with specified video, audio, and output paths.Read 10 Oct 2026 · github.com
Inference modes
The repository documents cross mode to drive a video with an audio source and reconstruction mode to drive the video's first frame with an audio source.Read 4 Oct 2026 · github.com
Input and output
The project describes its task as turning arbitrary speech and face videos into high-quality lip-synced video.Read 4 Oct 2026 · github.com
Method
It uses an audio-conditioned diffusion model for lip synchronization in the wild.Read 10 Oct 2026 · soumik-kanad.github.io
Not real time
The project states that Diff2Lip is not real time yet.Read 4 Oct 2026 · soumik-kanad.github.io
Quality goals
The project describes its approach as preserving identity, pose, emotions, and image quality while generating lip movements.Read 10 Oct 2026 · arxiv.org
Requirements
The setup instructions use Python 3.9 and FFmpeg 5.0.1, and inference can use one or more GPUs.Read 10 Oct 2026 · github.com
Research publication
The repository identifies Diff2Lip as a WACV 2024 paper.Read 4 Oct 2026 · github.com
Research result
The paper reports better FID and user Mean Opinion Scores than Wav2Lip and PC-AVS in its evaluations.Read 10 Oct 2026 · arxiv.org
Research results
The project reports reconstruction and cross-audio-video results on VoxCeleb2 and LRW datasets.Read 4 Oct 2026 · soumik-kanad.github.io
Setup
The README setup instructions use Python 3.9, FFmpeg 5.0.1, and the repository's requirements file.Read 4 Oct 2026 · github.com
Speed limit
The project website says Diff2Lip is not real time yet.Read 10 Oct 2026 · soumik-kanad.github.io
Try it
The project links to a Google Colab notebook for trying Diff2Lip.Read 4 Oct 2026 · github.com
Use cases
The project lists movies, education, virtual avatars, and eventually video conferencing as applications.Read 10 Oct 2026 · soumik-kanad.github.io

Strong points and limits

Strong points

  • Open-source and free to use
  • Preserves source identity in generated frames
  • Includes cross and reconstruction inference modes
  • Google Colab notebook is available
  • Supports multi-GPU distributed generation

Limits

  • Not real time yet
  • Commercial use is prohibited by the license
  • Requires Python 3.9 and FFmpeg 5.0.1
  • Linux and self-hosted setup required

Who it suits

Diff2Lip suits developers and researchers who can run a self-hosted Linux project and need speech-driven lip synchronization. It is relevant to video production, education and virtual-avatar work.

Verdict

Diff2Lip provides documented inference workflows for generating lip-synced video from audio and faces. Its non-real-time performance, technical setup and noncommercial CC BY-NC 4.0 license are central constraints.

Where Specifiction places it

Alternatives to Diff2LipAll alternatives →