Diff2Lip vs Hallo

Diff2Lip is a free, open-source Linux and self-hosted project that synchronizes speech with face videos using an audio-conditioned diffusion model. It accepts speech and face-video material and generates lip-synced video while aiming to preserve the identity shown in source frames. The repository includes inference scripts for audio-video pairs and for a single video. Cross mode drives a video with an audio source; reconstruction mode drives the video's first frame with audio. The project lists movies, education and virtual avatars as applications, with video conferencing described as a possible future use. Diff2Lip is not real time yet. Reported evaluations use the VoxCeleb2 and LRW datasets, and the repository identifies the work as a WACV 2024 paper. Setup instructions use Python 3.9, FFmpeg 5.0.1 and the requirements file. Inference supports one or more GPUs through NUM_GPUS, with values above one running distributed generation. A Google Colab notebook is linked for trying it.

Hallo generates an animated portrait video from a still image and driving speech audio. Its hierarchical audio-driven synthesis is designed to align lip, facial-expression, and pose movement with the audio, while separate inference weights let you adjust those three motion areas. The source image should be square, show a forward-facing face occupying 50%–70% of the frame, and have less than 30 degrees of rotation. Driving audio must be a clear-vocal English WAV file; background music is allowed. Inference saves an MP4 to `.cache/output.mp4` by default, with another output path available. Listed requirements are Ubuntu 20.04 or 22.04 and CUDA 12.1, with an A100 named as the tested GPU. Pretrained checkpoints and supporting models are available through Hugging Face. The project lists Linux, Windows, and self-hosted platforms, with community resources for Windows, ComfyUI, a WebUI, Docker, and RunPod. Its repository includes preprocessing and training code for multi-node and multi-GPU work. Hallo is free, open source under MIT, and its training data is English, so driving audio is limited to English.