Hallo vs LatentSync

Hallo generates an animated portrait video from a still image and driving speech audio. Its hierarchical audio-driven synthesis is designed to align lip, facial-expression, and pose movement with the audio, while separate inference weights let you adjust those three motion areas. The source image should be square, show a forward-facing face occupying 50%–70% of the frame, and have less than 30 degrees of rotation. Driving audio must be a clear-vocal English WAV file; background music is allowed. Inference saves an MP4 to `.cache/output.mp4` by default, with another output path available. Listed requirements are Ubuntu 20.04 or 22.04 and CUDA 12.1, with an A100 named as the tested GPU. Pretrained checkpoints and supporting models are available through Hugging Face. The project lists Linux, Windows, and self-hosted platforms, with community resources for Windows, ComfyUI, a WebUI, Docker, and RunPod. Its repository includes preprocessing and training code for multi-node and multi-GPU work. Hallo is free, open source under MIT, and its training data is English, so driving audio is limited to English.

LatentSync is a free, open-source lip-sync method for Linux and self-hosted deployment. It uses audio-conditioned latent diffusion models: Whisper converts mel spectrograms into audio embeddings, which enter a U-Net through cross-attention layers. The project offers a Gradio application and command-line inference, plus checkpoints, training code and a data-processing pipeline. LatentSync 1.6 was trained on 512 × 512 videos to address blurry teeth and lips. Minimum inference memory is listed as 8 GB of VRAM for version 1.5 and 18 GB for version 1.6. More inference steps can improve visual quality but slow generation; increasing guidance scale can improve lip-sync accuracy while introducing distortion or jitter. Training configurations for the U-Net require 20–55 GB of VRAM, depending on configuration. The documented pipeline uses PySceneDetect for scene detection and InsightFace landmarks for face transformation. The repository lists Apache-2.0, while the model card lists openrail++.