LatentSync vs MakeItTalk

LatentSync is a free, open-source lip-sync method for Linux and self-hosted deployment. It uses audio-conditioned latent diffusion models: Whisper converts mel spectrograms into audio embeddings, which enter a U-Net through cross-attention layers. The project offers a Gradio application and command-line inference, plus checkpoints, training code and a data-processing pipeline. LatentSync 1.6 was trained on 512 × 512 videos to address blurry teeth and lips. Minimum inference memory is listed as 8 GB of VRAM for version 1.5 and 18 GB for version 1.6. More inference steps can improve visual quality but slow generation; increasing guidance scale can improve lip-sync accuracy while introducing distortion or jitter. Training configurations for the U-Net require 20–55 GB of VRAM, depending on configuration. The documented pipeline uses PySceneDetect for scene detection and InsightFace landmarks for face transformation. The repository lists Apache-2.0, while the model card lists openrail++.

MakeItTalk is Linux, self-hosted software that creates expressive talking-head videos from one facial image and an audio track. Audio content drives lip movement and nearby facial motion, while speaker information influences expressions and other talking-head movement. The project describes use with human portraits, paintings, sketches, 2D cartoons, manga, and stylized caricatures, including faces or characters not seen during training. Its repository includes pretrained models, module training code, and Google Colab demos for natural faces. For human faces and paintings, the README calls for a centered 256×256 JPG portrait and a WAV file. Optional command-line settings adjust lip motion on the x and y axes and head-motion displacement. Local setup specifies Python 3.6, ffmpeg, and Python packages; cartoon face warping on Ubuntu also requires winehq-stable. Custom puppets are listed as a to-do item, and training sections for voice conversion, the speaker-aware branch, and image-to-image translation are marked unfinished. It is free open-source research code under CC BY-NC 4.0, allowing noncommercial use under that license.