DreamTalk

Spec sheet

What it does

DreamTalk is a diffusion-based framework for generating expressive talking-head videos from audio.Read 4 Oct 2026 · dreamtalk-project.github.io

Purpose

DreamTalk is a diffusion-based framework for generating expressive talking-head videos from audio.Read 9 Oct 2026 · dreamtalk-project.github.io

Support

The project page invites researchers interested in joining the team to email a resume to [email protected].Read 9 Oct 2026 · dreamtalk-project.github.io

License

The repository includes an MIT license, while its README says the method is intended for research and non-commercial use only.Read 9 Oct 2026 · github.com

Founded

2023Read 28 Sep 2026 · dreamtalk-project.github.io

Checkpoint access

Public checkpoint downloads have been discontinued; the README asks researchers to request checkpoints by email and states that doing so implies consent to use them solely for academic research.Read 9 Oct 2026 · github.com

CPU support

The repository says the program can run on CPU by adding the --device=cpu argument.Read 4 Oct 2026 · github.com

Duration limit

The max_gen_len setting controls the maximum generated video duration in seconds, and longer input audio is truncated.Read 9 Oct 2026 · github.com

Expressions

Its style predictor predicts a target expression from audio without requiring an expression reference video or text.Read 9 Oct 2026 · dreamtalk-project.github.io

Generation approach

The framework combines a denoising network, a style-aware lip expert, and a style predictor.Read 4 Oct 2026 · dreamtalk-project.github.io

How it works

The framework combines a denoising network, a style-aware lip expert, and a style predictor.Read 9 Oct 2026 · dreamtalk-project.github.io

Inference inputs

Inference accepts WAV, MP3, M4A, or MP4 audio, a portrait image, and reference 3DMM sequences for speaking style and head pose.Read 9 Oct 2026 · github.com

Input audio

The inference instructions list WAV, MP3, M4A, and MP4 files with sound as compatible audio inputs.Read 4 Oct 2026 · github.com

Input coverage

The project reports generalization across songs, noisy audio, out-of-domain portraits, and speech in Chinese, French, German, Italian, Japanese, Korean, and Spanish.Read 9 Oct 2026 · dreamtalk-project.github.io

Languages and audio

The project page lists speech examples in Chinese, French, German, Italian, Japanese, Korean, and Spanish, as well as songs and noisy audio.Read 4 Oct 2026 · dreamtalk-project.github.io

Maintainers

The project page lists authors affiliated with Tsinghua University, Alibaba Group, and Huazhong University of Science and Technology.Read 9 Oct 2026 · dreamtalk-project.github.io

Motion references

Speaking-style and head-pose references use 3DMM parameter sequences extracted from videos, and the stated video frame rate is 25 FPS.Read 4 Oct 2026 · github.com

Output

The inference script saves the generated video as an MP4 file in the output_video folder.Read 4 Oct 2026 · github.com

Portrait requirements

The input portrait should be larger than 256×256 pixels; the instructions say frontal portraits usually produce satisfactory results and are cropped to 256×256.Read 4 Oct 2026 · github.com

Resolution

The README says resolution was not considered in the initial design and describes optional super-resolution methods with tradeoffs in speed, temporal consistency, or facial emotion intensity.Read 9 Oct 2026 · github.com

Setup

The repository provides inference code and installation instructions for a Python 3.7 environment with PyTorch 1.8.0 and CUDA 11.1 packages.Read 9 Oct 2026 · github.com

Speaking styles

The project demonstrates speaking-style manipulation by adjusting classifier-free guidance and interpolating style codes.Read 9 Oct 2026 · dreamtalk-project.github.io

Style prediction

Its style predictor can infer a target speaking style from audio without an expression reference video or text.Read 4 Oct 2026 · dreamtalk-project.github.io

Use restriction

The repository states that the method is intended for research and non-commercial use only.Read 4 Oct 2026 · github.com

Where Specifiction places it

• AI Video Lip Sync Tools#13 of 287.6