FoleyCrafter

Spec sheet

What it does

FoleyCrafter is a text-based video-to-audio generation framework that generates high-quality audio semantically relevant to and temporally synchronized with input videos.Read 9 Oct 2026 · foleycrafter.github.io

Purpose

FoleyCrafter is a text-based video-to-audio generation framework that produces high-quality audio semantically relevant and temporally synchronized with input video.Read 10 Oct 2026 · foleycrafter.github.io

Integrations

The project is built upon Auffusion, CondFoleyGen, and SpecVQGAN.Read 10 Oct 2026 · github.com

Support

The repository README lists contact email addresses for Yiming Zhang, YiCheng Gu, and Yanhong Zeng.Read 10 Oct 2026 · github.com

License

The repository lists an Apache-2.0 license and advises commercial users to check the Auffusion license as well.Read 10 Oct 2026 · github.com

Base model

FoleyCrafter is built on a pretrained text-to-audio generator, and the project page says the T2A model remains fixed during the framework's training.Read 9 Oct 2026 · foleycrafter.github.io

Checkpoints

The repository says checkpoints download automatically when running `inference.py` and also documents manual downloads.Read 9 Oct 2026 · github.com

Code and models

The project provides its models and code through its GitHub repository.Read 9 Oct 2026 · github.com

Dependencies

The repository setup instructions use a Conda environment and Git LFS to download checkpoints.Read 9 Oct 2026 · github.com

Hosted demo

The linked Hugging Face Space is paused, and its page says users can ask its authors to restart it.Read 10 Oct 2026 · huggingface.co

How to run

The repository documents a Gradio interface launched with `python app.py --share` and command-line inference through `inference.py`.Read 9 Oct 2026 · github.com

Implementation

The framework has a Gradio interface and command-line inference usage.Read 10 Oct 2026 · github.com

Intended use

The project describes sound effects as useful for cinema and gaming to enhance realism, impact, and emotional depth.Read 10 Oct 2026 · github.com

Makers

The project page lists authors affiliated with Shanghai AI Laboratory and the Chinese University of Hong Kong, Shenzhen.Read 9 Oct 2026 · foleycrafter.github.io

Prompt control

Text prompts can be used to control video-to-audio generation according to user intent.Read 9 Oct 2026 · foleycrafter.github.io

Prompt controls

Text prompts support controllable video-to-audio generation, and the project demonstrates both positive and negative prompts.Read 10 Oct 2026 · github.com

Prompt options

The command-line interface includes prompt and negative-prompt options, a random seed, temporal alignment controls, input and checkpoint paths, and an output path.Read 9 Oct 2026 · github.com

Semantic alignment

Its semantic adapter conditions audio generation on video features to produce sounds relevant to the visual content.Read 10 Oct 2026 · foleycrafter.github.io

Setup

The README provides a Conda environment setup and says model checkpoints can be downloaded automatically by running inference.py.Read 10 Oct 2026 · github.com

Sound generation

The framework uses a pretrained text-to-audio model to generate sound effects for video.Read 10 Oct 2026 · foleycrafter.github.io

Temporal alignment

Its temporal controller uses an onset detector and timestamp-based adapter to align audio with video.Read 10 Oct 2026 · foleycrafter.github.io

Temporal synchronization

Its temporal controller uses an onset detector and timestamp-based adapter for audio-video alignment.Read 9 Oct 2026 · foleycrafter.github.io

Where Specifiction places it

• AI Sound Effect Generators#23 of 297.3