Databricks Workflows vs Luigi

Databricks Workflows has no write-up on Specifiction yet; its figures are above.

Luigi is a free Python package for building and coordinating pipelines of batch jobs. It manages dependencies, workflow execution, visualization, failure handling and command-line integration. The documentation presents it as workflow plumbing for long-running work such as Hadoop jobs, database transfers and machine-learning jobs, including pipelines with thousands of tasks that can take days or weeks. Documented systems include Hadoop, Hive, Pig, Spark, HDFS, databases and local files. A Luigi server provides a web interface for searching and filtering tasks, while its HDFS and local filesystem abstractions support atomic file operations. The project instructs users to install the latest stable version with `pip install luigi`; TOML configuration is available through the optional `luigi[toml]` installation. The repository says Python 3.10 through 3.14 are tested and identifies Apache-2.0 as the license. Luigi is self-hosted, and Spotify's Data Team maintains the project. A mailing list is linked for discussion and questions. Its security policy directs people to report sensitive issues through Spotify's bug bounty program rather than GitHub.