Luigi vs Windmill

Luigi is a free Python package for defining and coordinating batch-job pipelines, dependencies, workflow execution, and failures. Workflows and dependency graphs are specified in Python, and tasks can also trigger work outside Python. Its documentation describes task support for Hadoop, Hive, Pig, and Spark, plus HDFS and local file abstractions. Those abstractions make file operations atomic, helping avoid pipelines being left with partial data. The Luigi server includes a web interface for searching and filtering tasks and viewing dependency graphs. It also supports command-line integration and use from Python code. The documentation describes pipelines with thousands of tasks that may run for days or weeks, including Hadoop jobs, database transfers, and machine-learning algorithms. The package is self-hosted, with no registration wall. Install the latest stable version with `pip install luigi`; adding `[toml]` enables TOML-based configuration. Documentation lists Python 3.10, 3.11, 3.12, and 3.13 as tested versions. A mailing list is available for discussion and questions.

Windmill is an open-source workflow engine and developer platform for building endpoints, workflows and user interfaces. It is aimed at teams making internal software across engineering, data, sales, support and marketing. Scripts can be written in TypeScript, Python, Go, PHP, Bash, C#, SQL, Rust, Ruby and R, among other languages. Windmill combines a script editor, low-code flow builder, and low-code, React or Svelte app builders. Scripts and flows can be triggered by webhooks, schedules, HTTP routes, queues or an open API. Its Hub lists more than 200 resource types, including GitHub, Slack, AWS, BigQuery and PostgreSQL; users can also define custom resource types with schemas. Local development through the CLI and Git sync support version control and deployment workflows. Self-hosting is supported. The free and open-source plan is free and includes three workspaces, 50 users, 10 GiB of workspace object storage and 100 guest app users per 30 days. Enterprise is $120 a month and includes a month-long trial. Enterprise features include granular permissions, RBAC, secret management, OAuth, SSO and audit logs.