Google Cloud Workflows vs Luigi

Google Cloud Workflows is a managed orchestration platform for running Google Cloud services, custom services, and HTTP APIs in a chosen sequence. Workflow definitions use YAML or JSON and can combine ordered steps with conditions, loops, parallel work, and reusable subworkflows. Start an execution manually, through client libraries or REST, or on a schedule with Cloud Scheduler. Connectors link workflows to Google Cloud APIs such as Pub/Sub, Firestore, and Secret Manager, while workflow steps can also call custom or private HTTP and HTTPS endpoints. Reliability features include retries, polling, error handling, callbacks, and waits of up to one year. A single execution can run up to 100,000 steps and last no more than one year. IAM controls access; Secret Manager can store and retrieve secrets. Cloud Logging and Cloud Monitoring surface execution logs, volume, errors, and duration. Pricing is based on executed steps, including failed and retried steps, while idle workflows incur no charges. The free plan includes 5,000 internal and 2,000 external steps; paid usage starts at $0.01 per month, with rates that vary by step type and volume.

Luigi is a free Python package for defining and coordinating batch-job pipelines, dependencies, workflow execution, and failures. Workflows and dependency graphs are specified in Python, and tasks can also trigger work outside Python. Its documentation describes task support for Hadoop, Hive, Pig, and Spark, plus HDFS and local file abstractions. Those abstractions make file operations atomic, helping avoid pipelines being left with partial data. The Luigi server includes a web interface for searching and filtering tasks and viewing dependency graphs. It also supports command-line integration and use from Python code. The documentation describes pipelines with thousands of tasks that may run for days or weeks, including Hadoop jobs, database transfers, and machine-learning algorithms. The package is self-hosted, with no registration wall. Install the latest stable version with `pip install luigi`; adding `[toml]` enables TOML-based configuration. Documentation lists Python 3.10, 3.11, 3.12, and 3.13 as tested versions. A mailing list is available for discussion and questions.