Apache DolphinScheduler vs Prefect

Apache DolphinScheduler is a free, open-source workflow orchestration platform for designing and running data workflows through visual directed-acyclic-graph interfaces or code. Workflows can be built with drag-and-drop tools or defined in Python, YAML, or the Open API. The project lists more than 30 task types, including Shell, SQL, Spark, Flink, Hive, Kubernetes, and HTTP. Scheduling supports cross-project and cross-workflow dependencies, pause and resume operations, and serial or parallel batch backfills. A decentralized multi-master, multi-worker design is described as increasing scheduling capacity with cluster size. Deployment options are standalone, cluster, and Kubernetes. Standalone is for evaluation, recommended for fewer than 20 workflows, and its in-memory H2 database is cleared when the service stops or restarts. Authentication includes password, LDAP, Casdoor SSO, and OAuth2, with administrator-managed resource controls. The security model assumes a trusted network.

Prefect is a workflow orchestration platform for data, machine learning and agent workflows written in Python. Its open-source framework lets users add decorators to existing Python code to create workflows, without a rigid DAG structure or custom DSL. Execution features include state tracking, automatic retries, caching and recovery from failures. For observability, Prefect provides run timelines, logs and alerts for failures, missed SLAs and late runs. Workflows can run on cron or event-driven schedules, support backfills, and be triggered by webhooks, cloud events or state changes. Deployment choices include managed serverless execution, hybrid execution in a customer’s infrastructure, and self-hosted orchestration. In hybrid mode, workers connect through outbound-only connections while code and data stay in the customer’s infrastructure; Prefect Cloud hosts the control plane and metadata. The free Hobby plan is limited to two users, five deployments, 500 minutes of monthly serverless credits and seven days of run retention. Paid plans start at $100/mo.