Apache DolphinScheduler vs Flyte

Apache DolphinScheduler is a free, open-source workflow orchestration platform for designing and running data workflows through visual directed-acyclic-graph interfaces or code. Workflows can be built with drag-and-drop tools or defined in Python, YAML, or the Open API. The project lists more than 30 task types, including Shell, SQL, Spark, Flink, Hive, Kubernetes, and HTTP. Scheduling supports cross-project and cross-workflow dependencies, pause and resume operations, and serial or parallel batch backfills. A decentralized multi-master, multi-worker design is described as increasing scheduling capacity with cluster size. Deployment options are standalone, cluster, and Kubernetes. Standalone is for evaluation, recommended for fewer than 20 workflows, and its in-memory H2 database is cleared when the service stops or restarts. Authentication includes password, LDAP, Casdoor SSO, and OAuth2, with administrator-managed resource controls. The security model assumes a trusted network.

Flyte 2 is a durable runtime for AI workloads involving training, serving and agents. Users write workflows as plain Python async functions and use a TaskEnvironment to declare images, resources, secrets and retry policies. Flyte resumes from failed tasks and can recover from out-of-memory kills, preempted nodes and disappearing GPUs. Branches, loops and runtime decisions support dynamic workflows and agents. Native serving integrations include Streamlit, FastAPI, vLLM, SGLang and Ollama, with replicas able to scale to zero when idle. Listed integrations also cover Apache Spark, Ray, PyTorch Elastic, BigQuery, Snowflake, Weights & Biases, OpenAI Agents SDK, LangGraph, CrewAI, Pydantic AI, MLflow, Dask and Databricks. It runs locally with a devbox or on a customer cluster. Union.ai says customer-cloud data planes keep workflow inputs, outputs, logs, code and reports outside its control plane. Flyte is available as free Apache 2.0 open-source software.