Apache Flink vs Arroyo

Apache Flink is a free, self-hosted framework and distributed processing engine for stateful computations over unbounded and bounded data streams. It supports event-driven applications, stream and batch analytics, and data pipelines and ETL. Listed correctness features include exactly-once state consistency, event-time processing and late-data handling. APIs include SQL for stream and batch data, DataStream and ProcessFunction for time and state. Operational capabilities cover flexible deployment, high availability, savepoints, scale-out architecture, very large state and incremental checkpoints. The project lists low latency, high throughput and in-memory computing. Flink runs in common cluster environments; for Kubernetes, the Kubernetes Operator is identified as the standard deployment mechanism. Connectors include AWS, Cassandra, Elasticsearch, Google Cloud PubSub, HBase, Hive, JDBC, Kafka, MongoDB, OpenSearch, Prometheus, Pulsar and RabbitMQ. Security requires configuration because SSL/TLS, REST authentication and SQL Gateway authentication are disabled by default. It is available at free under Apache License v2.

Arroyo is an open-source distributed engine for processing stateful computations on data streams, with sub-second results. Users build streaming pipelines in analytical SQL, including complex queries, windows and joins. Arroyo supports checkpointing for fault tolerance and recovery, exactly-once processing, and horizontal and vertical rescaling; it says the engine can handle millions of events per second. Its connectors include Kafka, Kinesis, MQTT, MySQL, Postgres, Redis, Delta Lake, HTTP and WebSocket, among others. It reads and writes JSON, Avro, Parquet, text and binary, and includes more than 300 SQL functions. Rust scalar, aggregate and asynchronous user-defined functions can extend SQL. The web UI supports connection management, query development and testing, and pipeline monitoring; pipeline management is also available through a REST API. Single-node clusters are intended for development and testing, while production deployments use a distributed cluster with Arroyo’s scheduler or Kubernetes. It is Apache 2.0 licensed, free at free, and self-hostable.