Apache Spark vs Arroyo

Apache Spark is a free, open-source engine for large-scale data analytics. It processes batches and real-time streams using Python, SQL, Scala, Java, or R. Spark SQL runs distributed ANSI SQL queries on structured tables and unstructured data such as JSON or images; MLlib provides machine-learning algorithms for local use or cluster scale. The project also describes exploratory analysis on petabyte-scale data without downsampling. Listed ecosystem components include pandas, TensorFlow, PyTorch, scikit-learn, Apache Kafka, Kubernetes, Delta Lake, and Apache Iceberg. Deployment options include standalone, Hadoop YARN, and Kubernetes. Spark runs on Windows and UNIX-like systems, including Linux and macOS, when a supported Java version is available. The 4.2.0 documentation lists Java 17, 21, or 25, Scala 2.13, Python 3.10+, and R 4.0+ (deprecated). Spark Connect separates client applications from clusters for remote access. Security is not enabled by default; Spark supports TLS for RPC connections and encryption of temporary local-disk data. The project is licensed under Apache License, Version 2.0.

Arroyo is an open-source distributed engine for processing stateful computations on data streams, with sub-second results. Users build streaming pipelines in analytical SQL, including complex queries, windows and joins. Arroyo supports checkpointing for fault tolerance and recovery, exactly-once processing, and horizontal and vertical rescaling; it says the engine can handle millions of events per second. Its connectors include Kafka, Kinesis, MQTT, MySQL, Postgres, Redis, Delta Lake, HTTP and WebSocket, among others. It reads and writes JSON, Avro, Parquet, text and binary, and includes more than 300 SQL functions. Rust scalar, aggregate and asynchronous user-defined functions can extend SQL. The web UI supports connection management, query development and testing, and pipeline monitoring; pipeline management is also available through a REST API. Single-node clusters are intended for development and testing, while production deployments use a distributed cluster with Arroyo’s scheduler or Kubernetes. It is Apache 2.0 licensed, free at free, and self-hostable.