Apache Arrow DataFusion vs Apache Kylin

Apache Arrow DataFusion is a free, extensible query engine written in Rust that uses Apache Arrow as its in-memory format. It is intended for developers building database and analytics systems tailored to particular workloads, rather than as a ready-made hosted database. Queries run in-process, using threads for parallel execution. DataFusion offers SQL and DataFrame APIs, while related subprojects provide Python and Java interfaces. Documented capabilities include SQL parsing and planning, parallel and streaming execution, and query optimization. Built-in data sources include CSV, Parquet, JSON, Avro, and Arrow. Developers can add sources through the TableProvider trait and customize functions, operators, and other components. Runtime features include enforced memory limits and disk spilling for sorts, grouping, and joins. The core project is designed for in-process use; Ballista is a related distributed processing extension. Rust users commonly add DataFusion from crates.io, while official Apache releases are source artifacts. The project recommends verifying release artifacts with an OpenPGP signature or SHA-512 checksum.

Apache Kylin is an open-source OLAP engine for data analytics, designed for large-scale workloads with many concurrent users. Its pre-calculation approach is intended to support sub-second SQL queries. Kylin 5 can build models from SQL text, suggest index changes using query history, and use Apache Gluten as a native compute engine; the product overview describes a 2–4× improvement compared with vanilla Spark. For mixed data workloads, it supports Kafka streaming sources and fusion models for batch and streaming analysis, as well as internal tables for flexible queries and lakehouse scenarios. Teams can connect Tableau, Power BI, or Excel. A web UI supports table management, model design, job execution, and configuration. Deployment offers query, job, or all server modes. Kylin runs on Linux and uses Spark with Yarn or Standalone mode. Data storage can use HDFS or S3-compatible systems, while MySQL or PostgreSQL can hold metadata. The project is free under Apache License v2, with source and binary packages available. APIs use Basic Authentication; HTTPS is supported in Kylin 5 but disabled by default.