Apache Arrow DataFusion vs Apache Impala

Apache Arrow DataFusion is a free, extensible query engine written in Rust that uses Apache Arrow as its in-memory format. It is intended for developers building database and analytics systems tailored to particular workloads, rather than as a ready-made hosted database. Queries run in-process, using threads for parallel execution. DataFusion offers SQL and DataFrame APIs, while related subprojects provide Python and Java interfaces. Documented capabilities include SQL parsing and planning, parallel and streaming execution, and query optimization. Built-in data sources include CSV, Parquet, JSON, Avro, and Arrow. Developers can add sources through the TableProvider trait and customize functions, operators, and other components. Runtime features include enforced memory limits and disk spilling for sorts, grouping, and joins. The core project is designed for in-process use; Ballista is a related distributed processing extension. Rust users commonly add DataFusion from crates.io, while official Apache releases are source artifacts. The project recommends verifying release artifacts with an OpenPGP signature or SHA-512 checksum.

Apache Impala is a free, open-source native analytic database for SQL queries on open data and table formats. It targets low-latency, high-concurrency business intelligence and analytics in Hadoop environments and is described as scaling linearly, including for multitenant use. Impala works with HDFS, HBase and Amazon S3, plus Apache Kudu, Apache Iceberg and other listed sources. Supported formats include delimited text, Parquet, Avro, SequenceFile and RCFile, with Snappy, GZIP, Deflate and BZIP codecs. Clients can connect through impala-shell, Hue, JDBC or ODBC. The system reuses Hadoop file formats, metadata, security and resource-management frameworks, and shares metadata and an ODBC driver with Apache Hive. Kudu tables support queries and efficient updates or deletes; Iceberg tables can be registered with the Hive Metastore. Kerberos, Apache Ranger authorization, auditing and encrypted JDBC or ODBC connections are supported. Impala runs on Linux and macOS, supports API use and self-hosted deployment, and is licensed under Apache License 2.0.