Apache Arrow DataFusion vs Apache Drill

Apache Arrow DataFusion is a free, extensible query engine written in Rust that uses Apache Arrow as its in-memory format. It is intended for developers building database and analytics systems tailored to particular workloads, rather than as a ready-made hosted database. Queries run in-process, using threads for parallel execution. DataFusion offers SQL and DataFrame APIs, while related subprojects provide Python and Java interfaces. Documented capabilities include SQL parsing and planning, parallel and streaming execution, and query optimization. Built-in data sources include CSV, Parquet, JSON, Avro, and Arrow. Developers can add sources through the TableProvider trait and customize functions, operators, and other components. Runtime features include enforced memory limits and disk spilling for sorts, grouping, and joins. The core project is designed for in-process use; Ballista is a related distributed processing extension. Rust users commonly add DataFusion from crates.io, while official Apache releases are source artifacts. The project recommends verifying release artifacts with an OpenPGP signature or SHA-512 checksum.

Apache Drill is a schema-free SQL query engine for Hadoop, NoSQL databases, and cloud storage. It can query raw data where it resides without first loading it, defining schemas, or transforming it. Its JSON data model supports complex and nested structures, and a single query can join data across multiple datastores. Listed sources include HBase, MongoDB, HDFS, Amazon S3, Azure Blob Storage, Google Cloud Storage, Swift, NAS, and local files. JDBC and ODBC drivers connect Drill to tools including Tableau, Qlik, MicroStrategy, Spotfire, SAS, and Excel; developers can also use its REST API in custom applications. Drill can run embedded on a laptop or in distributed mode on a server cluster, and the homepage lists Mac, Windows, and Linux. Documented security options include Kerberos, username and password authentication, Digest, authorization, and impersonation. The software is free under Apache License 2.0. A compatibility limit: official support for Hadoop 2 and Java 8 was dropped with Drill 1.22.0.