Apache Arrow DataFusion vs Apache Hive

Apache Arrow DataFusion is a free, extensible query engine written in Rust that uses Apache Arrow as its in-memory format. It is intended for developers building database and analytics systems tailored to particular workloads, rather than as a ready-made hosted database. Queries run in-process, using threads for parallel execution. DataFusion offers SQL and DataFrame APIs, while related subprojects provide Python and Java interfaces. Documented capabilities include SQL parsing and planning, parallel and streaming execution, and query optimization. Built-in data sources include CSV, Parquet, JSON, Avro, and Arrow. Developers can add sources through the TableProvider trait and customize functions, operators, and other components. Runtime features include enforced memory limits and disk spilling for sorts, grouping, and joins. The core project is designed for in-process use; Ballista is a related distributed processing extension. Rust users commonly add DataFusion from crates.io, while official Apache releases are source artifacts. The project recommends verifying release artifacts with an OpenPGP signature or SHA-512 checksum.

Apache Hive is a distributed data warehouse system for reading, writing, and managing large datasets with SQL. Its Metastore keeps shared metadata for tables and partitions and serves clients including Hive, Impala, and Spark; HiveServer2 supports concurrent clients through JDBC and ODBC. Hive provides full ACID transactions for ORC tables and insert-only transactions for other formats. Documented storage options include S3, Azure Data Lake, and Google Cloud Storage, and Hive supports Apache Iceberg tables through a StorageHandler. Named integrations include Spark, Presto, and Impala, with Apache Ranger for authorization and Apache Atlas for lineage and governance. Security options described by the project include Kerberos, fine-grained access control, and audit logging; HiveServer2 also lists LDAP, PAM, and custom authentication. The quickstart includes Docker instructions for the server and Metastore. Hive is free open-source software. The installation manual lists Java 8, Maven 3.6.3, Protobuf 2.5, Hadoop 3.3.6, and Tez among current prerequisites; Hive 4.2.x instead requires JDK 21 minimum and works with Hadoop 3.4.1 and Tez 0.10.5.