Apache Arrow DataFusion vs Teiid

Apache Arrow DataFusion is a free, extensible query engine written in Rust that uses Apache Arrow as its in-memory format. It is intended for developers building database and analytics systems tailored to particular workloads, rather than as a ready-made hosted database. Queries run in-process, using threads for parallel execution. DataFusion offers SQL and DataFrame APIs, while related subprojects provide Python and Java interfaces. Documented capabilities include SQL parsing and planning, parallel and streaming execution, and query optimization. Built-in data sources include CSV, Parquet, JSON, Avro, and Arrow. Developers can add sources through the TableProvider trait and customize functions, operators, and other components. Runtime features include enforced memory limits and disk spilling for sorts, grouping, and joins. The core project is designed for in-process use; Ballista is a related distributed processing extension. Rust users commonly add DataFusion from crates.io, while official Apache releases are source artifacts. The project recommends verifying release artifacts with an OpenPGP signature or SHA-512 checksum.

Teiid is a free, open-source data virtualization project that gives applications a uniform way to query multiple underlying sources. It works as a gateway rather than storing the data, and virtual databases can present separate sources together so applications can join or union their results. Access is available through JDBC, ODBC, OData, and REST. Built-in translators are listed for Oracle, DB2, SQL Server, MySQL, PostgreSQL, and files; custom translators can also be made. Its optimizer uses source capability information to send query work to underlying systems where possible. Documented runtimes include Spring Boot, WildFly, embedded Java, and OpenShift, with Spring Boot noted as less feature-complete than WildFly. Teiid is aimed especially at Java developers integrating enterprise data sources. Connecting to sources commonly requires third-party drivers that are not included. The FAQ cautions that big-data, LDAP-style, and primarily streaming workloads may not be a good fit. Teiid Designer is deprecated, with no further releases expected.