Apache Arrow DataFusion vs Query

Apache Arrow DataFusion is a free, extensible query engine written in Rust that uses Apache Arrow as its in-memory format. It is intended for developers building database and analytics systems tailored to particular workloads, rather than as a ready-made hosted database. Queries run in-process, using threads for parallel execution. DataFusion offers SQL and DataFrame APIs, while related subprojects provide Python and Java interfaces. Documented capabilities include SQL parsing and planning, parallel and streaming execution, and query optimization. Built-in data sources include CSV, Parquet, JSON, Avro, and Arrow. Developers can add sources through the TableProvider trait and customize functions, operators, and other components. Runtime features include enforced memory limits and disk spilling for sorts, grouping, and joins. The core project is designed for in-process use; Ballista is a related distributed processing extension. Rust users commonly add DataFusion from crates.io, while official Apache releases are source artifacts. The project recommends verifying release artifacts with an OpenPGP signature or SHA-512 checksum.

Query is a cloud security data mesh that searches distributed security sources without copying their data into a central store. Federated search queries connected systems in parallel and normalizes results to OCSF, while Federated Search Query Language translates general queries into each source’s native language at query time. Query Workers are AI agents that triage, investigate, hunt, and detect across connected sources; analysts review recommendations or inconclusive findings. Federated Detections run SPL, KQL, or Sigma rules where the data resides. Query says it supports more than 50 sources, including Splunk, Microsoft Sentinel, CrowdStrike Falcon, Okta, Snowflake, and Google BigQuery. Static connectors map fixed schemas; dynamic connectors let customers configure custom mappings. Access is read-only, with no ingestion required and data staying at its source. The SaaS product runs on AWS data centers in us-east-1. Single sign-on supports Google, Microsoft, and Okta providers. Query states SOC 2 certification, uses NIST CSF 800-53 as a control baseline, and offers weekday standard support plus paid Platinum support.