data.table vs Polars

data.table is a free R package providing a high-performance version of base R’s data.frame. Its code interface supports filtering, grouping, aggregation, ordered and rolling joins, overlapping and non-equi joins, reshaping with dcast and melt, and adding, updating, or deleting columns. fread() reads delimited text, including CSV, TSV, compressed .gz and .bz2 files, and can source web URLs or shell commands. fwrite() writes delimited files with an emphasis on speed for large files. Many operations use multiple CPU threads internally. Columns can be changed by reference without copies, and the project reports no external Imports, LinkingTo, or Depends dependencies. Queries can call R functions from other packages. data.table runs on Linux, macOS, and Windows and requires base R. It is self-hosted and licensed under MPL-2.0. The project’s stated scope excludes binary formats such as Parquet and data stored on disk or in remote SQL databases.

Polars is an open-source DataFrame library for data manipulation, designed for single-machine processing with a typed, expressive API. Its multithreaded query engine is written in Rust and uses vectorized columnar processing to execute work in parallel. A query optimizer selects an execution plan, and the streaming API can process datasets larger than memory without holding all results in memory. Installation examples cover Python, Rust, and JavaScript. Supported formats include CSV, JSON, Parquet, Delta Lake, Avro, Excel, Feather, and Arrow IPC; the homepage also lists databases and cloud storage. Polars can read and write data in AWS S3, Azure Blob Storage, and Google Cloud Storage. The open-source library costs nothing under the MIT license. Polars Cloud runs distributed queries through the Polars API and scales work from local execution to a cluster; it is available on AWS. Its 30-day trial begins when an organization connects its first workspace to AWS, and its compute runs in the customer's cloud environment. Polars On-Prem supports Kubernetes and bare-metal deployments, including air-gapped Enterprise deployments.