Timely Dataflow

Spec sheet

Purpose

Timely Dataflow is a system for implementing distributed streaming computation and a way to structure computation generally.Read 4 Oct 2026 · timelydataflow.github.io

Intended users

The project invites people building dataflow programs and contributors to report their experiences and help improve examples and documentation.Read 9 Oct 2026 · github.com

License

The GitHub repository identifies the project as MIT licensed.Read 9 Oct 2026 · github.com

Built-in operators

The project includes operators such as map, filter, concat, and operators for entering and exiting loops.Read 9 Oct 2026 · github.com

Cluster execution

The README describes running multiple processes using a host file with hostname-and-port entries, plus process-count and process-index arguments.Read 4 Oct 2026 · github.com

Custom operators

Users can supply closures to generic unary and binary operators, and can create source operators for external inputs.Read 9 Oct 2026 · timelydataflow.github.io

Distributed execution

It scales the same program from one thread on a laptop to execution across a cluster of computers.Read 9 Oct 2026 · github.com

Documentation status

The README describes the documentation as a work in progress and cautions that some blog examples may need tweaks to build against current code.Read 9 Oct 2026 · github.com

Ecosystem

Differential Dataflow is described as a higher-level language built on Timely Dataflow, with operators including group, join, and iterate.Read 9 Oct 2026 · github.com

Installation

The README shows adding the timely crate to a Rust project's Cargo.toml dependencies.Read 9 Oct 2026 · github.com

Iteration

The programming model supports higher-level control constructs such as iteration in stream computations.Read 9 Oct 2026 · timelydataflow.github.io

Operators

Built-in operators include map, filter, concat, enter, and leave, with generic unary and binary operators also available.Read 4 Oct 2026 · github.com

Origin

The project site says Timely Dataflow arose from work at Microsoft Research on scalable distributed data processing platforms.Read 4 Oct 2026 · timelydataflow.github.io

Parallelism

Its operators can process independent parts of data concurrently across parallel workers.Read 9 Oct 2026 · timelydataflow.github.io

Programming model

It is described as a low-latency cyclic dataflow computational model implemented in Rust.Read 4 Oct 2026 · github.com

Progress tracking

Operators use timestamp capabilities and input frontiers to track whether more records may arrive at particular times.Read 9 Oct 2026 · timelydataflow.github.io

Scaling

The same program can scale from one thread on a laptop to distributed execution across a cluster of computers.Read 4 Oct 2026 · github.com

Streaming

Its core data type is a stream that can represent unbounded data arriving as a computation proceeds.Read 9 Oct 2026 · timelydataflow.github.io

What it is

Timely Dataflow is a low-latency cyclic dataflow computational model implemented in Rust.Read 9 Oct 2026 · github.com

Where Specifiction places it

• Stream Processing Software#23 of 266.6