dagster-slurm

Spec sheet

Purpose

dagster-slurm lets Dagster assets run locally during development and on a Slurm cluster in production.Read 5 Oct 2026 · dagster-slurm.geoheil.com

Intended users

The docs describe laptop-first development and cluster use for compute-heavy Dagster assets, including model training and hyper-parameter sweeps.Read 5 Oct 2026 · dagster-slurm.geoheil.com

Support

The project directs users to GitHub Discussions for questions and how-to requests, GitHub Issues for bugs and feature requests, and Dagster Slack for broader ecosystem questions.Read 5 Oct 2026 · dagster-slurm.geoheil.com

Cluster connectivity

The product connects to Slurm clusters over SSH and supports key-based authentication, jump hosts, and configurable host-key checking.Read 5 Oct 2026 · dagster-slurm.geoheil.com

Environment packaging

Jobs use Pixi environments, and production mode expects a pre-deployed environment bundle on the cluster.Read 5 Oct 2026 · dagster-slurm.geoheil.com

Execution modes

The stable execution modes are local and Slurm, with one Slurm job per asset by default.Read 5 Oct 2026 · dagster-slurm.geoheil.com

Experimental features

Heterogeneous Slurm jobs are experimental; run-scoped allocation reuse is currently focused on Ray assets.Read 5 Oct 2026 · dagster-slurm.geoheil.com

Job submission

It transfers the current checkout payload and Pixi-locked environment, submits jobs with Slurm, and returns job status and logs to Dagster.Read 5 Oct 2026 · dagster-slurm.geoheil.com

Local development

Assets can run locally without SSH or Slurm, and the project provides a Docker Compose Slurm stack for testing cluster workflows.Read 5 Oct 2026 · dagster-slurm.geoheil.com

Observability

Dagster can display Slurm state, logs, CPU efficiency, memory, elapsed time, and node-hours for asset runs.Read 5 Oct 2026 · dagster-slurm.geoheil.com

Pipeline orchestration

Dagster asset dependencies and lineage can connect data ingestion, Slurm compute, and downstream aggregation or publishing.Read 5 Oct 2026 · dagster-slurm.geoheil.com

Ray integration

Ray is a first-class launcher for distributed execution, GPU scheduling, and optional run-scoped allocation reuse.Read 5 Oct 2026 · dagster-slurm.geoheil.com

Secrets handling

Cluster connection details can be provided through runtime environment variables so credentials can be kept out of the repository.Read 5 Oct 2026 · dagster-slurm.geoheil.com

Spark integration

The documentation labels the Spark integration as work in progress.Read 5 Oct 2026 · dagster-slurm.geoheil.com

Plans and prices

Every dagster-slurm plan and price →

Where Specifiction places it

• Workflow Orchestration Software#20 of 387.9