Skip to content
SPECIFICTION
Get listed

Buildkite vs FlakyTest.dev

Buildkite

buildkite.com

FlakyTest.dev

flakytest.dev

Side by side

FigureBuildkiteFlakyTest.dev
Price from$30/moFree
Free planYesYes
Free trialYesYes
Open source——
WebYesYes
WindowsYes—
macOSYes—
LinuxYes—
APIYes—
Self-hostedYes—
Plans published32
Spec score · Test Flakiness Management Software8.7 (#7)7.6 (#19)
IntegrationsDocumented integrations include Slack, PagerDuty, Datadog, Honeycomb, OpenTelemetry, Amazon EventBridge, Artifactory, Docker Hub, and Backstage.The setup guide documents GitHub Actions configuration using a repository secret named FLAKYTEST_SECRET_TOKEN.
PurposeBuildkite Pipelines is a platform for building CI/CD workflows, with pipeline steps dispatched as jobs to agents.FlakyTest is a place to manage, monitor, and report on test suite performance.
SupportThe Free Plan includes community support, Pro includes priority email support, and Enterprise includes support with SLA credits and Slack Connect.The documentation directs users needing help, support, or feedback to email the maker.

Plans

Buildkite

  • Free PlanFree
  • Pro Plan$30 a month
  • Enterprise PlanPriced on request

Buildkite pricing →

FlakyTest.dev

  • OSSFree
  • BusinessPriced on request

FlakyTest.dev pricing →

In their own words

Buildkite

Buildkite Pipelines is a CI/CD platform that dispatches pipeline steps as jobs to agents. Agents can be hosted by Buildkite or run in a customer's own infrastructure, including on-premises and cloud environments. Dynamic pipelines can generate and upload steps during a build, allowing a workflow to respond to code changes, test results or runtime logic. Test Engine collects results across testing frameworks and provides tools to track, analyze and act on them. It can detect and quarantine flaky tests, split tests across parallel jobs, and monitor test performance, reliability and ownership. Documented integrations include Slack, PagerDuty, Datadog, Honeycomb, OpenTelemetry, Amazon EventBridge, Artifactory, Docker Hub and Backstage. Buildkite lists TLS 1.2+ for data in transit and AES-256 encryption at rest, along with an annually audited SOC 2 Type II report. The Free Plan costs nothing and allows up to 5 users and 10 concurrent jobs. Pro costs $30 a month, billed per active user monthly, and includes up to 50 users and up to 250 concurrent agents, with 10 included.

Visit buildkite.com

FlakyTest.dev

FlakyTest.dev is a web service for managing, monitoring, and reporting on test suite performance, aimed at Python teams handling unreliable test suites. Its pytest plugin tracks tests and sends results when the FLAKYTEST_SECRET_TOKEN environment variable is set. Test ownership routes flaky-test notifications to the person who wrote the test. In the interface, users can mute a test without changing code or opening a pull request; the test keeps running and logging errors but does not fail the CI build. Each test has a run history, and failing tests show standard output, standard error, and other debug information. Run statistics help identify tests that slow builds. Flake detection and a quarantine workflow support unreliable-test management. The setup guide documents GitHub Actions configuration with a repository secret. The OSS plan is free for personal projects, non profits, and open source projects. Business includes a 60-day free trial, with later pricing undetermined.

Visit flakytest.dev