Buildkite vs Flakiness.io

Buildkite Pipelines is a CI/CD platform that dispatches pipeline steps as jobs to agents. Agents can be hosted by Buildkite or run in a customer's own infrastructure, including on-premises and cloud environments. Dynamic pipelines can generate and upload steps during a build, allowing a workflow to respond to code changes, test results or runtime logic. Test Engine collects results across testing frameworks and provides tools to track, analyze and act on them. It can detect and quarantine flaky tests, split tests across parallel jobs, and monitor test performance, reliability and ownership. Documented integrations include Slack, PagerDuty, Datadog, Honeycomb, OpenTelemetry, Amazon EventBridge, Artifactory, Docker Hub and Backstage. Buildkite lists TLS 1.2+ for data in transit and AES-256 encryption at rest, along with an annually audited SOC 2 Type II report. The Free Plan costs nothing and allows up to 5 users and 10 concurrent jobs. Pro costs $30 a month, billed per active user monthly, and includes up to 50 users and up to 250 concurrent agents, with 10 included.

Flakiness.io provides test analytics for GitHub and GitLab, tracking outcomes across commits to distinguish flaky tests from regressions. It labels a test that alternates between passing and failing on the same commit as flaky, while persistent outcome changes at a commit count as regressions. Listed runners include Playwright, Pytest, Vitest, Jest, CucumberJS, Bun, Rust, and a JUnit XML bridge; a Node.js SDK supports custom integrations. The service works with any CI provider, ingests results as they arrive, merges shards, and separates staging and production histories. Reports include test history, grouped errors, system telemetry, and a test waterfall. Artifacts can include logs, screenshots, videos, traces, and HTML files. Recorded durations can help Playwright balance shards using historical runtime. Hosted plans include a Free option with 1 GB of artifact storage and 90 days of retention; Startup costs $49 a month with 10 GB and 180 days. Growth costs $199 a month. Self-hosting uses PostgreSQL and an S3-compatible object store. Customers must not upload specified sensitive data.