Flakiness.io vs Mergify
Flakiness.io
flakiness.io
Mergify
mergify.comSide by side
| Figure | Flakiness.io | Mergify |
|---|---|---|
| Price from | $49/mo | $21/mo |
| Free plan | — | Yes |
| Free trial | — | Yes |
| Open source | — | — |
| Web | Yes | Yes |
| API | — | Yes |
| Self-hosted | Yes | Yes |
| Plans published | 4 | 3 |
| Spec score · CI Failure Analysis And Flaky Test Tools | 7.4 (#8) | 7.9 (#7) |
| Spec score · Test Flakiness Management Software | 7.6 (#18) | 7.9 (#12) |
| Spec score · Test Analytics Software | 7.6 (#18) | 8.2 (#12) |
| Purpose | Flakiness.io provides test analytics for GitHub and GitLab and tracks test performance across commits to detect regressions. | Mergify automates the path from pull request to green main with merge queues, CI insights, flaky-test tools, and merge protections. |
| Security | The privacy policy lists encryption in transit, Cloudflare network protection, access controls, environment isolation, and monitoring; customers are responsible for securing self-hosted deployments and credentials. | The pricing comparison lists data encryption, SSO, and delegated roles, and identifies SOC 2 Type 2 for Max and Enterprise. |
| Support | The pricing page lists standard support for Free and Startup plans and priority support for Growth and Enterprise plans. | Plans list email support and Slack Connect support, while Enterprise can include optional 24/7 premium support. |
Plans
Flakiness.io
- Startup$49 a month
- Growth$199 a month
- EnterprisePriced on request
- FreePrice not published
In their own words
Flakiness.io
Flakiness.io provides test analytics for GitHub and GitLab, tracking outcomes across commits to distinguish flaky tests from regressions. It labels a test that alternates between passing and failing on the same commit as flaky, while persistent outcome changes at a commit count as regressions. Listed runners include Playwright, Pytest, Vitest, Jest, CucumberJS, Bun, Rust, and a JUnit XML bridge; a Node.js SDK supports custom integrations. The service works with any CI provider, ingests results as they arrive, merges shards, and separates staging and production histories. Reports include test history, grouped errors, system telemetry, and a test waterfall. Artifacts can include logs, screenshots, videos, traces, and HTML files. Recorded durations can help Playwright balance shards using historical runtime. Hosted plans include a Free option with 1 GB of artifact storage and 90 days of retention; Startup costs $49 a month with 10 GB and 180 days. Growth costs $199 a month. Self-hosting uses PostgreSQL and an S3-compatible object store. Customers must not upload specified sensitive data.
Mergify
Mergify automates the path from pull request to a green main branch with merge queues, CI insights, flaky-test tools, and merge protections. Its queue rebases each pull request onto the latest main and merges only when CI passes against the combined state. Teams can combine pull requests into a single CI run; if that batch fails, Mergify can bisect it to locate the culprit. CI Insights reports unreliable jobs, retries them automatically, and provides data for investigating causes.
Merge Protections supports custom rules, scheduled merge windows, freeze periods, and exceptions for urgent fixes. The queue works with any CI that reports status checks to GitHub, while deeper CI Insights integrations are listed for selected providers. Integrations include GitHub, GitHub Enterprise, GitHub Actions, Jenkins, Buildkite, Slack, Datadog, and Terraform, among others. Mergify offers a free plan for open-source projects or private teams of up to five users. Max costs $21 per user per month and includes a 14-day free trial. Enterprise pricing is custom; it can include optional on-premise deployment and dedicated support. Mergify was founded in 2018 and is headquartered in Toulouse, France.