FlakyTest.dev vs Mergify
FlakyTest.dev
flakytest.devMergify
mergify.comSide by side
| Figure | FlakyTest.dev | Mergify |
|---|---|---|
| Price from | Free | $21/mo |
| Free plan | Yes | Yes |
| Free trial | Yes | Yes |
| Open source | — | — |
| Web | Yes | Yes |
| API | — | Yes |
| Self-hosted | — | Yes |
| Plans published | 2 | 3 |
| Spec score · Test Flakiness Management Software | 7.6 (#19) | 7.9 (#12) |
| Headquarters | Ljubljana, Slovenia | Toulouse, France |
| Integrations | The setup guide documents GitHub Actions configuration using a repository secret named FLAKYTEST_SECRET_TOKEN. | The integrations page lists GitHub, GitHub Enterprise, GitHub Actions, Jenkins, Buildkite, Slack, Datadog, Terraform, and multiple test frameworks. |
| Intended users | The maker describes FlakyTest as a tool for Python teams dealing with unreliable test suites. | Mergify describes its users as engineering and platform teams handling shared branches, monorepos, and long or costly CI pipelines. |
| Purpose | FlakyTest is a place to manage, monitor, and report on test suite performance. | Mergify automates the path from pull request to green main with merge queues, CI insights, flaky-test tools, and merge protections. |
| Support | The documentation directs users needing help, support, or feedback to email the maker. | Plans list email support and Slack Connect support, while Enterprise can include optional 24/7 premium support. |
Plans
In their own words
FlakyTest.dev
FlakyTest.dev is a web service for managing, monitoring, and reporting on test suite performance, aimed at Python teams handling unreliable test suites. Its pytest plugin tracks tests and sends results when the FLAKYTEST_SECRET_TOKEN environment variable is set. Test ownership routes flaky-test notifications to the person who wrote the test. In the interface, users can mute a test without changing code or opening a pull request; the test keeps running and logging errors but does not fail the CI build. Each test has a run history, and failing tests show standard output, standard error, and other debug information. Run statistics help identify tests that slow builds. Flake detection and a quarantine workflow support unreliable-test management. The setup guide documents GitHub Actions configuration with a repository secret. The OSS plan is free for personal projects, non profits, and open source projects. Business includes a 60-day free trial, with later pricing undetermined.
Mergify
Mergify automates the path from pull request to a green main branch with merge queues, CI insights, flaky-test tools, and merge protections. Its queue rebases each pull request onto the latest main and merges only when CI passes against the combined state. Teams can combine pull requests into a single CI run; if that batch fails, Mergify can bisect it to locate the culprit. CI Insights reports unreliable jobs, retries them automatically, and provides data for investigating causes.
Merge Protections supports custom rules, scheduled merge windows, freeze periods, and exceptions for urgent fixes. The queue works with any CI that reports status checks to GitHub, while deeper CI Insights integrations are listed for selected providers. Integrations include GitHub, GitHub Enterprise, GitHub Actions, Jenkins, Buildkite, Slack, Datadog, and Terraform, among others. Mergify offers a free plan for open-source projects or private teams of up to five users. Max costs $21 per user per month and includes a 14-day free trial. Enterprise pricing is custom; it can include optional on-premise deployment and dedicated support. Mergify was founded in 2018 and is headquartered in Toulouse, France.