CircleCI vs FlakyTest.dev
CircleCI
circleci.comFlakyTest.dev
flakytest.devSide by side
| Figure | CircleCI | FlakyTest.dev |
|---|---|---|
| Price from | $15/mo | Free |
| Free plan | — | Yes |
| Free trial | — | Yes |
| Open source | — | — |
| Web | Yes | Yes |
| Windows | Yes | — |
| macOS | Yes | — |
| Linux | Yes | — |
| API | Yes | — |
| Self-hosted | Yes | — |
| Plans published | 3 | 2 |
| Spec score · Test Flakiness Management Software | 8.1 (#9) | 7.6 (#19) |
| Headquarters | San Francisco, California, United States | Ljubljana, Slovenia |
| Support | Community support is included across plans; ticket-based global support and 8x5 support SLAs are available on Performance and Scale, with 24x7 SLAs available on Scale. | The documentation directs users needing help, support, or feedback to email the maker. |
Plans
In their own words
CircleCI
CircleCI provides CI/CD tooling for engineering teams to ship code and validate changes. Smarter Testing includes test impact analysis, dynamic test splitting, and automatic reruns of failed tests. Execution options include Docker, Linux, Arm, Windows, macOS, and self-hosted runners. CircleCI Cloud supports repositories hosted on GitHub, GitLab, GitLab self-managed, Bitbucket Cloud, and Cursor Origin, which is in beta. Teams can use Orbs, shareable configuration packages, and private orbs are available across listed plans. CircleCI Deploys lets teams monitor incremental releases and roll back automatically. Security controls include encrypted source-code communication, restricted contexts, audit logging, isolated build sandboxes, and OpenID Connect support. CircleCI says it is SOC 2 Type II compliant and has a FedRAMP Tailored designation. Its API supports programmatic interaction with pipelines, projects, and organizations. A free plan is listed, and Performance starts at $15 a month. Free-plan credits expire after a month without rollover; paid credits roll over and expire after a year.
FlakyTest.dev
FlakyTest.dev is a web service for managing, monitoring, and reporting on test suite performance, aimed at Python teams handling unreliable test suites. Its pytest plugin tracks tests and sends results when the FLAKYTEST_SECRET_TOKEN environment variable is set. Test ownership routes flaky-test notifications to the person who wrote the test. In the interface, users can mute a test without changing code or opening a pull request; the test keeps running and logging errors but does not fail the CI build. Each test has a run history, and failing tests show standard output, standard error, and other debug information. Run statistics help identify tests that slow builds. Flake detection and a quarantine workflow support unreliable-test management. The setup guide documents GitHub Actions configuration with a repository secret. The OSS plan is free for personal projects, non profits, and open source projects. Business includes a 60-day free trial, with later pricing undetermined.