Azure Test Plans vs CloudBees CI
Azure Test Plans
azure.microsoft.com
CloudBees CI
cloudbees.comSide by side
| Figure | Azure Test Plans | CloudBees CI |
|---|---|---|
| Price from | $6/mo | — |
| Free plan | Yes | — |
| Free trial | Yes | Yes |
| Open source | — | — |
| Web | Yes | Yes |
| Windows | Yes | Yes |
| macOS | Yes | Yes |
| Linux | Yes | Yes |
| API | Yes | Yes |
| Self-hosted | Yes | Yes |
| Plans published | 5 | 1 |
| Spec score · Continuous Integration Software | 9.4 (#3) | 8.7 (#6) |
| Company legal name | The maker is Microsoft Corporation. | CloudBees documentation identifies the maker as CloudBees, Inc. |
| Founded | Microsoft was founded in 1975. | CloudBees was founded in 2010 by former JBoss CTO Sacha Labourey and colleagues. |
| Integrations | Azure DevOps exposes integrations with GitHub, Docker, Slack, Sentry and more through its Marketplace extension ecosystem. | Official integration guides cover GitHub, Bitbucket, Slack and Microsoft Teams for source reporting, notifications and collaboration. |
| Purpose | Azure Test Plans provides planned manual and exploratory testing tools for apps. | CloudBees CI is a centrally managed, self-service Jenkins environment for delivering continuous integration at scale. |
| Rating on Capterra | 4.4 out of 5 from 148 reviews | 4.0 out of 5 from 1 review |
| Rating on Gartner Peer Insights | 4.3 out of 5 from 197 ratings | 4.5 out of 5 from 72 ratings |
| Self-hosting | The same test-management documentation covers Azure DevOps Services and Azure DevOps Server, including Server 2022. | Customers provision and operate the infrastructure for CloudBees CI, including Kubernetes clusters for the modern-cloud edition. |
| What it does | Azure Test Plans lets teams plan, execute and track manual, exploratory and automated testing within Azure DevOps. | CloudBees CI is a centrally managed Jenkins-based continuous integration and delivery system deployable on premises or in public clouds. |
Plans
Azure Test Plans
- Basic + Test Plans$52 a month
- BasicFree
- StakeholdersFree
- Stakeholders (Free)Free
- Basic$6 a month
In their own words
Azure Test Plans
Azure Test Plans gives teams tools to plan, run, and track manual and exploratory testing for apps. For scripted tests, teams can track actionable defects and connect test cases and suites to user stories, features, or requirements. Exploratory sessions let testers design and run tests at the same time. During manual tests, Test Runner can capture screenshots, image action logs, and screen recordings. Reports, dashboard widgets, and customizable Power BI reports show plan progress and trends. The service integrates with Azure Pipelines for CI/CD testing and offers REST APIs and an extension contribution model. Manual testing is available in the web portal through supported browsers on major platforms; a free Test & Feedback extension supports exploratory testing in Chrome and Microsoft Edge. The free Stakeholders plan does not allow users to create or manage Test Plans. Basic is free for the first five users, then $6 per user per month; Basic + Test Plans costs $52 a month, billed at $52 per user per month. A 30-day trial is listed. Azure DevOps cloud-based load testing is deprecated; Microsoft points users to Azure Load Testing for managed load testing.
CloudBees CI
CloudBees CI is a centrally managed Jenkins environment for teams delivering continuous integration at scale. It can run on-premise or in the public cloud, with a modern cloud platform designed for Kubernetes and a traditional platform edition. Operations Center provides a central view across CI controllers. Pipeline capabilities include complex delivery pipelines, declarative syntax, and builds that can survive controller and agent failures. The CloudBees Assurance Program vets plugins, while Beekeeper helps administrators monitor and manage plugin upgrades. Documented integrations include GitHub and Bitbucket, plus Slack and Microsoft Teams. Security options include central management, role-based access control, single sign-on, and connections to external credential stores such as CyberArk and HashiCorp Vault. CloudBees CI is paid, with pricing available on request, and has no free plan. 24/7 expert support depends on the customer’s support agreement. CloudBees support policies cover only CloudBees-certified plugins on services based on Jenkins core.