MutPy vs pytest-gremlins
MutPy
github.compytest-gremlins
github.comSide by side
| Figure | MutPy | pytest-gremlins |
|---|---|---|
| Price from | Free | Free |
| Free plan | Yes | Yes |
| Free trial | — | — |
| Open source | — | — |
| Linux | Yes | Yes |
| Self-hosted | Yes | — |
| Plans published | 1 | 1 |
| Spec score · Mutation Testing Tools | 7.9 (#7) | 7.7 (#10) |
| Installation | The project README gives pip installation and source installation instructions. | The project documents installation with pip, uv, and Poetry. |
| Integrations | The documented test runner integrations are unittest and experimental pytest; no other integrations are listed. | The documentation describes compatibility with pytest-xdist and a GitHub Action for running mutation tests and enforcing a score threshold. |
| License | The project is licensed under the Apache License, Version 2.0. | The project is MIT licensed. |
| Purpose | MutPy is a mutation testing tool for Python source code that evaluates the quality of software tests. | pytest-gremlins is a pytest plugin that injects small code mutations and checks whether tests catch them. |
| Reports | MutPy can generate YAML and HTML reports and colorful console output. | It supports console, HTML, and JSON reports, including multiple report formats in one run. |
Plans
In their own words
MutPy
MutPy is a mutation testing tool for Python that checks whether a test suite catches changes to its source code. It creates mutations at the abstract syntax tree level, then runs tests to reveal which changes go undetected. The project describes compatibility with Python 3.3 and later. It uses unittest by default, while pytest support is experimental. Reports can be generated in YAML or HTML, and coverage analysis can restrict mutation to code exercised by tests. Users can also select or disable operators and sample a percentage of mutants. MutPy supports higher order mutations and includes standard and experimental operator sets. Installation is documented through pip from PyPI or from the repository source. The software is free at free and licensed under Apache License 2.0. PyPI lists version 0.6.1, released November 17, 2019, as the latest release. The README warns that disabling stdout can damage tests that interact with sys.stdout.
pytest-gremlins
pytest-gremlins is a pytest plugin for checking whether an existing test suite catches small changes to code. It runs mutation tests against Python projects and can report results in the console, HTML, or JSON, including multiple formats in one run. Getting started requires Python 3.11 or later, pytest 7.0 or later, and existing tests; installation is documented with pip, uv, and Poetry. The project’s quick-start command is `pytest --gremlins`. To manage execution, it uses mutation switching, coverage-guided test selection, incremental analysis, and parallel execution, and it works with pytest-xdist. Options can be set in `pyproject.toml` or with command-line flags, while inline pragmas can pardon specific mutations. A GitHub Action can run the checks and fail a workflow if its score misses a threshold; CI examples also cover GitLab CI, CircleCI, and Dagger. It is a free, MIT-licensed package for Linux, classified as Alpha. The guide warns that an initial run may take 10–100 times as long as a normal test run, though caching can shorten later runs. It also notes that equivalent mutations can make a perfect score unattainable and unnecessary.