Amazon Deequ vs pgTAP
Amazon Deequ
github.compgTAP
pgtap.orgSide by side
| Figure | Amazon Deequ | pgTAP |
|---|---|---|
| Price from | Free | — |
| Free plan | Yes | — |
| Free trial | — | — |
| Open source | — | — |
| Windows | Yes | — |
| macOS | Yes | — |
| Linux | Yes | Yes |
| API | Yes | — |
| Self-hosted | Yes | Yes |
| Plans published | 1 | 0 |
| Spec score · Database Testing Tools | 8.7 (#3) | 6.9 (#25) |
| Installation | The README provides Maven and sbt dependency examples and directs users to select a release matching their Spark version. | pgTAP must be installed on a host running PostgreSQL and cannot be installed remotely. |
| Integrations | Deequ is built on Apache Spark, is distributed through Maven artifacts, and has a Python interface called PyDeequ. | The integration guide describes running pgTAP tests with Perl's Test::Harness and TAP::Harness, and with PostgreSQL module testing tools. |
| License | The library is licensed under the Apache 2.0 License. | The PGXN package page grants permission to use, copy, modify, and distribute pgTAP and its documentation without a fee, subject to the stated conditions. |
| Purpose | Deequ is an Apache Spark library for defining unit tests that measure data quality in large datasets. | pgTAP is a PostgreSQL unit testing framework that emits TAP from psql scripts or xUnit style test functions. |
Plans
In their own words
Amazon Deequ
Amazon Deequ is an Apache Spark library for writing data unit tests that measure quality in large datasets, including datasets with billions of rows. Its checks cover row counts, completeness, uniqueness, allowed values, nonnegative values, patterns, and approximate quantiles. Deequ also supports data profiling, stored and queryable metrics, anomaly detection over time, automatic constraint suggestions, and incremental metric computation. Its declarative Data Quality Definition Language includes rules for counts, completeness, uniqueness, statistics, schema matching, freshness, and custom SQL. Row-level evaluation can identify records that pass or fail supported rules, while dataset-level rules such as RowCount and Mean are skipped in that mode. Deequ targets specific Apache Spark versions, so the release must match the Spark version; versions 2.1.0 and later require Java 11. The project provides Maven and sbt dependency examples. It is licensed under Apache 2.0, and Python users can use the PyDeequ interface.