Amazon Deequ vs pgroll

Amazon Deequ is an Apache Spark library for writing data unit tests that measure quality in large datasets, including datasets with billions of rows. Its checks cover row counts, completeness, uniqueness, allowed values, nonnegative values, patterns, and approximate quantiles. Deequ also supports data profiling, stored and queryable metrics, anomaly detection over time, automatic constraint suggestions, and incremental metric computation. Its declarative Data Quality Definition Language includes rules for counts, completeness, uniqueness, statistics, schema matching, freshness, and custom SQL. Row-level evaluation can identify records that pass or fail supported rules, while dataset-level rules such as RowCount and Mean are skipped in that mode. Deequ targets specific Apache Spark versions, so the release must match the Spark version; versions 2.1.0 and later require Java 11. The project provides Maven and sbt dependency examples. It is licensed under Apache 2.0, and Python users can use the PyDeequ interface.

pgroll is an open-source command-line tool for PostgreSQL schema migrations that keeps old and new schema versions available while applications move between them. Its expand/contract workflow lets clients choose a version through PostgreSQL's search_path. The project describes migrations as zero-downtime and says it applies changes without locking the database; it can also backfill columns automatically when needed. For breaking changes, pgroll builds virtual schemas from views over physical tables and uses triggers to carry writes between old and new columns. Active migrations can be rolled back to the previous schema version. It supports PostgreSQL 14 and later, including RDS and Aurora, and is available for Linux, macOS, and Windows or self-hosting. It can convert SQL generated by ORMs, with examples for Alembic, Django, and Drizzle, though manual data migrations and SQL consolidation may be needed. On PostgreSQL 14, its versioned views do not respect row-level security policies, which can make it a poor choice for RLS users on that version. The project is licensed under Apache License 2.0.