Apache Airflow vs DataKitchen TestGen

Apache Airflow is a platform for authoring, scheduling, and monitoring workflows written in Python. It is designed for batch workflows with a defined beginning and end that run on a schedule. Its web application lets users manage and schedule workflows, trigger DAGs manually, inspect task status and logs, backfill runs, and rerun failed tasks. A modular architecture uses a message queue to coordinate workers and can orchestrate an arbitrary number of them. Built-in operators connect workflows to Google Cloud Platform, Amazon Web Services, Microsoft Azure, and other services. The community maintains more than 80 separately installable provider packages, which can be upgraded or downgraded independently of Airflow core. Airflow can read connections, variables, and configuration from secret backends, but those integrations do not support writes. JWT authentication is used for its public REST API and internal Execution API. From Airflow 3 onward, sensitive credentials are masked in logs, the interface, and API output. Airflow is free to download and use under the Apache License.

DataKitchen TestGen profiles databases, generates data quality tests, and shows results and quality scores in dashboards. Its listed checks include 55 profiling characteristics, 32 hygiene detector tests, and 49 test types, including generated tests, business rules, and custom SQL. Monitors cover freshness, volume, schema, and custom metrics, using predictive models to flag changes such as late arrivals and schema drift. Supported databases include Snowflake, Databricks SQL, Azure Synapse Analytics, Azure SQL Database, SQL Server, OneLake, BigQuery, Redshift, Aurora PostgreSQL, Oracle 12c and later, SAP HANA, PostgreSQL, and Salesforce Data 360. It can also profile and test structured data in Iceberg, Parquet, Avro, ORC, CSV, and JSON through supported external-table options. TestGen runs in the customer’s infrastructure; DataKitchen says it does not host the application or receive customer data. It queries target databases with read-only access and stores results and metadata in its application database. The free Open Source plan is limited to one user, one connection, and one project. Enterprise costs $100 a month, billed per user and per database connection.