Splink vs Treasure AI Identity Resolution

Splink is a free Python package for probabilistic record linkage, used to deduplicate records and connect them when unique identifiers are absent. Its linkage method follows the Fellegi-Sunter model and can be trained without labeled examples. Matching can include term-frequency adjustments and user-defined fuzzy logic. Splink creates SQL for a selected backend, with documented support for DuckDB, Spark, SQLite, and PostgreSQL. DuckDB is recommended for most users; Spark is recommended for very large linkages or when a Spark cluster is more accessible. The maker says it can run on DuckDB or big-data backends such as Spark for linkages of 100+ million records. It works best with standardized data across several columns that are not highly correlated, rather than a single bag-of-words column. Interactive visualizations help examine and diagnose models, predictions, and clusters. Install with pip or conda, with optional backend-specific installation for Spark and PostgreSQL. It runs on Linux, macOS, and Windows, as well as self-hosted environments.

Treasure AI Identity Resolution has no write-up on Specifiction yet; its figures are above.