Entify vs Splink

Entify is a free, open-source workspace for linking records, resolving entities, and deduplicating datasets. It uses Splink 4 and DuckDB for matching, with exact, Jaro-Winkler, Jaccard, and Levenshtein comparisons. Automatic setup estimates column types, proposes blocking rules, and explains its choices for users to adjust. It reads CSV, TSV, Parquet, JSON, and Excel files, and reports row counts, distinct values, and completeness by column. Review tools include match evidence, score and threshold analysis, and PDF audit reports. Merge exports keep one surviving record per entity and retain source IDs for tracing or undoing merges. Entify runs locally with Docker or with Python 3.11+ and Node 20+; demo mode works without optional sign-in or shared storage. Its Docker quickstart needs no accounts, API keys, or external services, and uploaded data stays on the machine. The project is intended for data stewards and teams without specialist record-linkage expertise. Optional semantic blocking requires an extra install of roughly 600 MB of Torch dependencies.

Splink is a free Python package for probabilistic record linkage: it deduplicates and connects records without unique identifiers. Its core algorithm is based on the Fellegi–Sunter model, and it can train models without training data using an unsupervised approach. Matching supports term-frequency adjustments and user-defined fuzzy logic. Interactive outputs help users inspect models and diagnose linkage problems. Splink documents DuckDB, Spark, PostgreSQL, and SQLite backends; DuckDB is installed by default, while Spark and PostgreSQL have optional installs. The project says it can run with big-data backends such as Spark on datasets of 100 million or more records, and can link a million records on a laptop in around a minute. It works best with standardised data containing multiple columns that are not highly correlated. Installation is through pip or conda and requires Python 3.10 or later. The project lists Linux and self-hosted use and an MIT license.