AWS Entity Resolution vs Splink

AWS Entity Resolution links records that may describe the same customer, product, business, or healthcare entity across applications, channels, and data stores. It offers rule-based, machine learning, and data service provider matching. Rule-based users can adjust input fields, priorities, and rules; the preconfigured machine learning model compares consumer fields such as names, email addresses, phone numbers, addresses, and birth dates, then assigns match groups confidence scores from 0.0 to 1.0. Inputs come from AWS Glue, with a limit of 20. Rule-based matching can process new Amazon S3 data incrementally, and the GetMatchId API can synchronously look up an existing match ID while hashing personally identifiable attributes. Access is through the AWS console and API. Personally identifiable data is encrypted at rest by default with AWS-owned keys, with customer-managed symmetric AWS KMS keys also supported. Provider matching includes LiveRamp, TransUnion, and Unified ID 2.0 identifiers, subject to a provider license agreement. AWS states that the service meets PCI DSS and SOC 1, SOC 2, and SOC 3 measures. Pricing is paid, from $0.10; there is no free plan.

Splink is a free Python package for probabilistic record linkage: it deduplicates and connects records without unique identifiers. Its core algorithm is based on the Fellegi–Sunter model, and it can train models without training data using an unsupervised approach. Matching supports term-frequency adjustments and user-defined fuzzy logic. Interactive outputs help users inspect models and diagnose linkage problems. Splink documents DuckDB, Spark, PostgreSQL, and SQLite backends; DuckDB is installed by default, while Spark and PostgreSQL have optional installs. The project says it can run with big-data backends such as Spark on datasets of 100 million or more records, and can link a million records on a laptop in around a minute. It works best with standardised data containing multiple columns that are not highly correlated. Installation is through pip or conda and requires Python 3.10 or later. The project lists Linux and self-hosted use and an MIT license.