AWS Entity Resolution vs Splink

AWS Entity Resolution links records that may describe the same customer, product, business, or healthcare entity across applications, channels, and data stores. It offers rule-based, machine learning, and data service provider matching. Rule-based users can adjust input fields, priorities, and rules; the preconfigured machine learning model compares consumer fields such as names, email addresses, phone numbers, addresses, and birth dates, then assigns match groups confidence scores from 0.0 to 1.0. Inputs come from AWS Glue, with a limit of 20. Rule-based matching can process new Amazon S3 data incrementally, and the GetMatchId API can synchronously look up an existing match ID while hashing personally identifiable attributes. Access is through the AWS console and API. Personally identifiable data is encrypted at rest by default with AWS-owned keys, with customer-managed symmetric AWS KMS keys also supported. Provider matching includes LiveRamp, TransUnion, and Unified ID 2.0 identifiers, subject to a provider license agreement. AWS states that the service meets PCI DSS and SOC 1, SOC 2, and SOC 3 measures. Pricing is paid, from $0.10; there is no free plan.

Splink is a free Python package for probabilistic record linkage, used to deduplicate records and connect them when unique identifiers are absent. Its linkage method follows the Fellegi-Sunter model and can be trained without labeled examples. Matching can include term-frequency adjustments and user-defined fuzzy logic. Splink creates SQL for a selected backend, with documented support for DuckDB, Spark, SQLite, and PostgreSQL. DuckDB is recommended for most users; Spark is recommended for very large linkages or when a Spark cluster is more accessible. The maker says it can run on DuckDB or big-data backends such as Spark for linkages of 100+ million records. It works best with standardized data across several columns that are not highly correlated, rather than a single bag-of-words column. Interactive visualizations help examine and diagnose models, predictions, and clusters. Install with pip or conda, with optional backend-specific installation for Spark and PostgreSQL. It runs on Linux, macOS, and Windows, as well as self-hosted environments.