pyJedAI vs Splink

pyJedAI is a free, open-source Python framework for creating entity-resolution workflows. It addresses record linkage, entity resolution and link discovery across structured and semi-structured data. Its collection of algorithms includes supervised and unsupervised machine-learning methods. The project says its blocking and matching methods can apply deep learning and NLP without requiring users to provide labeled instances. Installation is available through pip, Git checkout or Docker. The README says pyJedAI has been tested on Windows and Linux and requires Python 3.8 or later; Windows also requires Microsoft Visual C++ 14.0. It uses packages including NLTK, pandas, transformers, sentence-transformers and FAISS CPU. pyJedAI is licensed under Apache-2.0. The project describes it as suitable for both expert and novice users, and its package metadata also identifies developer and science/research audiences. GitHub Discussions are recommended for general questions, while bugs can be reported through GitHub Issues. The README points Java users to JedAI, which includes a web app for building workflows interactively.

Splink is a free Python package for probabilistic record linkage, used to deduplicate records and connect them when unique identifiers are absent. Its linkage method follows the Fellegi-Sunter model and can be trained without labeled examples. Matching can include term-frequency adjustments and user-defined fuzzy logic. Splink creates SQL for a selected backend, with documented support for DuckDB, Spark, SQLite, and PostgreSQL. DuckDB is recommended for most users; Spark is recommended for very large linkages or when a Spark cluster is more accessible. The maker says it can run on DuckDB or big-data backends such as Spark for linkages of 100+ million records. It works best with standardized data across several columns that are not highly correlated, rather than a single bag-of-words column. Interactive visualizations help examine and diagnose models, predictions, and clusters. Install with pip or conda, with optional backend-specific installation for Spark and PostgreSQL. It runs on Linux, macOS, and Windows, as well as self-hosted environments.