dedupe

Spec sheet

What it does

The dedupe Python library uses machine learning for fuzzy matching, deduplication, and entity resolution on structured data.Read 10 Oct 2026 · github.com

Intended users

The maintainers describe the library as suitable for integration into applications or import scripts, while the former Dedupe.io service was designed as an online tool for non-developers.Read 10 Oct 2026 · docs.dedupe.io

API

The former Dedupe.io web API supported matching and training against projects through REST calls, including incremental checks, matches and additions after deduplication.Read 4 Oct 2026 · dedupe.io

Integrations

The project describes dedupe as a library that can be integrated into applications or import scripts; it also documents CSV deduplication through the separate csvdedupe command-line tool.Read 4 Oct 2026 · docs.dedupe.io

Privacy

The former service’s FAQ says its privacy policy prohibited renting or selling user data and third-party access without signed consent, and limited company access to debugging and service improvement.Read 4 Oct 2026 · dedupe.io

Compliance

The former service's FAQ states that Dedupe.io was certified FERPA compliant as of July 2019.Read 10 Oct 2026 · dedupe.io

Support

The library documentation directs bug reports to GitHub Issues and lists a mailing list for users.Read 10 Oct 2026 · docs.dedupe.io

License

The GitHub repository identifies the project as MIT licensed.Read 10 Oct 2026 · github.com

Headquarters

Chicago, Illinois, United StatesRead 28 Sep 2026 · github.com

Data formats

The former service’s FAQ says uploads accepted Excel spreadsheets and CSV files, with one record per row and column names in the first row.Read 4 Oct 2026 · dedupe.io

Data privacy

The former service's FAQ says it would not rent or sell submitted data and would not allow third-party access without signed consent.Read 10 Oct 2026 · dedupe.io

Features

Documented features include learned weights and blocking rules, laptop execution, library integration, and support for custom data types, comparators, and blocking rules.Read 10 Oct 2026 · docs.dedupe.io

How it learns

It uses human-labeled training data to learn rules for finding similar records, including in large databases.Read 10 Oct 2026 · docs.dedupe.io

Input formats

The former Dedupe.io service accepted spreadsheet files (.xls, .xslx) and comma-separated text (.csv).Read 10 Oct 2026 · dedupe.io

Installation

The documentation gives `pip install dedupe` as the installation command.Read 10 Oct 2026 · docs.dedupe.io

Open source library

dedupe is a Python library that uses machine learning for fuzzy matching, deduplication and entity resolution on structured data.Read 4 Oct 2026 · github.com

Product status

Dedupe.io, the hosted service, shut down on January 31, 2023.Read 10 Oct 2026 · dedupe.io

Self-hosting

The developer page says Python developers can run the library on local machines or servers and build self-hosted entity-resolution pipelines.Read 4 Oct 2026 · dedupe.io

Training

The library uses human-labeled examples to learn rules for finding similar records, including in very large databases.Read 4 Oct 2026 · docs.dedupe.io

Training limit

The former service required at least 10 positive and 10 negative record-pair examples during training.Read 4 Oct 2026 · dedupe.io

Training requirement

The former Dedupe.io service required at least 10 positive and 10 negative record-pair examples during model training.Read 10 Oct 2026 · dedupe.io

Use cases

The library can remove duplicate spreadsheet entries and link customer, order history, or campaign contribution records without relying on unique IDs.Read 10 Oct 2026 · docs.dedupe.io

Web service workflow

The former Dedupe.io service described a workflow to upload data, train a model, review automatically found matches, and download results.Read 4 Oct 2026 · dedupe.io

Where Specifiction places it

• Data Deduplication Software#12 of 277.9