A data deduplication API removes duplicate records in milliseconds as each new record enters your system. Batch deduplication processes entire files or tables in a single scheduled run. Use the API when your pipeline requires real-time decisions; use batch when you have a large historical dataset to clean or when latency does not matter. Most production environments ultimately need both.
Ready to see how Match Data Pro handles both modes? Book a 30-minute demo and we will walk through a live deduplication run against your own data schema.
What Is a Data Deduplication API?
A data deduplication API exposes fuzzy matching and duplicate detection as HTTP endpoints. Your application submits a record — a contact, a company, an address — and the API returns a match decision and a confidence score in real time. The record either maps to an existing master entity or is written as a new one.
Match Data Pro’s live fuzzy search and deduplication API accepts a JSON payload, runs multi-algorithm similarity scoring across your master dataset, and returns a structured response — including the top candidate match, its match score, and the field-level breakdown — in under 200 milliseconds for most datasets under 5 million records.
How the API call works
A typical request looks like this:
POST /api/v1/deduplicate
{
"first_name": "Jon",
"last_name": "Smyth",
"email": "jsmith@acme.com",
"company": "Acme Corp"
}The engine applies Jaro-Winkler on name fields, token-based matching on company, and exact matching on email. It returns a ranked candidate list:
| Candidate ID | Name in Master | Score | Decision |
|---|---|---|---|
| CID-10042 | John Smith — jsmith@acme.com | 0.94 | Likely duplicate |
| CID-10887 | Jonathan Smythe — j.smythe@acme.co | 0.71 | Possible — review |
The calling application can auto-merge above 0.90, queue for human review between 0.70 and 0.89, and create a new record below 0.70. These thresholds are configurable per project in Match Data Pro.
What Is Batch Deduplication?
Batch deduplication processes a full file or database table in one run. You ingest the source — CSV, Excel, SQL table, or a connector pull — and the engine compares every record pair (or candidate pair after blocking) to produce a deduplicated master output.
The batch pipeline in Match Data Pro runs AI data profiling first to surface field quality issues: null rates, format inconsistencies, value distributions. Profiling informs which fields are reliable enough to use as blocking keys, which reduces the candidate pair space from O(n²) to a manageable set before fuzzy scoring begins.
The batch deduplication pipeline, step by step
- Ingest. Load the source file or connect directly to a database, CRM, or data warehouse via an import connector.
- Profile. Identify field completeness, dominant formats, and outlier values. A field with 40% null rate is a poor blocking key.
- Cleanse and standardise. Normalise name casing, strip punctuation, parse and standardise addresses. See Match Data Pro’s data cleansing pipeline for the full standardisation process.
- Block. Partition records into candidate groups using high-cardinality keys (postal code + first three letters of surname, for example) so the matching engine only compares plausible pairs.
- Score. Apply multi-algorithm fuzzy matching algorithms — Jaro-Winkler, Levenshtein, phonetic (Soundex, Double Metaphone), and token sort — weighted by field importance.
- Cluster. Group records that score above threshold into duplicate clusters.
- Survivorship. Apply field-level survivorship rules to each cluster: most recent email, longest company name, verified address, most complete phone. The survivorship and golden record merge step produces one authoritative record per cluster.
- Export. Push the clean master dataset to your target system or download via connector.
API Deduplication vs. Batch Deduplication: Side-by-Side Comparison
| Dimension | API Deduplication | Batch Deduplication |
|---|---|---|
| Trigger | Per-record event (form submit, CRM write, webhook) | Scheduled job or manual run |
| Latency | <200 ms per record | Minutes to hours for millions of records |
| Volume per run | 1 to ~1,000 records | Thousands to hundreds of millions |
| Survivorship | Real-time merge or flag | Full cluster-level golden record creation |
| Profiling | Pre-configured field weights | AI-driven profiling before each run |
| Use cases | CRM on-write, form intake, API-first pipelines | Data migration, ERP go-live, periodic master data refresh |
| Human review | Threshold-based auto-flag, async review queue | Review before or after export |
When to Use a Deduplication API
CRM record creation
A sales rep creates a new contact. Before the record is committed, your CRM workflow calls the deduplication API. The API returns “likely duplicate of CID-10042 — score 0.94.” The rep sees a prompt: “This contact may already exist. View existing record?” Duplicate rate drops to near zero without any manual list hygiene work.
Web form intake
A prospect fills in a demo request form. Your intake handler posts the submission to the deduplication API. The API confirms the prospect is already a known contact, looks up their account tier, and routes the request to the correct sales owner — in real time, before the confirmation email fires.
Event-driven pipelines
A Kafka or Pub/Sub consumer reads new customer events from an e-commerce platform. Each event triggers an API call to check for duplicates before writing to the analytics warehouse. This prevents the warehouse from accumulating thousands of near-duplicate rows from users who register with slight name variations across sessions.
Match Data Pro’s REST API for data matching supports both synchronous responses and async callbacks for high-throughput event streams, with configurable timeout and retry logic.
When to Use Batch Deduplication
Historical data cleanup before migration
You are migrating 4 million customer records from a legacy CRM to a new platform. You cannot query those records one at a time — you need the entire dataset profiled, cleansed, deduplicated, and merged before go-live. Batch deduplication with job automation handles this as a single pipeline run, producing a clean master file for import.
Post-merger consolidation
Two companies merge and their customer databases need to be combined into one. Neither organisation has an active API integration. Batch mode ingests both export files, profiles them independently, then runs cross-file matching to identify records that appear in both datasets. See the full workflow in our guide to post-merger customer deduplication.
Periodic master data refresh
Marketing operations runs a monthly batch job to deduplicate the CRM before campaign sends. The job pulls the full contact table, identifies 2,400 duplicate clusters across 1.1 million contacts, merges them using survivorship rules, and pushes a clean list back to the CRM. Match Data Pro’s merge purge job automation schedules these runs with zero manual intervention.
Multi-source integration
You pull customer data from a CRM, a billing system, and a marketing automation platform. All three use different ID schemes and different name formats. Batch deduplication ingests all three, standardises field formats, and uses Senzing entity resolution to resolve the same customer across all three sources — producing one unified golden record per entity regardless of the source system’s naming conventions.
Running Both Modes Together
Most mature data pipelines run batch and API deduplication together, not as alternatives. The pattern is:
- Initial batch run. Clean the existing dataset to establish a deduplicated master. This removes the historical debt before the API goes live.
- API on new records. Every new record created after go-live is checked against the clean master in real time. Duplicates never enter the master in the first place.
- Periodic batch refresh. A scheduled batch job — weekly or monthly — reprocesses the master to catch any duplicates that slipped through low-confidence API decisions, or records that became duplicates after a field update (a user who changed their email to match an existing contact’s).
This hybrid model reduces the ongoing cost of data maintenance. The API prevents new duplicates. The scheduled batch catches edge cases. Together they keep duplicate rates below 0.5% in production CRM and MDM environments.
Match Data Pro supports both modes on the same platform — the same fuzzy matching engine, the same configurable thresholds, the same survivorship rules — so you never have to reconcile two separate tool outputs. Start with a free trial to configure both modes against your own data before committing to any contract.
Choosing the Right Threshold for Each Mode
Thresholds behave differently in API mode versus batch mode. In the API, a false positive (flagging a non-duplicate as a duplicate) interrupts a user’s workflow — the sales rep gets a wrong “already exists” warning. So API thresholds tend to be set conservatively: auto-merge at 0.95+, human review at 0.80 to 0.94, create new below 0.80.
In batch mode, false negatives (missing a duplicate) compound over time in the master dataset. Teams tend to set lower auto-merge thresholds and rely on the review queue to catch borderline clusters. A 0.75 threshold in batch mode with a review queue produces a cleaner master than a 0.90 threshold with no review. For detailed guidance on tuning, see our article on deterministic vs. probabilistic matching thresholds.
Frequently Asked Questions
What is a data deduplication API?
A data deduplication API is an HTTP endpoint that accepts a single record, runs fuzzy similarity scoring against a master dataset, and returns a match decision and confidence score in real time — typically under 200 milliseconds. It allows any application to check for duplicates before writing a new record to the database, preventing duplicates at the point of entry rather than cleaning them up later.
When should I use batch deduplication instead of an API?
Use batch deduplication when you have a large existing dataset to clean — a migration, a post-merger consolidation, or a periodic master data refresh. Batch mode processes millions of records in a single run with AI profiling, blocking, multi-algorithm fuzzy scoring, and survivorship rules. It is the right choice when record volume, not latency, is the primary constraint.
Can I run API and batch deduplication on the same platform?
Yes. Match Data Pro runs both modes on the same matching engine with shared configuration — same field weights, same thresholds, same survivorship rules. The recommended pattern is a one-time batch run to clean the existing dataset, then the API on all new records, with a periodic batch refresh to catch edge cases that slipped through low-confidence API decisions.
How fast is a deduplication API call in practice?
For most datasets under 5 million master records, a single deduplication API call returns a ranked match list in under 200 milliseconds. Blocking and pre-indexed candidate sets keep query time low. For very large master datasets — tens of millions of records — response times typically remain under 500 milliseconds with proper index configuration and blocking key design.
What fuzzy matching algorithms does a deduplication API use?
A well-designed deduplication API applies multiple algorithms in a weighted pipeline: Jaro-Winkler for short name strings (handles transpositions and prefixes well), Levenshtein edit distance for general character-level comparison, phonetic algorithms like Double Metaphone for names that sound alike but are spelled differently (e.g. “Smith” vs “Smythe”), and token-based methods for company names or addresses with variable word order.