Entity resolution and data linkage are how modern data teams connect records that describe the same real-world entity across two or more databases — without a shared primary key. The process uses fuzzy scoring, blocking, and graph-based identity stores to link customer, supplier, or patient records that arrive with different names, addresses, or IDs. Without it, every siloed system carries its own incomplete view of each entity, and analytics, compliance, and operations pay the price.
Ready to connect the dots across your databases? Start a free trial of Match Data Pro and run your first entity resolution job in minutes — no contract required.
Why Siloed Databases Produce Fragmented Identities
Most organisations accumulate data in at least four to six separate systems: a CRM for customer contacts, an ERP for transactions, a marketing automation platform for campaign responses, a support ticketing system, and often one or more third-party data sources. Each system assigns its own internal ID. Each captures fields in its own format. A customer named “Robert J. Smith” in the CRM arrives as “Bob Smith” in the support system and “R. Smith” at 42 Oak St in the ERP.
This fragmentation has measurable consequences:
- Analytics teams count the same customer two or three times, inflating headcount figures.
- Sales reps contact the same prospect from two queues simultaneously.
- Compliance teams cannot produce a single view of customer exposure across product lines.
- Migration projects carry three times as many records as the destination system needs.
Data profiling typically reveals that 10–30% of records in a mature CRM are either duplicates or fragments of a record that exists in another system. Entity resolution collapses those fragments into a single, authoritative identity.
What Is Entity Resolution vs. Data Linkage?
The two terms overlap but describe slightly different scopes:
- Entity resolution identifies all records — within one dataset or across many — that refer to the same real-world entity and groups them into a cluster. The cluster is then collapsed into a golden record.
- Data linkage connects records across two distinct datasets using a persistent link, rather than necessarily merging them. The source records remain separate; a cross-reference table maps them to a common entity ID.
In practice, most data teams need both. A healthcare team links patient records across EHR, pharmacy, and lab systems (data linkage) and then resolves duplicates inside each source system (entity resolution) before analysis. Match Data Pro’s Senzing-powered entity resolution handles both operations in a single pipeline.
The Common Element: Similarity Scoring
Both entity resolution and data linkage depend on comparing field values that are not identical but describe the same thing. The algorithms that power this comparison include:
- Levenshtein distance: counts the minimum character edits between two strings. “Smyth” vs “Smith” scores 0.80.
- Jaro-Winkler: weights prefix agreement more heavily, useful for given names. “Jonathan” vs “Jon” scores 0.87.
- Phonetic encoding: NYSIIS or Double Metaphone, useful for names that sound the same but are spelled differently. “Garcia” and “Garsia” produce the same phonetic code.
- Token-based matching: splits fields into tokens before scoring, handling “Oak Street” vs “St. Oak” correctly.
No single algorithm works across all field types. A multi-algorithm weighted scoring model combines several scores into a composite match confidence that is more robust than any single metric.
The Six-Stage Entity Resolution and Data Linkage Pipeline
The diagram below shows the full pipeline from raw source records to a linked, golden-record output.
Stage 1: Data Profiling
Before linking anything, you need to understand what you have. AI-powered data profiling scans every field in every source dataset and reports null rates, format distributions, value ranges, and anomalies. A profile on a typical 500,000-record CRM might reveal that 22% of phone fields are null, that 14% of company names contain legal suffixes (“LLC”, “Inc.”, “Corp.”) in inconsistent formats, and that 8% of zip codes are five-digit strings stored as integers with leading zeros stripped. Each of these is a matching hazard that profiling surfaces before it causes false negatives downstream.
Stage 2: Standardisation
Data standardisation normalises field values so that matching algorithms compare like with like. Common transformations include:
- Expanding abbreviations: “St.” to “Street”, “Blvd.” to “Boulevard”.
- Stripping legal suffixes from company names: “Acme Inc.” becomes “Acme”.
- Normalising phone formats: “(800) 555-1234”, “800.555.1234”, and “8005551234” all resolve to E.164 format.
- Upper-casing and trimming whitespace on name fields.
Skipping this stage is the most common cause of false negatives. Two records that describe the same company — “Acme Corp.” in one system, “ACME CORPORATION” in another — will score below any reasonable threshold without pre-normalisation.
Stage 3: Blocking
Comparing every record against every other record is computationally impractical at scale. A dataset of one million records produces 500 billion candidate pairs. Blocking reduces this by grouping records into buckets using a shared key — first three characters of surname, five-digit zip code, or domain extracted from email address — and only comparing records within the same bucket. A well-designed blocking strategy reduces candidate pairs by 99.9% while retaining 98%+ of true matches.
Stage 4: Fuzzy Scoring
Each candidate pair within a block receives a composite similarity score. Match Data Pro’s AI-powered fuzzy matching engine applies configurable algorithm weights per field type and returns a single confidence score between 0 and 1. Consider these two records:
| Field | Record A (CRM) | Record B (ERP) | Score |
|---|---|---|---|
| First name | Robert | Bob | 0.72 |
| Last name | Smith | Smyth | 0.80 |
| Company | Acme Corp. | Acme | 0.91 |
| ZIP code | 94105 | 94105 | 1.00 |
| rsmith@acme.com | bob.smith@acme.com | 0.78 | |
| Composite | 0.84 |
A composite score of 0.84 sits in the review zone for most configurations. Raising the auto-link threshold to 0.90 routes this pair to a review queue; lowering it to 0.80 auto-links it. The right threshold depends on the cost of false positives vs. false negatives for your use case.
Stage 5: Entity Graph and Survivorship
Confirmed matches are written into an entity graph — a persistent identity store that maps every source record ID to a canonical entity ID. Match Data Pro’s integration with Senzing builds this graph incrementally: as new records arrive, they are scored against existing entities and either linked to one or registered as new. The graph persists across pipeline runs, so you never re-score pairs that were already adjudicated.
Survivorship rules then determine which field value “wins” in the golden record. Common rules include most-recent-write, most-complete-record, or source-system priority. A company record with a full address from the ERP overrides the partial address from the CRM because the ERP has higher field completeness. Match Data Pro’s configurable survivorship engine applies these rules field by field.
Stage 6: Output and Distribution
The golden records and the entity-to-source cross-reference table are pushed to downstream systems via REST API, file export, or direct database connector. Match Data Pro’s import/export connectors support CSV, JSON, SQL databases, Salesforce, HubSpot, and common data warehouse formats. The live fuzzy search API also allows downstream applications to query the entity graph in real time — useful for preventing new duplicates at point of entry.
Common Data Linkage Scenarios by Industry
B2B Sales and RevOps
A typical B2B SaaS company runs Salesforce for CRM, HubSpot for marketing, Zendesk for support, and a data warehouse for analytics. The same account appears in all four with slightly different company names and contact spellings. Without entity resolution, account-level revenue attribution is impossible, and territory assignment collides. With a linked entity graph, every system shares a canonical account ID and a clean contact list.
Financial Services
Banks and asset managers must link customer records across retail banking, brokerage, and insurance systems to produce a consolidated customer view for KYC and AML screening. A customer who holds both a checking account and an investment account appears in two core systems with slightly different address formats. Entity resolution links them under a single entity ID, enabling accurate exposure calculation and watchlist screening. Learn more in our guide to entity resolution in financial services.
Healthcare
Patient matching across EHR, lab, pharmacy, and claims systems is a patient safety issue, not just a data quality problem. A patient whose name is recorded as “Maria Elena Gutierrez” in the EHR and “M.E. Gutierrez-Lopez” in the lab system may receive duplicate orders or, worse, have results attributed to the wrong chart. Entity resolution in healthcare must handle hyphenated names, DOB transpositions (1987-03-14 vs 1987-14-03), and address changes since last visit. See our detailed walkthrough in entity resolution in healthcare.
Key Configuration Decisions
Threshold Strategy
Most production pipelines use a three-zone threshold model:
- Auto-link zone (≥ 0.85): records are linked without human review.
- Review zone (0.60–0.84): pairs are queued for adjudication by a data steward or ML classifier.
- No-link zone (below 0.60): records are kept separate.
The boundary between zones shifts based on domain. A healthcare team may set the auto-link threshold at 0.92 to minimise false positives. A marketing deduplication job may accept 0.80 because the cost of a missed merge is lower than the cost of a false positive.
Blocking Key Selection
Blocking keys must be both selective (small buckets) and robust (consistent despite data quality issues). Using a raw company name as a blocking key fails if one system stores “Acme” and another stores “Acme Corp.” A better key is the first five characters of the standardised company name combined with the five-digit zip code. Match Data Pro supports composite blocking keys and multiple overlapping blocking passes to maximise recall.
Incremental vs. Full Refresh
Full refresh re-scores every record pair from scratch on each pipeline run. This is accurate but expensive at scale. Incremental processing scores only changed or new records against the existing entity graph. Match Data Pro’s job automation engine supports both modes, with incremental runs typically completing in under five minutes for datasets under 10 million records when the entity graph is already populated.
How Match Data Pro Handles Entity Resolution and Data Linkage
Match Data Pro is a cloud SaaS platform that combines every stage of the entity resolution and data linkage pipeline in one environment. There is no contract and no infrastructure to provision. You upload or connect your source datasets, configure field mappings and algorithm weights, set your threshold zones, and run. The Senzing entity resolution engine handles graph construction and incremental updates. The AI profiling module flags data quality issues before the match job runs. CASS address verification standardises and validates every postal address before it reaches the scoring stage.
For teams building event-driven or real-time pipelines, the live fuzzy search API allows any application to query the entity graph and check whether an incoming record matches an existing entity before it is written — preventing duplicates at source rather than cleaning them up later.
Book a 30-minute demo to see a live entity resolution run on sample data from your industry, or register for a free trial and connect your first dataset today.
Frequently Asked Questions
What is the difference between entity resolution and record linkage?
Record linkage connects records across two datasets using a cross-reference table, leaving source records intact. Entity resolution goes further: it clusters all records referring to the same real-world entity — within and across datasets — and collapses them into a single golden record. Most enterprise pipelines need both, and the two terms are often used interchangeably in practice.
How many source systems can entity resolution handle at once?
There is no hard limit on the number of source systems. Each system’s records are profiled and standardised independently before entering the shared scoring pipeline. Match Data Pro supports ingestion from CSV files, SQL databases, REST APIs, and CRM connectors simultaneously. The entity graph accumulates matches from all sources into a single identity store regardless of how many systems feed it.
What match threshold should I use for entity resolution?
A starting point of 0.85 for auto-link and 0.60 for the review zone works for most general-purpose deduplication. Regulated use cases — healthcare patient matching, financial KYC — typically raise the auto-link threshold to 0.90 or higher. Tune thresholds by reviewing 200–300 manually adjudicated pairs from your specific data and measuring precision and recall at each candidate threshold.
Can entity resolution handle international names and addresses?
Yes. Phonetic encoders like Double Metaphone handle transliterated names across Latin-script languages. Token-based algorithms are effective for address fields where component order varies by country. For postal validation, Match Data Pro’s CASS address verification covers US addresses, and the platform’s fuzzy address matching handles international formats using configurable field weights for country-specific address structures.
How long does an entity resolution job take to run?
Runtime depends on dataset size, blocking strategy, and the number of source systems. A full-refresh job on 1 million records with two source systems typically completes in 20–45 minutes on Match Data Pro’s cloud infrastructure. Incremental runs against an existing entity graph — processing only new or changed records — typically complete in 2–8 minutes for datasets under 5 million records.