Entity Resolution: Reflections on the most common data science challenge
TL;DR
- A typical entity resolution implementation ingests records, standardizes useful fields, blocks candidate pairs, scores matches, assembles entities, preserves link evidence, and exposes resolved context to applications.
- The hardest scale problems come from pair explosion, weak blocking, repeated batch reruns, missing transitive links, and treating an entity ID as enough evidence.
- Tilores focuses on resolving and assembling records at ingestion so query-time applications can retrieve current resolved context without rerunning the whole matching process.
Table of Contents
Implementation checklist
| Stage | What happens | Common failure to avoid |
|---|---|---|
| Ingest and standardize | Bring source records into the workflow, preserve source IDs, normalize fields, and keep the original values available for review. | Losing provenance or overwriting messy source values before the team can explain a match. |
| Block or filter candidates | Reduce the comparison space by grouping records that are plausible matches before scoring individual pairs. | Using blocks that are so broad they recreate the full pair problem, or so narrow they miss true matches. |
| Score and decide matches | Compare candidate records with rules, similarity functions, machine learning, or a controlled combination of methods. | Treating a model score as final without thresholds, review paths, or an understanding of false positives and false negatives. |
| Assemble entities and links | Turn accepted links into entity groupings while preserving how records are connected and how certain each link is. | Keeping only a final entity ID and losing the link evidence needed for audit, correction, or explainability. |
| Expose resolved context | Let applications query the resolved entity context they need for fraud, KYC, support, analytics, or customer experience. | Assembling entities from scratch at query time or serving stale batch output without clear freshness guarantees. |
| Monitor updates and corrections | Handle new records, corrections, deletes, entity splits, and match-quality drift as the dataset changes. | Rerunning the whole pipeline for every change or leaving incorrect links in place because correction workflows were not designed. |
We noticed recently a post on Hacker News titled “Entity Resolution: The most common data science challenge”, which generated an interesting discussion about how difficult entity resolution is at scale.
We were too late to join the comments in the discussion thread, but thought it was worthwhile to write a post in answer to the top comment in the discussion, from user “smeeth”, which addressed the entity resolution tutorial linked from the original post:
Danger awaits all ye who enter this tutorial and have large datasets.The tutorial is fun marketing material and all but its FAR too slow to be used on anything at scale. Please, for your sanity, don’t treat this as anything other than a fun toy example.ER wants to be an O(n^2) problem and you have to fight very hard for it not to turn into one. Most people doing this at scale are following basically the same playbook:1) Use a very complicated speed optimized non-ml algorithm to find groups of entities that are highly likely to be the same, usually based on string similarity, hashing, or extremely complicated heuristics. This process is called blocking or filtering.2) Use fancy ML to determine matches within these blocks.If you try to combine 1 & 2, skip 1, or try to do 1 with ML on large datasets you are guaranteed to have a bad time. The difference between mediocre and amazing ER is how well you are doing blocking/filtering. Quote from user smeeth on Hacker News
From our experience, what most solutions, that follow this pattern, lack is that:
- 1) they do not provide the possibility to do real time data modifications,
- 2) they are missing transitive links between matches, or,
- 3) the links between the entries are not persisted.
(1): most of the time blocking and matching is done from within batch processes. Assign each entry from your data set to at least one block. And then compare each entry from a block with all other entries from the same block. Something that btw. does not require “fancy ML”, but could also be done using “simple” rules, that often are more explainable.
Eventually, you still end up with O(n^2) within each block. And only the size of each block determines how much of a problem that is for you. Ideally, each block contains only the entries that belong to each other - but that never happens. The issue starts once new data is added, as often people would just rerun the same process again - wasting valuable computing resources and time.
One thing that can be optimized is to not rerun the blocking for everything - only new data needs to be added to the existing blocks. And then you don’t need to rerun the whole matching among all blocks, but only the ones that were actually affected by the new data. Reality is however, that people are lazy (sometimes), have to solve other topics (often) or that even an optimized process still takes too long for real time updates (almost always). We’ve often seen solutions that due to their real-time requirements end up with some kind of crappy self-build recursive search that gets slower and slower the more data is added.
(2): As mentioned before, each entry of your data set can be assigned to at least one block. Unfortunately it often ends up with not “at least” but with “exactly one”. This results in scenarios where A is matching with B and C is matching with D, but ignoring the fact that also B might match with C, but C not with A and D not with B. We would call that deduplication, but not proper entity resolution. But even if you end up with putting entries into multiple blocks, you’re going to struggle with the next issue: identifying from all these connections you now have, the actual entities. It can be done e.g. with a connected components algorithm but is also tricky to scale.
(3): This one is fairly implementation specific, but one thing we often noticed was that tools that help you with entity resolution only provide you with an entity ID for each entry you have. Every entry with the same entity ID belongs to the same entity - simple, isn’t it?
But you just lost a bunch of valuable information: how all entries within one entity are actually connected and how certain each link is. The reason for this is duplicates. Let’s consider you have three records A1, A2 and A3, each containing the exact same information and therefore clearly matching. Now if you create all the links between these you end up with the tuples A1-A2, A1-A3 and A2-A3, so three links. However, if you have A1..A5 it is already 10 links. For A1..An it is n*(n-1)/2 links. For n=1000 that is almost half a million links. Now you could argue, that all of these duplicates don’t provide any value, but reality is, that often we’re not working with real duplicates, but with non-identical duplicates. That means data that is exactly the same for the matching relevant attributes, but differs in other attributes and therefore cannot be thrown away. A good solution needs to handle these differently than a mediocre solution.
See the discussion about this post on Hacker News here.
Short answer
A typical entity resolution implementation starts with ingestion and normalization, then uses blocking or filtering to reduce the number of candidate comparisons. Candidate records are compared with rules, models, or both; accepted links are assembled into entities; and the system keeps enough evidence to review, correct, and query the result.
The important production distinction is that entity resolution is not only a one-time deduplication job. Teams also need a way to handle new records, preserve transitive links, avoid repeated full reruns, and serve current resolved context to the systems that need it.
Why does blocking decide whether entity resolution scales?
Blocking or filtering is the step that keeps a matching problem from becoming a comparison of every record with every other record. It narrows the candidate set before the more expensive scoring step happens.
The tradeoff is precision versus recall at the candidate stage. Blocks that are too strict hide true matches; blocks that are too loose push too much work into the matcher and can make updates slow.
How should teams think about transitive links?
Entity resolution often has to reason over chains of evidence, not just direct pair matches. If one record links to a second record and that second record links to a third, the system needs a policy for whether those records belong to the same entity.
That policy should be explainable and reviewable. Otherwise a team can end up with simple deduplication labels instead of a reliable entity view that shows how records are connected.
Why is persisting link evidence different from storing an entity ID?
A final entity ID tells a downstream system which records currently belong together, but it does not explain the path that created the grouping. Link evidence is what lets reviewers inspect, correct, split, or defend a match.
This matters most when records are non-identical duplicates: they may share matching fields but still carry different attributes that cannot simply be discarded.
What changes when updates need to be current?
A one-time batch job can be enough for a historical cleanup, but operational systems need a plan for new records, changed attributes, deletions, and corrections. Reprocessing everything for each change can become expensive and slow.
Tilores addresses this by resolving and assembling records at ingestion. Query-time systems then retrieve the resolved context they need for the current workflow.
Frequently Asked Questions
- What does a typical entity resolution implementation look like end to end?
- It usually includes ingestion, standardization, blocking or filtering, match scoring, entity assembly, link persistence, review or correction workflows, and an API or query layer that exposes resolved context to applications.
- Why is blocking important in entity resolution?
- Blocking reduces the number of record pairs that need detailed comparison. Without it, large datasets can create too many comparisons to run efficiently, especially when new records keep arriving.
- Does entity resolution require machine learning?
- Not always. Some workflows use rules or similarity functions, others use machine learning, and many production systems combine methods. The right design depends on data quality, explainability needs, scale, and the cost of false matches.
- What are transitive links in entity resolution?
- Transitive links appear when record A links to record B and record B links to record C, even if A and C do not directly match. A production system needs a way to reason about those connections without collapsing distinct entities unsafely.
- Why should an entity resolution system preserve link evidence?
- Preserving link evidence helps teams explain why records were connected, review uncertain matches, correct wrong links, handle duplicates, and support audit or compliance workflows. A final entity ID alone is often not enough.
- How does Tilores handle query-time entity context?
- Tilores resolves and assembles records at ingestion. At query time, applications retrieve or use the current resolved context instead of rebuilding the entity from scratch for every request.
Evaluate Tilores on your own data
Use the next step that matches your evaluation stage.
See what resolved entity data does for your business — and your AI.