The Complexities of Entity Resolution Implementation
TL;DR
- A typical entity resolution implementation starts with record profiling and fuzzy matching, then reduces candidate pairs with blocking, scores likely matches, groups matches into entities, and stores the evidence needed to explain, update, split, or delete entities later.
- The hard part is not one string-comparison algorithm. The hard part is controlling quadratic pair growth, choosing blocking keys that do not miss real matches, handling transitive links, and keeping entities current as records change.
- Building in-house can work for narrow, batch-oriented data sets. A dedicated entity-resolution layer is usually a better fit when teams need multi-system customer matching, ingestion-time entity assembly, deletions, explainability, and API access to current unified customer context.
Table of Contents
Short answer
A typical entity resolution implementation has six practical stages: normalize incoming records, choose fuzzy matching rules or models, reduce the candidate set with blocking, compare candidate pairs, assemble matching records into entities with graph logic, and expose the resulting entity plus match evidence to downstream systems.
The implementation becomes difficult when the data set grows, changes, or needs auditability. Teams have to manage blocking tradeoffs, false positives, false negatives, transitive matches, graph size, real-time ingestion, delete requests, and the operational cost of rerunning or repairing batches.
Next step with Tilores
Use the next step that matches your evaluation stage.
Decision guide
| Question | Use Tilores when | Watch-outs |
|---|---|---|
| Can every record be compared with every other record? | The data set is large enough that candidate selection, blocking, and entity assembly need to be handled as production architecture rather than a one-off script. | Naive pairwise comparison grows quadratically. A rule that works on thousands of records may become unusable on millions. |
| How often do records change? | New, changed, or deleted records need to update resolved entities without waiting for a full monthly batch rerun. | Real-time and incremental approaches still need careful blocking, merge tracking, split behavior, and deletion handling. |
| Do teams need explainable matches? | Reviewers, analysts, or downstream systems need to understand why records were linked and retrieve current unified customer context through an API. | Dropping match evidence can make entities harder to audit, split, debug, or improve later. |
| Is the use case narrow or business-critical? | Entity resolution is central to customer, fraud, risk, compliance, or AI-agent workflows across multiple systems. | A small static batch can justify a custom implementation. A changing multi-source environment usually exposes maintenance and accuracy costs quickly. |
What happens in an entity resolution pipeline?
The pipeline usually starts by standardizing records so fields such as names, addresses, emails, dates, and identifiers can be compared consistently. It then creates candidate pairs, applies fuzzy matching rules or models, decides which pairs are likely matches, and assembles those pairwise decisions into entity groups.
The final system needs to store more than a winning entity ID. It should retain enough match evidence, edges, thresholds, and source-record history to support explainability, entity splits, data deletion, and future rule tuning.
Why blocking is the core scaling decision
Blocking reduces the number of record pairs that need expensive comparison. Instead of comparing every record against every other record, the system groups records into candidate buckets based on keys such as city, phonetic name, q-grams, email fragments, or other domain-specific signals.
Good blocking improves speed without losing too many real matches. Poor blocking either creates huge buckets that bring back the quadratic problem or creates narrow buckets that separate records that should have matched.
Where graph logic enters the implementation
Pairwise matches are not the same as resolved entities. If record A matches record B and record B matches record C, the system may need to treat all three as one connected component, even when A and C were not directly compared or did not score above the same threshold.
That graph layer is useful for explainability and repair, but it also creates operational questions: how to merge entities, how to split them when a link is removed, how much edge evidence to keep, and how to keep the graph manageable when many records look similar.
How to evaluate build versus buy
An in-house build is most realistic when the data volume is small, the matching rules are stable, and delayed batch processing is acceptable. The team still needs ownership for rule tuning, blocking design, performance testing, monitoring, and deletion handling.
A dedicated entity-resolution layer is a better fit when the business needs reliable matching across messy customer data, frequent updates, explainable results, and unified customer context available to applications or AI agents without rebuilding the matching engine from scratch.
Entity resolution is the process of determining whether two or more records in a data set refer to the same real-world entity, often a person or a company. At a first glance entity resolution may look like a relatively simple task: e.g. given two pictures of a person, even a small child could determine if it shows the same person or not with a quite high accuracy. And the same is true for computers: comparing two records that contain attributes like names, addresses, emails etc., can easily be done. However, the deeper one digs into that topic, the more challenging it gets: various matching algorithms need to be evaluated, handling millions or billions of records means quadratic complexity, not to mention real-time and data deletion use cases.
Fuzzy Text Matching
Lets start with looking into comparing two records of the famous artist Vincent Van Gogh – or was it Van Gough?
There are a few errors in the second record (beside being born a century later and an email address): the name is spelled incorrectly, the day of birth is mixed up, the postcode is missing and the email address is slightly different.
So how can we compare those values? If, lets say, the names would be equal, then a simple string comparison of those values would be sufficient. As this is not the case, we need some more advanced fuzzy matching. There are many different algorithms available for text-based fuzzy matching and they can be roughly separated into three groups. Phonetic algorithms focus on how similar a text is pronounced. The most famous algorithms are Soundex and Metaphone, which are mostly used for English texts, but variations of those exist for other languages too like the Kölner Phonetik (Cologne Phonetic) for German. Text distance algorithms typically define how many characters of a text need to change to reach the other text. Levenshtein and Hamming distance are two well known algorithms in that group. Similarity algorithms, such as Cosine Similarity or the Jaccard Index, compute the structural similarity of texts and often represent the similarity as a percentage.
For the purpose of this article, we will use a very simple approach, using only a Levenshtein distance on the name and equality on the city. This example and all following examples will use golang as a programming language and use existing libraries where possible. Converting this into python, java or any other language should be trivial. Also it will only perform matching on the name attribute. Adding more attributes or even making it configurable is not the purpose of this article.
Try in the Go Playground: https://go.dev/play/p/IJtanpXEdyu
The Levensthein distance between the two names is exactly 3. That is because, there are three additional characters (“en” in the first name and “u” in the last name). Note, that this works with this specific input. It is however far away from perfect. E.g. the names “Joe Smith” and “Amy Smith” also have a Levenshtein distance of three, but are obviously not the same person. Combining a distance algorithm with a phonetic algorithm could solve the issue, but that is out of scope for this article.
When using a rule-based approach, instead of a ML-based approach, choosing the right algorithms that yield the best results for your use case is the most crucial aspect of your business success. This is where you should be spending most of your time. Unfortunately, as we will discover now, there is a ton of other things that will distract you from optimizing those rules if you decide to develop an entity resolution engine yourself.
Naïve Entity Resolution
Now that we know how to compare two records, we need to find all records that match with each other. The easiest approach is to simply compare each record with all other records. For the purpose of this example we work with randomly chosen names and cities. For the names we force up to three errors (replace any character with x).
Try in the Go Playground: https://go.dev/play/p/ky80W_hk4S3
You should see some output similar like this (you may need to run it multiple times if you do not get any matches for the randomized data):
If you are lucky, you will also get mismatches like “Daisy” and “Dave”. That is because we are using a Levenshtein distance of three, which is way to high as the sole fuzzy algorithm on short names. Feel free to improve this on your own.
Performance wise, the real problematic bit is the 9,900 comparisons needed to get to the result, because doubling the amount of the input will roughly quadruple the amount of needed comparisons. 39,800 comparisons are needed for 200 records. For the small amount of only 100,000 records that would mean almost 10 billion comparisons are needed. No matter how big your system is, there will be a point at which the system will not be able to finish this in an acceptable time as the amount of data grows.
A quick, but almost useless optimization is to not compare every combination twice. It should not matter if we compare A with B or B with A. However, this would only reduce the amount of comparisons needed by factor two, which is neglectable due to the quadratic growth.
Complexity Reduction by Blocking
If we are looking at the rules we created, we easily notice that we will never have a match if the cities are different. All those comparisons are completely wasted and should be prevented. Putting records which you suspect are similar into a common bucket and others that are not into another one, is in entity resolution called blocking. As we want to use the city as our blocking key, the implementation is fairly simple.
Try in the Go Playground: https://go.dev/play/p/1z_j0nhX-tU
The result will now be the same, but we have only roughly a tenth of the comparisons as previously, because we have ten different cities. In a real application this effect would be tremendously bigger due to the much higher variance of the cities. Furthermore, each block can be processed independently of the others, e.g. in parallel on the same or on different servers.
Finding the right blocking key can be a challenge on its own. Using an attribute like the city could result in uneven distributions, and therefore result in cases where one huge block (e.g. a large city) takes a lot longer than all other blocks. Or the city contains a tiny spelling error and is no longer considered as a valid match. Using multiple attributes and/or using phonetic keys or q-grams as a blocking key can solve these issues, but increases the complexity of the software.
From Matches to Entity
So far all we can say about our records is, whether two of them are matching or not. For very basic use cases this might already be sufficient. However, in the majority you want to know all matches that belong to the same entity. This can reach from simple star-like patterns where A matches with B, C and D, over chain-like patterns where A matches B, B matches C and C matches D, to very complex graph-like patterns. This so called transitive record-linkage can easily be implemented using a connected component algorithm as long as all data fits in memory on a single server. Again, in a real world application, this is much more challenging.
Try in the Go Playground: https://go.dev/play/p/vP3tzlzJ2LN
The connected components function iterates over all edges and either creates a new component, adds the new id to the existing component or merges two components into a single one. The result then looks something like that:
Keeping those edges gives us a few advantages. We can use them to make the resulting entity understandable and explainable, ideally even with a nice UI that shows how the records of an entity are connected. Or when working with a real-time entity resolution system, we can use the edges to split an entity as data was removed. Or you use them when constructing a graph neural network (GNN), leading to even better ML results instead of just the records alone.
Similar Data and Other Challenges
One issue from the edges can arise when there are a lot of very similar records. If e.g. A matches with B and B matches with C, then C might also match with A depending on the used rules. If D, E, F and so on also match with the existing records, then we are back to the quadratic growth issue, soon resulting in so many edges that they become no longer handleable.
Remember how we built the blocking buckets? Surprise! For very similar data, which all ends up in a few huge buckets, the performance of the computation drops drastically again – even if you followed the previous advice of creating buckets from multiple attributes.
A typical example for these kind of non-identical duplicates is someone ordering regularly in the same shop, but with guest access (sorry, no nice customer id). That person might be using almost always the same delivery address and is mostly capable of writing their own name correctly. Those records therefore should be treated in a special way to ensure a stable system performance, but that is a topic on its own.
Before you feel too comfortable with your gained knowledge and want to start implementing your own solution, let me quickly crush your dreams. We have not yet talked about the challenges of doing any of this in real-time. Even if you think you will not need an always up-to-date entity (the obvious benefit), a real-time approach yields further value: you don’t need to do the same calculations over and over again, but only for the new data. On the other hand, it is much more complex to implement. Want to do blocking? Compare the new records with all the records of the bucket(s) it belongs to, but that can take a while and could be rather considered as incremental batch. Also until that finally finished, there are a ton of new records waiting already to be processed. Want to calculate the entities using connected components? Sure, keep the whole graph in memory and just add the new edges. But do not forget to keep track of the two entities that just have been merged together due to the new record.
So, you are still willing to implement this on your own. And you made the (in that case) wise decision, to not store edges and not support real-time. So you successfully ran your first entity resolution batch job with all your data. It took a while, but you will only do this once per month, so that is fine. It is probably just about then when you see your data protection officer running around the corner and telling you to remove that one person from your data set due to a GDPR complaint. So you run the whole batch job again for a single removed entity – yay.
Conclusion
Doing entity resolution may at first look fairly simple, but it contains a lot of significant technical challenges. Some of these can be simplified and/or ignored, but others need to be solved for good performance.
Frequently Asked Questions
- What does a typical entity resolution implementation look like end to end?
- It profiles and normalizes source records, selects candidate pairs through blocking, scores likely matches with fuzzy matching rules or models, assembles matches into entities, stores match evidence, and exposes the resolved entity to downstream systems.
- Why does entity resolution become difficult at scale?
- The simple approach compares every record with every other record, which grows quadratically. Large data sets also introduce uneven blocking buckets, similar records, graph-management problems, update handling, and deletion requirements.
- What is blocking in entity resolution?
- Blocking is the practice of grouping records into likely candidate buckets before comparison. It reduces wasted comparisons, but the blocking keys must be chosen carefully so the system does not miss real matches or create oversized buckets.
- Should companies build their own entity resolution engine?
- A custom engine can work for limited, stable, batch-oriented use cases. Companies should consider a dedicated entity-resolution layer when matching is business-critical, multi-source, high-volume, explainable, or needed for current customer context in operational systems.
Evaluate Tilores on your own data
Use the next step that matches your evaluation stage.
See what resolved entity data does for your business — and your AI.