Basics of entity resolution

Author: Steven Renwick, CEO and co-founder, Tilores. Tilores builds a real-time entity resolution API whose matcher combines deterministic and probabilistic, fuzzy machine-learning matching; it resolves and assembles records at ingestion and returns the resolved context at query time, sitting next to existing MDM, CDP, data-warehouse and KYC and AML systems rather than replacing them.

TL;DR

• Choose an entity resolution platform by testing match quality, source-system coverage, update speed, explainability, and operational controls on messy real data.
• Tilores is a fit when teams need to link and deduplicate customer, product, or other entity records across disparate systems while keeping source references available.
• Regulated or sensitive data requires explainable matching, deletion and access-request workflows, encryption, and confirmed privacy controls; do not rely on a black-box matcher.

Table of Contents

1. Next step with Tilores
2. Decision guide
3. You must know the person you are serving and the product that you are selling.
4. 1. What is entity resolution?
5. 2. What are the challenges in entity resolution?
6. 3. How can you perform entity resolution?
7. 4. Is there a method to balance the advantages and disadvantages of batch and real-time processing?
8. 5. Does entity resolution completely replace your database?
9. 6. Are there any data-related regulations that we should be aware of?
10. 7. How does GDPR apply to entity resolution?
11. 8. How does entity resolution apply to the real world?
12. 9. How does entity resolution help with products?
13. Conclusion
14. Short answer
15. What should an evaluator test first?
16. Which batch and real-time features matter?
17. Why does explainability matter in entity resolution?
18. How should entity resolution support privacy requests?
19. Does entity resolution replace existing databases?
20. Frequently Asked Questions

Next step with Tilores

Use the next step that matches your evaluation stage.

Book a Demo Get the Evaluation Build

Decision guide

Question	Use Tilores when	Watch-outs
Does the platform match messy records without collapsing distinct entities?	You need to link contacts, customers, accounts, or products across systems despite misspellings, partial fields, and duplicate entries.	Test false positives and false negatives on your own data before relying on any match score.
Can it support both retrospective cleanup and current operational queries?	You need batch-style cleanup for existing records and current entity context for new incoming data.	A pure batch workflow can leave the queried dataset stale; a pure real-time workflow must be tested for read and write contention.
Can teams explain why records were linked?	Matches influence fraud, credit, support, compliance, or customer experience decisions where auditability matters.	Black-box matching may be hard to defend when two entities are merged incorrectly or when a regulator, customer, or internal reviewer asks why.
Does the entity layer preserve source references?	You need a unified view that still points back to CRM systems, internal databases, marketing systems, email tools, or other source records.	A single entity view should not hide where attributes came from or make deletion and correction workflows harder.
Can the platform isolate source systems from resolution workload?	You need entity resolution to run without slowing the systems that create, update, and serve operational data.	Large imports, event streams, and deletion workflows still need load testing and clear operational ownership.

You must know the person you are serving and the product that you are selling.

If you know exactly who you serve as a business, or the product you are selling, you have successfully performed entity resolution. What this means is that you have taken any entity, be it a person or a product, collected any and every information regarding it, and then connected the information to that unique entity in such a way that there exists no other duplicate of the same entity.

In this article, we aim to explain the following basics of entity resolution,

• What is entity resolution?
• What are the challenges in entity resolution?
• How can you perform entity resolution?
• Is there a method to balance the advantages and disadvantages of batch and real-time processing?
• Does entity resolution completely replace your existing databases?
• Are there any data-related regulations that we should be aware of?
• How does GDPR apply to entity resolution?
• How does entity resolution apply to the real world?
• How does entity resolution help with products?

1. What is entity resolution?

Let’s say there are two names for the same person, Robert and Bob, and this “entity” (person) in question has acquired two different credit cards under both names. A bank with a proper entity resolution system in place can quickly call out the fraud, but if not, they are simply at a loss here.

There are varying levels of entity resolution for various industries. The level of entity resolution varies based on the use cases; eCommerce industries might only require basic information to resolve entities, but a finance company may need more data points for entity resolution.

Entity resolution in the real world has existed for a long time, and many industries have been doing this process for ages, but they often don’t have a specific name for it and can’t point a finger to explain what it is or how it applies to their business.

Knowing what entity resolution is and understanding how it impacts your business plays a huge role in shaping your business because data is everything and more importantly, correct data is everything!

With the enormous incoming stream of data available, businesses are discovering new ways to decipher data and apply artificial intelligence and machine learning to extract valuable insights from the data, but what use is all this data if they are badly structured?

This is where entity resolution comes in. Entity resolution is the process of deduplicating and matching the data that belongs to a particular entity in a way that there is only one single source of truth for that entity.

For example, a CRM database has the contact information of a person who is a real-world entity, but due to misspellings, wrongly entered data, broken imports, or multiple entries, the same person has many entries in the same system.

But with Entity resolution, you can quickly link all these contacts/datasets to one entity, remove duplicates, enrich the information by combining the available data across all datasets, and create one truth of data.

That is entity resolution for you!

2. What are the challenges in entity resolution?

Is it as easy as it looks? When you say entity data, you are talking about vast amounts of data, and performing entity resolution on a large scale can be a challenge. It is a challenge simply because you are comparing one data set with a sea of other data sets in the pool, and these datasets are also housed in different disparate data silos.

This means that for every dataset, there are n*(n-1)/2 possible unique pairs (also known as “edges”). So, for 10,000 datasets, there are already 49,995,000 potential combinations to compare. For a million datasets, this would equate to 499,999,500,000 possible combinations.

Several scientific techniques exist to solve this problem, such as blocking, clustering, and matching. Still, only a few software solutions can function at scale without slipping into the exponential edges trap.

Now that is a challenge right there.

So how do you approach solving entity resolution use cases?

3. How can you perform entity resolution?

You can resolve entities using rule-based methods or AI/ML-based approaches.

Let’s decode each of these approaches here.

Rule-based entity resolution involves defining the specific string-based rules to identify similarities between two entities. When you enter a request or a record per se, the rule-based process will match the record you entered with every other record in the database to deduce their similarities based on the predefined rules for that use case.

But rule-based entity resolution cannot be defined for all the possible use cases; it is unscalable and might sometimes give false positives and false negatives.

Whereas with AI/ML-based approaches, a set of data is fed into the system to provide the kind of similarities or dissimilarities that you want the system to look into.

Based on systematic learning from the training datasets and with the help of specific algorithms the machine learns to identify the similarities in an entity while matching records.

Though AI/ML-based entity resolution is easier to set up as you do not have to configure rules, the matches are not easily explainable which leads to problems when mixing up two entities without knowing why.

Whether you follow Rule-based or AI/ML-based approaches, there are two common ways the data is processed within these approaches: They are Batch Processing and Real-time processing.

Batch Processing

• Batch processing means the incoming data is collected and stored until a particular amount of data is collected and then processed together in a batch.
• Batch processing has two steps: live query and data processing.
• Live query provides immediate results during searches and a read-only copy of the data.
• Data processing involves matching the incoming data, handling data deletions, adding new data to the existing records, etc.
• The advantage of the batch query is that the live query is immediate with no lag since the data in a batch is already processed, but the downside is it doesn’t include the new incoming data, and every time you perform batch processing, the live query has to be shifted to the latest batch.
• The data processing is performed once a day or week, but these intervals are not always set since the amount of incoming data continuously varies, meaning that the dataset on which you are performing a live query is always outdated.

Real-time processing

• Unlike live queries in batch processing, real-time processing involves processing incoming data in real-time as soon as it enters the system. The finest thing is that, unlike a read-only copy, the data is current and instantaneously accessible.
• The disadvantage in real-time processing can be due to the nature of competing interests — read and write — the writes can slow down reads and vice versa. Large batch uploads can also cause the reads to be timed out.

4. Is there a method to balance the advantages and disadvantages of batch and real-time processing?

Can you achieve batch processing speed combined with the up-to-the-minute timely query replies of real-time processing?

The answer is serverless technology for entity resolution.

Serverless technology abstracts all the underlying technology stack that a user must maintain, such as backend infrastructure, integration platforms, applications, etc.

Instead, it provides a function-based environment that responds to user requests as and when it comes, thus providing scalability and speed to process any user request.

Serverless technology for entity resolution eliminates read and write conflicts, provides the latest up-to-date data, query times are more predictable, and the hard data switches that occur with batch processing are eliminated.

5. Does entity resolution completely replace your database?

Entity resolution effectively adds a new database to the existing ones, but there is also the option of creating a stand-alone data layer, such as a database or a data lake that stores entity data in a user-centric manner.

The entity resolution ensures that all the data is stored in one single plane with references to source databases such as CRM systems, internal databases, collaboration software like email systems, marketing databases, etc.

As a result, any live query to the entity resolution software provides you with the desired data and retrieves all the references to source databases.

Thus you can perform client data requests or deletion requests within seconds without any human involvement.

However, the disadvantage of centralizing all company data is that this system is in high demand in terms of availability and performance. The entity resolution software must not cause any data loss or slowdown in the source systems.

So how can you tackle this issue?

Event streams were developed to address the issue of source systems slowing down, allowing source systems to be isolated from entity resolution software.

Communication happens asynchronously between source systems and entity resolution software with event streams. Any data from source systems is pushed to the event stream, which subsequently reaches the entity resolution software asynchronously.

The result is

• Requests for data, changes, or deletions from entity resolution software are sent to the source database in the form of events.
• You have a data layer in addition to your source database.
• Any slowdowns in entity resolution software do not slow down the source database.

Now that we know more about entity resolution, how is data handled, and where is data stored.

6. Are there any data-related regulations that we should be aware of?

While processing various data for entity resolution, there is one very important thing to remember, entity data about individuals may be private, and there are laws in place to protect an individual’s data rights.

While processing sensitive personal data, it is critical that any mechanisms you use to execute entity resolution conform with rules such as GDPR or CCPA; failure to do so implies you will be unable to process personal data.

This means that

• The purpose for combining two datasets must be explainable (which in the case of ML sometimes is not easily possible)
• You must automatically erase data after a set length of time or when the legitimate reason for retaining it has passed.
• A person’s entire dataset must be made available upon request.
• You must remove data upon request.
• You must encrypt data before being stored.

7. How does GDPR apply to entity resolution?

General Data Protection Regulations (GDPR) for the European Union came into effect in 2018, keeping in mind the exploitation of personal data that was taking place by companies and other organizations for their own personal gain.

In light of such exploitation, a set of laws were released to protect consumer data.

Any organization that processes personal data must disclose the information used for data processing upon the user’s request and must also delete data from their systems on request.

Why is it difficult to implement GDPR compliance at an enterprise scale?

The problem is that with massive data stored in disparate systems and an entity not having a unique identification number, it is rather tricky to query the data immediately upon a consumer’s request.

Now the data science team and the legal team must be involved in extracting this data from the disparate systems manually. Even this process is complicated, since two subsequent data disclosure requests could lead to different results.

The most challenging part is dealing with data deletion requests, which might come from consumers or as part of GDPR compliance, which is challenging to perform due to information residing in diverse data sources.

One solution to handle GDPR compliance is entity resolution.

Though entity resolution has its roots in personal data management, this can be very beneficial since the entity’s data is stored in one single space. Hence, it is easier to respond to consumer requests for data access or data deletions.

Using entity resolution, the data in all the source systems can be easily deleted, ensuring GDPR compliance and saving time and effort for the data science and legal teams.

8. How does entity resolution apply to the real world?

Let’s look at a few industries that use entity resolution to tackle business issues head-on.

Ecommerce

Ecommerce is an industry where decisions have to be made in real-time since customers can take advantage of the slightest loopholes.

Suppose an eCommerce store offers vouchers or discounts to newbie customers. In that case, an existing customer can take advantage of these first-time offers simply by registering with a different email address, leading to voucher fraud.

The buy now pay later option provided by Ecommerce stores can be taken advantage of because the response to queries from payment and security systems is slow or all the data regarding the customer is unavailable in one single place.

Fraudulent orders are placed using the address of an unsuspecting customer who receives invoices for the orders they never placed, leading to delivery fraud.

Health Industry

Timely intervention is possible with all the information regarding a patient in one single database with even the minute details of the medicines they are allergic to and their past surgeries.

Thus a better relationship with the patient develops, and a complete view of their health is available, helping them achieve the best possible treatment.

Finance industry

The finance industry is the most vulnerable to fraudulent activities. Any slight mismatch in the data related to a person might lead to wrongly evaluating their credit scores, not detecting debt frauds, or even issuing loans to an ineligible individual.

The common thread among all these industries is that even a slight mismatch in the entity data can lead to enormous losses, negatively affect the customer experience, and fraud can go undetected.

To ensure good entity resolution for any industry use case, two things must happen: one, the data must be correctly related to the same person or product, and two, the data regarding the person or the product must be accessible in a split second.

We looked at how entity resolution helps decode information connected to people, but what about product data?

9. How does entity resolution help with products?

When processing product data, it’s not uncommon for products to be duplicated because the title or description differs significantly.

Assume you plan to purchase a camera and search for “canon Ixus 285 black.”

These are the search results that crop up.

“Canon Ixus 285 Hs Black Compact Camera”, “Canon Ixus 285 HS black”, “Canon IXUS 285 HS 20,2 Megapixel Full HD Compact Camera, 25–300 mm “, “Canon Ixus 285 HS BLACK 20,2 “, “Canon IXUS 285 HS Essential Kit black “, “Canon Ixus PowerShot 285 HS (Black) “

But all you just wanted to see was the Canon Ixus 285 black model and not six different Canon Models.

With entity resolution, the details of only the model you searched for comes up in the catalog and not six other products helping your search get accurate, faster, and better.

Conclusion

Entity resolution is a necessity for the success of a business. Entity resolution backed with serverless technology is even better.

With serverless technology, you can scale entity resolution to handle large-scale requests and, at the same time, maintain speed.

When done correctly, entity resolution allows you to have all of the information you need at your fingertips. With this single source of truth, you can make split-second decisions, cater to customer needs, assess business needs, and derive insights from every segment of your business, effectively increasing your enterprise value exponentially.

Short answer

The most important features in an entity resolution platform are accurate matching, duplicate detection, source-system linking, real-time or near-real-time updates, explainability, privacy controls, and the ability to scale without slowing operational databases.

For a Tilores evaluation, test the platform against real customer, product, or account records with misspellings, partial identifiers, changed addresses, duplicate emails, and records that should not be merged. The platform should create a useful single entity view while preserving enough context to review, correct, delete, or trace the underlying source records.

What should an evaluator test first?

An evaluator should start with real records, not clean sample data. Useful test cases include nicknames, spelling mistakes, changed addresses, duplicate emails, incomplete product names, and two similar records that must remain separate.

The goal is to see whether the platform can find the same person, product, or account across messy inputs while avoiding unsafe merges. That test is more useful than a generic feature checklist because entity resolution quality depends on the data and use case.

Which batch and real-time features matter?

Batch processing is useful for historical cleanup, but it can leave the query layer behind the newest data. Real-time processing is useful when new records need to be linked as they arrive, but it must be designed so writes do not slow reads and reads do not block writes.

The original Tilores article frames serverless technology and event streams as ways to reduce those trade-offs. In an evaluation, ask how incoming data is processed, how current the resolved entity view is, and how the platform behaves during large uploads or deletes.

Why does explainability matter in entity resolution?

Explainability matters because a wrong merge can affect fraud detection, credit decisions, customer support, product search, or compliance workflows. Teams need to understand why two records were linked and how to correct the link if the match is wrong.

Rule-based systems can be easier to explain but harder to scale across every edge case. AI or machine-learning approaches can reduce manual rule configuration, but evaluators should confirm how match decisions are reviewed, audited, and corrected.

How should entity resolution support privacy requests?

An entity resolution platform that processes personal data should help teams find the full dataset attached to a person, make that dataset available on request, remove data on request, and support encryption before storage.

GDPR and similar privacy laws depend on the organization, data type, lawful basis, and implementation. Treat entity resolution as part of the privacy workflow, not as a blanket compliance guarantee.

Does entity resolution replace existing databases?

Entity resolution should not be evaluated as a simple replacement for a CRM, database, or data lake. The better question is whether it can add a resolved entity layer that links source records while keeping references back to the systems of record.

That matters when customer data sits across CRM, marketing, internal, email, ecommerce, or product systems. A useful platform gives applications a unified entity context without making the original systems disappear.

Frequently Asked Questions

What features should I look for when choosing an entity resolution platform?

Look for accurate matching, deduplication, source-system linking, batch and real-time processing options, explainable decisions, privacy controls, and the ability to scale without slowing operational databases.

Is entity resolution only data deduplication?

No. Deduplication is part of entity resolution, but entity resolution also links related records, combines attributes from multiple datasets, preserves source references, and creates a more reliable view of a person, product, account, or other entity.

Should entity resolution run in batch or real time?

It depends on the use case. Batch processing can be useful for historical cleanup, while real-time processing is important when new data must be matched as it enters the system.

Does an entity resolution platform replace a CRM or database?

Usually no. Entity resolution commonly adds a layer that links and deduplicates records across existing systems while keeping references back to source databases, CRMs, and other operational tools.

How should an entity resolution platform support GDPR requests?

It should help teams find the data attached to a person, return that data when required, delete it when required, and handle storage and processing with appropriate controls. Compliance depends on the full implementation and should be validated for the specific organization.

Evaluate Tilores on your own data

Use the next step that matches your evaluation stage.

Book a Demo Get the Evaluation Build