Why Batch Entity Resolution Isn't Enough for Real-Time KYC
TL;DR: Batch entity resolution tools like Splink are excellent at offline analysis: deduplicating a list, labelling ground truth, modelling match probabilities. But KYC onboarding and sanctions screening happen at query time. When a customer signs up, you must resolve them against your existing entities and against watchlists within milliseconds, not wait for the next nightly batch. That gap is architectural, not a matter of tuning. A batch run produces a one-off pair list; real-time KYC needs a persistent, queryable entity index. In our benchmark that index answered searches in 8.6ms p95 on a 1M-record dataset. Use open-source batch tools for offline list analysis and modelling, and Tilores for the real-time production path.
See it on your data: Book a Demo to walk through real-time KYC resolution on your own records, or Get the Evaluation Build to try it locally.
What does “batch” actually mean for KYC?
Batch entity resolution runs over a fixed snapshot of data. You hand a deduplication tool a file of records, it links the ones that refer to the same entity, and it returns a list of matched pairs or clusters. This is the model behind Splink, the most credible open-source record-linkage library available today: built inside the UK Ministry of Justice, probabilistic, explainable, and free. It is excellent at what it does.
The problem is that KYC is not a snapshot. A prospective customer fills in an onboarding form and clicks submit. In that moment, before you open an account and before you move money, you need to answer two questions. Have we seen this person before, under a slightly different name or a stale address? And does this person, once resolved, match anyone on a sanctions or PEP list? Both have to be answered now, against the live state of your data and the live state of the watchlists.
A batch pipeline cannot answer “now.” By the time the next run executes, the customer has already onboarded. That is the gap this post is about.
Why is onboarding a query-time problem, not a batch problem?
Onboarding is interactive. The applicant is waiting, the front-end is blocking on a decision, and downstream systems (your KYC platform, your sanctions-screening engine, your fraud rules) need a single resolved subject to act on. If you screen the raw application form instead of the resolved entity, you screen a fragment: one spelling of the name, one address, one phone number. You miss the alias the same person used two years ago, the duplicate account opened under a maiden name, the linked record that would have changed the risk rating.
Resolving first and screening second is the order that produces the right answer. But “resolve first” only works if resolution is available synchronously, in the request path, at single-record granularity. You are not re-resolving a million records. You are asking, “given everything we already know, what entity is this one new record, and what is its full context?” That is a query, and it needs a query engine.
The architectural distinction is the same one we found in our reproducible benchmark against Splink: a batch tool produces a one-off pair list, while a real-time engine maintains a persistent, queryable entity index. A pair list is a finished artifact, with no API to ask it a new question after the run. An index is a live structure you search every time a customer arrives.
How fast does query-time resolution need to be?
Fast enough that it disappears inside the onboarding flow. Resolution is one step in a chain that also includes form validation, document checks, screening calls, and risk scoring; it cannot be the slow link.
In the benchmark, search latency against the persistent 1M-record entity index was 8.6ms p95 on LAN. That is the cost of resolving a new record against the entire known population and getting back its resolved context: a single lookup, not a full re-run. At that latency, real-time resolution fits comfortably inside a synchronous onboarding request with room to spare for everything else in the chain.
For comparison, the same benchmark showed Splink deduplicating all 1M records in 9.24 seconds as an embedded DuckDB library, with outstanding batch throughput, and free. But 9.24 seconds is a batch number: it is the time to process the whole list once, not the time to answer a single live query. The two numbers measure different things, which is exactly the point. Batch optimises whole-list throughput; KYC needs per-record latency.
| Dimension | Batch deduplication (e.g. Splink) | Real-time resolution (Tilores) |
|---|---|---|
| Unit of work | Whole dataset, re-run | Single record / single query |
| Output | One-off pair list or cluster file | Persistent, queryable entity index |
| Freshness | As of the last run | Current, including the record just submitted |
| Onboarding fit | After the fact | Synchronous, in the request path |
| Sanctions screening | Against a snapshot | Against the live resolved entity |
| Benchmark figure | 1M records in 9.24s (throughput) | 8.6ms p95 per search (latency) |
How does this change sanctions and watchlist screening?
Sanctions screening is where the batch gap is most dangerous. Watchlists change constantly. OFAC publishes through its Sanctions List Service, the UK maintains the UK Sanctions List, and the EU publishes a consolidated financial sanctions list. Your customer data changes constantly too. Screening a stale, partial customer record against a list creates avoidable ambiguity and avoidable misses.
One distinction matters here, and it is easy to blur: the list source is not the resolution engine. Where the sanctions, PEP, and adverse-media data comes from (OpenSanctions, OFAC, a commercial screening vendor) is a separate concern from the engine that resolves your customers into clean entities. Tilores does not publish or maintain watchlists. What it does is make sure the subject you screen is the fully resolved customer, with every linked record, alias, and duplicate collapsed into one entity, so that when your screening tool compares against the list, it is comparing the right, complete thing.
In practice the flow is: resolve the incoming customer against your existing entities at query time, then hand the resolved entity (including known aliases and permitted linked records) to your screening engine. The screening engine still owns the match-against-list logic and the case decision. Resolution guarantees it is screening a whole person rather than a fragment. The Wolfsberg sanctions screening guidance is clear that an alert is only a first step, and additional context is needed to confirm or discount a true match. A resolved entity is that context.
What about customer-360 at onboarding?
The same query-time resolution that protects screening also builds the customer-360 view at the moment it is most useful: account opening. When a new application arrives, resolving it immediately against your existing base answers “is this a returning customer?” before you create a duplicate. That prevents the silent proliferation of split identities that later break monitoring, inflate risk exposure, and undermine every downstream analytic.
Tilores resolves and links records at ingestion, and returns the resolved entity context at query time. It sits next to your existing MDM, CDP, data-warehouse, KYC, and AML systems, supplying the resolved customer key they lack rather than replacing them. Your KYC platform keeps its rules and case management; Tilores feeds it a clean, current subject. See Tilores Customer 360 for how that resolved profile syncs back to source systems.
Is the resolution still explainable enough for audit?
Yes, and explainability is non-negotiable in regulated onboarding. A match you cannot defend is a liability when the decision affects account opening, payment release, or a sanctions escalation. The reviewer needs to see why an entity was returned: which source records were linked, which fields matched, which fields conflict.
Tilores exposes reviewable signals through its API. Search results include score and hitScore (in the API documentation, score reflects overall match quality within an entity, while hitScore reflects alignment with the search parameters), and recordInsights surfaces the underlying record evidence. These are signals to inform policy and feed your audit trail, not universal thresholds to apply blind; calibrate them to your data and risk appetite. The point for KYC is that the resolved entity arrives with its evidence attached, so an analyst, or an auditor after the fact, can reconstruct the decision.
So when should you use batch tools, and when Tilores?
It depends on whether your KYC question lives in a snapshot or in the request path.
| Choose | When |
|---|---|
| Open-source batch (e.g. Splink) | Offline list analysis · de-duplicating a fixed dataset · labelling ground truth · modelling and tuning match probabilities · zero-cost, full-control R&D where the output is a list and there is no real-time requirement. |
| Tilores | Real-time KYC onboarding · resolving a single new record against your base in the request path · screening the resolved entity against watchlists synchronously · a persistent, queryable entity index · customer-360 at account opening. |
| Both | Batch tooling for prototyping, ground-truth labelling, and offline model work; Tilores for the production real-time path. |
None of this says batch resolution is bad. Our benchmark showed Splink reaching strong accuracy (F1 0.8867) at no cost, with excellent batch throughput, and it remains a great fit for offline analysis. This is a point about fit. The moment your requirement becomes “decide about this customer right now, against current data and current lists,” a pair list cannot answer and a persistent index can. KYC onboarding and sanctions screening are squarely in that real-time camp.
FAQ
Can’t I just run my batch deduplication more often?
Running a batch job hourly or every few minutes narrows the gap but never closes it, and it scales badly — each run reprocesses the whole dataset. KYC onboarding is synchronous: the applicant is waiting and the decision blocks on resolution. That needs a single-record query answered in milliseconds, which is a different architecture from re-running a batch on a tighter schedule.
Does Tilores provide the sanctions and watchlist data?
No. The list source — OpenSanctions, OFAC, the UK or EU lists, or a commercial screening vendor — is separate from the resolution engine. Tilores resolves your customer records into clean, complete entities so your screening tool compares the right subject against whatever list you use. It does not publish or maintain watchlists.
Where does resolution happen — at ingestion or at query time?
Both, by design. Records are resolved and linked at ingestion so the entity index stays current. At query time, the onboarding or screening workflow retrieves the resolved entity context — including the record just submitted — rather than assembling identity from raw fragments. Resolution at ingestion keeps the identity decision governed; the query-time lookup is what makes real-time KYC possible.
How fast is a real-time resolution lookup?
In our published benchmark, search against a persistent 1M-record entity index ran at 8.6ms p95 on LAN. That is the latency to resolve a single new record against the full known population — fast enough to fit inside a synchronous onboarding request alongside validation, document checks, and screening.
Does Tilores replace my KYC or sanctions-screening platform?
No. Tilores is a real-time entity resolution API that sits next to your existing MDM, CDP, warehouse, KYC, and AML systems. It supplies the resolved, current customer subject those systems need; they keep their own screening rules, case management, and final decisions. Resolution improves the input to screening — it does not take over the screening.
Is the latency figure reproducible?
Yes. The 8.6ms p95 figure, along with the full accuracy comparison and methodology, comes from our open benchmark at github.com/tilotech/tilores-splink-benchmark. The full write-up is in Splink vs. Tilores: A Reproducible Benchmark on 1 Million Records.
See what resolved entity data does for your business — and your AI.