How to Run an Entity Resolution Vendor Bake-Off With Real Bank Data

TL;DR: A fair bake-off gives every entity resolution vendor the same governed data, hidden labels, operating constraints and scoring rule. Measure false merges, missed links, review effort, audit evidence and repeatability. Preserve the complete test record so procurement can explain the choice.

Book a Demo · Try Tilores Studio (Free)

On this page

1. What makes a vendor bake-off fair?
2. Which bank data should go into the test?
3. Who should label the truth set?
4. Which metrics should decide the result?
5. How should regulated bank workflows be tested?
6. How should the bank run each vendor session?
7. What should a Tilores evaluation prove?
8. How should procurement make the final decision?
9. Frequently asked questions
10. Sources

A bank can put several entity resolution vendors through the same presentation and still learn very little about the decision it has to make. The useful comparison begins when every supplier faces the same governed data, hidden truth set, operating constraints and evidence requirements.

That is where a bake-off becomes a procurement control rather than a sales demonstration. The bank needs to separate false merges from missed links, make review effort visible and keep the test record intact enough to explain a result later. The choice is then grounded in repeatable performance and operating fit, not in a vendor’s best demonstration case.

What makes a vendor bake-off fair?

A fair bake-off is a controlled decision exercise. The bank defines the customer population, truth policy, operating constraints, scoring weights and minimum gates before any vendor sees the holdout results. Every vendor receives the same input files, field definitions, compute boundary, tuning window and questions.

That control matters because entity resolution results depend on more than a matching algorithm. Normalisation, candidate generation, deterministic rules, fuzzy comparison, probabilistic scoring, thresholds, cluster assembly and reviewer decisions can each change the final entities. If one vendor receives cleaner data, longer tuning or a different definition of a correct match, the bank is comparing test conditions rather than systems.

For banks within the statement’s scope, the PRA expects firms using vendor models to validate their own use and monitor performance using their own outcomes. The same supervisory statement calls for clear criteria when selecting between candidate models. A procurement team can apply that discipline without pretending every entity resolution system is a regulatory capital model: own the question, own the labels and retain the evidence.

The joint guidance from the OCC, Federal Reserve and FDIC frames a third-party relationship as a life cycle: planning, due diligence and selection, contract negotiation, ongoing monitoring and termination. A bake-off should not just crown a winner. It should produce the evidence that life cycle needs for due diligence, contracting and ongoing oversight.

Write a one-page protocol and have risk, data protection, security, operations and procurement approve it. Vendors may challenge an impossible condition before the run starts. They should not be allowed to rewrite the scoring rule after results are visible.

Which bank data should go into the test?

Use a controlled extract that resembles the intended production workload. It should contain known duplicates, known distinct customers, incomplete records, stale values, transliterations, shared contact details, household relationships, legal entities and records that genuinely lack enough evidence for an automatic decision. Report the count and provenance of each slice.

The ICO says personal data must be adequate, relevant and limited to what is necessary for the stated purpose. That gives the bank a practical design rule: include fields that test the declared matching purpose, exclude fields that do not, and document the lawful basis, access controls, retention period and deletion path for the exercise.

Pseudonymised data is personal data in the hands of someone who holds the additional information. If the bank shares pseudonymised data without the additional information, it may be anonymous information in the recipient’s hands. That status depends on whether the recipient can reasonably identify people from the shared data or other information it may possess or obtain. Keep that additional information separate and secure, and test whether the transformation removes signals the matcher needs. A vendor should not receive a privacy advantage by being given a weaker problem, or an accuracy advantage by receiving unnecessary data.

Partition the approved extract into three packs:

• Tuning pack: representative labelled examples that every vendor can use to configure its system.
• Blind holdout: representative records with labels hidden from vendors until scoring is complete.
• Challenge pack: difficult cases chosen to expose false merges, missed links, unstable clusters and review burden.

Freeze all three packs by hash. Record every permitted transformation. If a vendor corrects an input defect, apply the same correction to every run or keep the original input and score the handling of that defect explicitly.

Who should label the truth set?

The bank should own the truth set. Vendors can explain their outputs, but they should not decide which output is correct. Use reviewers who understand the source systems and the operational consequence of a wrong merge. Give them a written label policy with three outcomes: match, non-match and insufficient evidence.

Send ambiguous pairs or clusters to a second reviewer without showing the first decision. Resolve disagreement through a named adjudicator and retain the evidence used. Keep the vendor identity hidden during scoring where practical. This reduces the chance that familiarity, presentation quality or a persuasive demo changes the ground truth.

The US Census Bureau’s record-linkage standard requires plans to cover verification, testing and evaluation of linkage results. The same standard identifies valid-link criteria and linking parameters as specification issues. It also identifies manual review as an accuracy-evaluation method. A bank truth set should therefore preserve more than a binary label. Keep the source fields considered, reviewer rationale, confidence, review date and any policy exception.

Do not tune on the blind holdout. Once a vendor sees a holdout label, that case belongs in the tuning history for any later round. Create a new blind set for a rerun that is meant to estimate unseen performance.

Three data packs pass through five mandatory gates before weighted scoring decides the result.

Which metrics should decide the result?

No single score captures the bank’s risk. Precision is the proportion of selected links that are correct. Recall is the proportion of correct links that the system selects. Both belong in the scorecard, but cluster-level errors and review effort show consequences that a pair-level average can hide.

Measure	What to count and report	Why procurement needs it
Precision	Correct selected links divided by all selected links	Shows the rate at which accepted links are right
Recall	Correct selected links divided by all known correct links	Shows how much known duplication the system finds
False merge rate	Predicted non-singleton clusters that contain records from more than one truth entity, divided by all predicted non-singleton clusters. Report the numerator, denominator, percentage and cluster as the unit.	Exposes decisions that can contaminate a customer view
Missed-link rate	Multi-record truth entities represented across more than one predicted cluster, divided by all multi-record truth entities. Report the numerator, denominator, percentage and truth entity as the unit.	Exposes fragmented customer identity
Cluster correctness	Whole entities reproduced correctly	Tests the output the bank will actually consume
Review burden	Cases referred, median handling time and disagreement rate	Turns uncertainty into an operating cost
Repeatability	Changed entities after an identical rerun	Reveals unstable behaviour
Decision evidence	Source records, attributes, rules, scores and change history available per decision	Shows whether reviewers can reconstruct a result

For mutually exclusive slices, report each slice’s numerator, denominator and percentage, then calculate the overall false-merge and missed-link rates from pooled numerators and pooled denominators. Do not average slice percentages. Set minimum gates before weighted scoring. For example, procurement might reject a system whose false merge rate, calculated as contaminated predicted non-singleton clusters divided by all predicted non-singleton clusters, exceeds the bank’s declared tolerance; it might also reject a system that loses required audit evidence or cannot complete an agreed deletion test. Only systems that pass every gate move to the weighted comparison.

Then apply weights that reflect the intended use. An illustrative score might place 45 points on match quality, 20 on review and operations, 15 on decision evidence, 10 on security and deployment, and 10 on total cost. Those numbers are an example, not a universal benchmark. The bank should approve its own weights before testing.

How should regulated bank workflows be tested?

Test the decisions around the matcher, not only the matcher. Build challenge slices for onboarding, duplicate customer masters, shared identifiers, customer remediation and identity preparation for sanctions, PEP and adverse media screening. Use bank-approved watchlist and case sources, with the required licences and controls.

Entity resolution should pass a resolved identity to the screening or case-management workflow. It should not silently replace the screening decision. For each slice, record whether the system surfaced aliases and linked records, which cases went to review, what evidence the reviewer saw and whether a correction could be applied without losing history. Our KYC and compliance workflow follows this resolve-first, screen-next boundary.

Add deliberate stress cases: common names, swapped fields, transliteration variants, reused phone numbers, old addresses, sparse legal-entity records and a customer whose records must remain separate. The point is to find the operating boundary. A polished average score is less useful than knowing which cases require review and why.

How should the bank run each vendor session?

1. Freeze the question and gates. State the intended use, unacceptable error, required evidence and decision owner.
2. Prepare the governed packs. Approve fields, access, purpose, retention, deletion and the hashes of all files.
3. Freeze the truth policy. Label the tuning pack, hide the holdout and record how disagreements are resolved.
4. Give equal conditions. Use the same tuning time, support boundary, hardware assumption and allowed transformations.
5. Run blind. Capture configuration, logs, outputs, exceptions, runtime and human interventions.
6. Score independently. Apply the bank’s labels and gates before revealing results to vendors.
7. Challenge and rerun. Review errors, apply one declared remediation round equally, then rerun an unchanged pack to test repeatability.

Keep a run ledger. It should connect each input hash to the configuration, software version, output hash, scorecard, reviewer decisions and remediation. A demo recording is useful context, but it is not the evidence packet.

What should a Tilores evaluation prove?

A Tilores evaluation should answer the same bank-owned questions as every other run. Test deterministic, fuzzy and probabilistic matching as distinct methods. Keep the rule configuration, thresholds, cluster outputs and reviewer evidence in the comparison pack. Production resolution uses the GraphQL API exclusively.

Tilores Studio runs and stores everything locally on the user’s computer. It exposes the same GraphQL API as the cloud product. Local mode is capped at 100,000 records per deployment. Connected mode is for running and monitoring live entity resolution. That allows a bank to use a representative local sample for a controlled evaluation, subject to its own approvals, while scoring production deployment separately against the bank’s target control boundary using our deployment options.

Do not give Tilores a different test because it can run locally. Keep the same data packs, time box, truth labels and gates. If the bank permits a second configuration round, grant the same opportunity to every shortlisted vendor and score the improvement alongside the extra effort.

How should procurement make the final decision?

Start with the failed gates. Record the false-merge and missed-link numerators, denominators, percentages and units for every slice and for the pooled overall result. A system that cannot meet an essential control should not win through a high average elsewhere. Then compare the weighted result, the confidence interval or sample uncertainty, the reviewer workload and the sensitivity of the ranking to reasonable weight changes.

Ask what changed between the first run and the final run. A large improvement may show useful configurability, but it may also reveal a heavy dependence on scarce specialist effort. Price that effort. Separate one-off implementation work from steady-state review, monitoring, change control and incident response.

The decision record should name the chosen system, rejected alternatives, data version, label policy, minimum gates, final weights, unresolved limitations and named risk owner. Retain enough material for an independent reviewer to reproduce the score. Then convert the winning configuration into a production acceptance test instead of treating the bake-off as a sales demonstration.

Book a Demo to plan a controlled comparison, or try Tilores Studio (free) to start with an approved local data pack.

Frequently asked questions

How large should a bank’s entity resolution bake-off data set be?

The bank needs a representative labelled set, not a universal record count. Size it around the diversity of sources, identities and costly edge cases, then report uncertainty and slice-level results. Add records when a material cohort has too few labelled examples to support a decision.

Should vendors be allowed to tune their matching rules?

Yes, under the same declared conditions. Give each vendor the same tuning pack, time allowance and access to subject-matter experts. Keep the blind holdout closed, log every configuration change and score the human effort required.

Can synthetic data replace real bank data in the bake-off?

No, synthetic data should be a rehearsal or supplement. It is useful for pipeline checks and rare constructed cases, but the final comparison needs an approved, representative sample of the data conditions the system will face.

Who should own the gold-standard labels?

The bank should own the labels and adjudication policy. Vendors may explain a result or identify an input defect, but bank-appointed reviewers should decide the reference outcome without changing the rule for one supplier.

Should the cheapest vendor win if accuracy is close?

Cost should decide only after every minimum gate passes. Compare total cost with review effort, configuration effort, monitoring, change control and the operational cost of the remaining errors.

Sources

1. Model risk management principles for banks, Supervisory Statement SS1/23, Prudential Regulation Authority.
2. Agencies Issue Final Guidance on Third-Party Risk Management, Office of the Comptroller of the Currency, Federal Reserve and FDIC.
3. Principle (c): Data minimisation, Information Commissioner’s Office.
4. Pseudonymisation, Information Commissioner’s Office.
5. How do we ensure anonymisation is effective?, Information Commissioner’s Office.
6. Statistical Quality Standard C4: Linking Data Records, United States Census Bureau.
7. Machine Learning and Multiple Imputation for Record Linkage, United States Census Bureau.
8. OpenSanctions Default dataset, OpenSanctions.
9. Tilores Studio free evaluation, Tilores.
10. Deployment options, Tilores.
11. KYC and compliance, Tilores.
12. About Tilores, Tilores.