How to Build a Ground-Truth Dataset for Entity Resolution in 2026

TL;DR: Build ground truth from the real source mix, deliberately sample hard candidate pairs and obvious controls, define a three-way labelling policy, label independently, adjudicate disagreements, preserve provenance, and lock a held-out test partition. Measure pair and entity outcomes by source and risk segment. Refresh the set through versioned additions without rewriting historical labels to suit the latest model.

Test your matching configuration on a controlled data extract. Book a Demo · Try Tilores Studio (Free).

On this page

1. What counts as ground truth in entity resolution?
2. How should teams sample for decisions?
3. Turn judgement into a reproducible label
4. Build the dataset in seven controlled steps
5. Measure the failures the business can feel
6. How does ground truth remain independent of the current matcher?
7. How do we connect labelled truth to Tilores rules?
8. FAQ
9. Sources

What counts as ground truth in entity resolution?

Ground truth is a versioned set of identity decisions with enough evidence to justify each label. At pair level, the labels are usually match, non-match and unresolved. At entity level, the set also describes which records belong together and which must remain apart. The unresolved class is essential: when evidence cannot distinguish two people or organisations, labelling the pair to make the spreadsheet complete introduces false certainty.

The set must represent the population and decisions the system will face. A clean sample of exact identifier matches is useful as a control but tells little about transliteration, common names, household addresses, stale contact details or conflicting source IDs. Conversely, a dataset made only from a review queue assembled on one day exaggerates ambiguity because the current system already selected those cases. Build the corpus from multiple sampling frames and document each one’s purpose.

The UK Government’s quality assessment guidance stresses that linkage quality depends on both false matches and missed matches, with evaluation shaped by the available reference data. That is why accuracy alone is unhelpful. A large number of easy non-matches can make a weak system look excellent while the scarce, consequential false merges disappear in the average.

How should teams sample for decisions?

Start with a population inventory: sources, time periods, locales, record types, missingness patterns and the downstream decisions that consume resolved entities. Draw a random sample to estimate ordinary performance. Add stratified samples for rare but important conditions, such as duplicated identifiers, shared addresses, source-specific truncation and records near current thresholds. Keep the sample reason on every case so weighted aggregate metrics can be calculated later.

Include both candidate pairs and missed-candidate probes. Reviewing only the pairs returned by candidate generation cannot reveal people the blocking strategy never placed together. Use known clusters, source relationships or controlled perturbations to test candidate recall. Synthetic variations can diagnose one transformation, but they should not replace real examples when estimating production quality because real missingness and correlation are harder than isolated typos.

Prevent entity leakage across partitions. If records from one real entity appear in both training and test sets, a model or tuning process can learn that identity rather than the intended general pattern. Split by entity, source group or time boundary as appropriate, and freeze the test partition before threshold selection. A temporal holdout is valuable when production performance must survive new records and shifting source quality.

Ground truth is produced through sampling, independent judgement, adjudication and protected evaluation, not by copying model outputs.

Turn judgement into a reproducible label

Write a labelling handbook before scaling annotation. Define what a match means for the use case, which evidence can be used, how conflicts are handled and when to select unresolved. Include examples at the boundary, not only obvious cases. A customer account, a legal person and a household are different entity definitions; a label is meaningless without saying which one the reviewers are judging.

Use at least two independent judgements for the difficult strata. Reviewers should not see each other’s first decision or the model’s preferred result. An adjudicator resolves disagreements using the same evidence policy and records why. Agreement metrics reveal ambiguous instructions, but high agreement does not prove correctness if both reviewers share the same missing context or bias.

Protect personal and sensitive information throughout the process. Show only attributes authorised for the labelling purpose, control access, and keep a record of which evidence version was available. Redaction can change the difficulty of a case, so document it. A later auditor must be able to distinguish a wrong label from a reasonable label made with incomplete authorised evidence.

Build the dataset in seven controlled steps

1. Write the entity definition. State whether the target is a person, organisation, account, household or another object. Name the downstream decision and the errors that matter most. This becomes the basis for every label.
2. Inventory the population. Record source, period, locale, schema version, missingness and known quality events. Decide which segments require separate measurement instead of one blended score.
3. Create sampling frames. Combine a representative random sample with risk-based strata, current ambiguous candidates and missed-candidate probes. Save inclusion probability or weighting information where aggregate estimates are needed.
4. Prepare evidence packets. Present original and normalised values, source provenance and permitted corroborating context without showing the model’s prediction. Give every case a durable ID so corrections do not create duplicate examples.
5. Label independently and adjudicate. Use match, non-match and unresolved. Preserve initial labels, disagreement and final rationale. Escalate policy gaps rather than resolving them through reviewer intuition.
6. Partition before tuning. Separate training, validation and held-out test data by entity and, where useful, time. Hash and version the partitions. Do not move difficult failures from test into training without issuing a new evaluation version.
7. Document the dataset. Record motivation, composition, collection, preprocessing, authorised uses, limitations and maintenance, following the questions popularised by Datasheets for Datasets. Add change logs and named stewardship.

Measure the failures the business can feel

Report precision and recall for the link class, but do not stop there. Add false-merge and missed-link counts, unresolved coverage, candidate recall and entity-level cluster measures. Segment results by source, locale, data completeness and decision risk. Show confidence intervals or sample sizes so a sparse high-risk segment does not look certain merely because its displayed percentage has two decimal places.

Evaluate operations as well as the matcher. Measure reviewer disagreement, adjudication reasons, time to resolution and later label reversals. A model can appear stable while the human policy around it changes. Conversely, a fall in aggregate precision may come from a new source sending harder cases rather than a degraded matcher. The dataset’s source and time metadata let the team tell those explanations apart.

Treat the test set as a finite instrument, not permanent truth. Add newly observed failure modes through a controlled version, and retain the previous release for comparison. Revisit labels when authoritative evidence changes, but never overwrite history silently. The original decision, correction reason and effective version together explain why two evaluations differ.

Measure	What it exposes	Required slice
Link precision	False merges among accepted links.	Source, locale and consequence.
Link recall	True relationships the system missed.	Candidate generation and matcher separately.
Unresolved coverage	Cases where evidence cannot support a safe answer.	Reason code and decision workflow.
Entity topology	Over-combined or fragmented clusters.	Cluster size and high-impact entity class.

How does ground truth remain independent of the current matcher?

Keep the sampling reason, source mix, original values and candidate path with every labelled case. Otherwise a future team cannot tell whether a pair came from random sampling, a difficult stratum, a production review queue or a known error. Those routes expose different parts of the population and should not be blended into one accuracy figure.

A label also needs the guideline version, independent judgements, adjudicated outcome and evidence available at the time. The held-out partition is separated by real entity and locked by hash. This prevents one person’s records from leaking across train and test and stops a convenient relabel from improving the historical score.

When a rule or model changes, replay it against the same protected partition and report pair and entity outcomes by source and consequence. New cases can extend the dataset in a new version, but the old truth remains available so the team can distinguish genuine improvement from a changed test.

How do we connect labelled truth to Tilores rules?

We configure deterministic, fuzzy and probabilistic matchers as distinct methods. Rule simulation lets a team inspect how selected record pairs behave before deploying a configuration, while a labelled corpus supplies the broader evaluation needed to measure missed candidates, false links and entity-level changes.

Keep the original record values with the normalised features used by the matcher. Submit and query controlled records through GraphQL, record the exact rule bundle, and store the resulting pair and entity decisions beside the dataset version. That creates a reproducible path from source case to measured outcome.

Our end-to-end rollout guide places labelled truth before parallel running. For this dataset, the proof is that an untouched test partition survives successive releases and still exposes performance by source, difficulty and business consequence.

Frequently asked questions

How large should an entity-resolution ground-truth set be?

There is no universal number. Size follows the error rate, desired precision of estimates, number of important segments and model complexity. A smaller, well-stratified and consistently adjudicated set can be more useful than a large convenient export.

Can production matches be used as labels?

They can supply candidate cases, but the current system’s output is not independent truth. Reusing it as the label mainly measures agreement with the old system and preserves its blind spots.

Should unresolved examples stay in the dataset?

Yes. They measure coverage and expose where available evidence cannot support a safe decision. Excluding them makes real ambiguity disappear from evaluation.

What is the biggest source of leakage?

Records from the same real entity crossing training and test partitions is a common risk. Repeated use of the held-out set for threshold decisions also turns it into another validation set.

How often should labels be refreshed?

Refresh when sources, entity definitions, rules or important failure modes change. Use versioned additions and documented corrections rather than a calendar-based overwrite of the existing record.

Sources

1. UK Government: Quality assessment in data linkage. Precision, recall, training data, clerical review limits and linkage-error assessment.
2. Office for National Statistics: RIO data linkage pilot. Clerical review as a resource-intensive gold-standard method and linkage error analysis.
3. Gebru et al.: Datasheets for Datasets. Dataset motivation, composition, collection, preprocessing, uses and maintenance documentation.
4. NIST AI Risk Management Framework Core. Govern, map, measure and manage functions for evidence-led AI risk controls.
5. Tilores documentation: Matcher Reference. Deterministic, fuzzy and probabilistic matcher mechanics.
6. Tilores documentation: Rules Configuration. Product-specific rule, transformation, matcher and simulation behaviour.
7. Tilores: What an Entity Resolution Rollout Actually Looks Like in 2026. Internal context on labelled truth, parallel running and operational ownership.
8. Tilores documentation: API. GraphQL operations and the entity event stream.

Ground truth is bounded by its population, time and policy. Report those limits whenever a metric derived from the dataset is used.