Entity Resolution Review Queues: How to Design Human Decisions for Ambiguous Matches in 2026

TL;DR: Route a candidate to human review when the evidence is genuinely ambiguous and the cost of an automatic mistake justifies the delay. Give the reviewer original and normalised values, competing candidates, rule-level evidence, source provenance and the consequence of each choice. Record link, non-link or unresolved as a versioned decision, then measure overturns and downstream corrections rather than celebrating queue throughput alone.

Test your matching configuration on a controlled data extract. Book a Demo · Try Tilores Studio (Free).

On this page

1. Which matches should enter a review queue?
2. Why should the queue follow consequence, not confidence alone?
3. What must the reviewer actually see?
4. A seven-step implementation sequence
5. How does the queue keep each judgement inspectable?
6. How do we support review without hiding the policy decision?
7. FAQ
8. Sources

Which matches should enter a review queue?

A candidate belongs in review when three conditions meet. The available evidence supports more than one plausible outcome. An automatic error would affect a meaningful business decision. A person has enough context and authority to decide, defer or request better data. If any of those conditions is missing, the queue becomes theatre: expensive manual work without a clearer decision.

This creates three legitimate states rather than two. A supported link can be accepted automatically. A supported non-link can remain separate. An ambiguous pair can stay unresolved until policy, evidence or time changes. The third state matters because forcing every record pair into match or non-match converts uncertainty into hidden error. In high-consequence workflows, an explicit unresolved outcome is often the more accurate operational result.

The Office for National Statistics describes linkage systems with an upper threshold for links, a lower threshold for non-links and an intermediate band for clerical review. That is a sound starting pattern, not a universal threshold. The band should be calibrated against the cost of false merges, missed links and review capacity for the decision at hand.

Why should the queue follow consequence, not confidence alone?

A score does not contain the business consequence of being wrong. A pair with moderate evidence may be safe to leave separate in an analytics segment but unsafe to merge before a payment, account closure or access decision. Route policy therefore needs two inputs: how ambiguous the identity evidence is and what the consumer will do with the answer. The same pair can legitimately receive different treatment in two authorised workflows.

Do not use a single global band merely because it is easy to configure. Segment policies by decision class, source reliability and evidence pattern. A stable government-issued identifier may support automation under one policy. A similar name and shared address may need review when households or common names make that combination weak. Missing required evidence should route to data repair, not to a reviewer asked to guess.

A useful queue preserves the path from candidate evidence to an outcome that can later be explained or changed.

What must the reviewer actually see?

Show the pair in context, not as two flattened profiles. A reviewer should see original source values beside the normalised values used for comparison, the source and timestamp of each value, the rules or score components that created the candidate, and any competing entities. If a field conflicts, that conflict should be prominent. If a value was absent, the interface must not render it as agreement.

Explain the consequence in plain language. ‘Merge these records’ is incomplete if the action will also move an account, alter a customer view or trigger a downstream eligibility decision. Reviewers need to know whether they are approving an identity relationship, correcting source data or authorising a business action. Combining those questions in one button makes later audit and reversal needlessly difficult.

The reviewer also needs an escape route. Permit an unresolved decision, escalation and request for source correction. Require reason codes that are specific enough to learn from but not so numerous that people choose at random. Free text is useful for exceptional detail, yet a small stable reason taxonomy is what allows the team to detect repeated data gaps and policy confusion.

A seven-step implementation sequence

1. Name the protected decision. Write down what a false merge and a missed link would cause in this workflow. Agree which outcomes can be reversed automatically and which require another owner. This prevents an abstract confidence threshold from silently becoming business policy.
2. Build a reviewed calibration set. Sample easy links, easy non-links and difficult boundary cases from the real source mix. Keep adjudicated examples outside day-to-day queue clearing so threshold work is measured against stable evidence rather than the latest reviewer habit.
3. Define routing rules. Combine evidence bands with consequence, source quality and required fields. Keep the policy versioned. A rule such as ‘send every score below 0.9 to review’ is not enough unless 0.9 has meaning for the chosen matcher, population and error cost.
4. Design the evidence view. Present originals, normalised values, provenance, conflicts, competing candidates and the active rule or model version. Hide irrelevant attributes so the reviewer is not influenced by data that policy does not permit for the decision.
5. Separate decision from execution. Record the review outcome first, then let an authorised service apply the entity change and notify consumers. This makes retry, rollback and audit possible when a downstream update fails.
6. Measure quality, not speed alone. Track queue age and capacity, but also agreement between reviewers, re-review overturns, later reversals, reason-code distribution and downstream corrections. Fast clearance can coexist with systematically poor decisions.
7. Close the learning loop carefully. Use reviewed outcomes to investigate source quality and recalibrate rules or models. Do not treat every human click as unquestionable training truth. Reviewers can disagree, policy can change and the queue itself can select a biased slice of cases.

How does the queue keep each judgement inspectable?

Keep the evidence that caused a case to enter review. That includes the candidate path, original and normalised values, source times, competing entities and the active rule or model version. A reviewer should be able to distinguish an exact identifier agreement from a fuzzy candidate or a probabilistic score instead of receiving one unexplained confidence value.

The decision record then adds the selected outcome, the reviewer’s reason, the policy version and any expiry or escalation trigger. That record lets a later reviewer reconstruct the choice and measure whether similar cases are repeatedly overturned. It also prevents an approved link from losing the uncertainty and consequence that made human judgement necessary.

Queue evidence must follow the outcome. A link, non-link or unresolved decision can change an entity and several downstream views. Preserve the source record IDs and event time so a correction can be applied to the affected references rather than treated as a note that only exists in the review tool.

How do we support review without hiding the policy decision?

We model records as nodes and matching relationships as edges, and our rules can combine deterministic, fuzzy and probabilistic methods. Rule simulation helps teams inspect intended behaviour before a configuration reaches live data. The rule configuration reference carries the product detail; the review policy still belongs to the team making the consequential decision.

We submit, search and query resolved data through GraphQL. A review service can use that boundary to retrieve current entity context, record its authorised judgement and keep the source evidence needed for correction. It should not flatten a review into a permanent profile field or expose more attributes than the reviewer needs.

Our end-to-end rollout guide places labelled truth, ownership and parallel running around the matcher. For a queue, the decisive proof is narrower: ambiguous cases reach an authorised person, their reasons survive, and downstream corrections converge after an outcome changes.

Frequently asked questions

Is human review only for probabilistic matching?

No. A deterministic rule can use the wrong field semantics, and a fuzzy candidate may be supported by strong independent evidence. Review is triggered by ambiguity and consequence, not by the name of the matching method.

Should every low-confidence pair be reviewed?

No. Many can safely remain separate, and some should be returned for data repair. Review only when a person can add authorised context and the decision value justifies the cost.

What is a good review-queue success metric?

Use a balanced set: time to decision, inter-reviewer agreement, unresolved rate, re-review overturns, later reversals and downstream corrections. Throughput alone rewards premature decisions.

Can reviewer decisions train a model?

They can contribute labelled examples after adjudication and bias checks. A queue is selected by existing rules, so its cases are not automatically representative of the whole population.

How should a team handle reviewer disagreement?

Keep both initial decisions, route the case to adjudication and record the final rationale. Repeated disagreement is evidence that the policy, interface or data contract needs clarification.

Sources

1. Office for National Statistics: RIO data linkage pilot. Clerical review as a resource-intensive gold-standard method and linkage error analysis.
2. Office for National Statistics: Developing standard tools for data linkage. Threshold bands and ambiguous pairs routed to clerical review.
3. UK Government: Quality assessment in data linkage. Precision, recall, training data, clerical review limits and linkage-error assessment.
4. Information Commissioner’s Office: Human review. Reviewer training, overrides, sampling, tolerances and documentation.
5. Tilores documentation: Rules Configuration. Product-specific rule, transformation, matcher and simulation behaviour.
6. Tilores documentation: Matcher Reference. Deterministic, fuzzy and probabilistic matcher mechanics.
7. Tilores: What an Entity Resolution Rollout Actually Looks Like in 2026. Internal context on labelled truth, parallel running and operational ownership.
8. Tilores documentation: API. GraphQL operations and the entity event stream.

Scope note: this guide covers entity-resolution operating design. It does not replace sector-specific legal, compliance or employment guidance.