What We Learned Building a Public Splink Benchmark

TL;DR: We published a reproducible benchmark comparing our own entity-resolution API against open-source Splink, including the proprietary rule config most vendors keep hidden. The headline number (F1 0.9949 vs 0.8867) matters less than the method: same dataset, same ground truth, published code, and an honest account of where Splink wins. The most interesting finding was architectural: the +19.4pp recall gap traces back to Splink’s blocking step, not to tuning. This post is about how we built a benchmark we’d trust if a competitor had run it.

See it on your data: Book a Demo to walk through real-time entity resolution on your own records, or Get the Evaluation Build to try it locally.

Why benchmark your own product in public?

Vendor benchmarks have a credibility problem, and we know it. The default assumption, usually correct, is that a benchmark published by a vendor was reverse-engineered from the conclusion: pick the dataset, pick the metric, tune until you win, publish. So why do it at all?

Because the alternative is worse. Practitioners evaluating entity resolution ask a concrete question. I’m using Splink for deduplication; when, if ever, does it make sense to switch? The answer is “it depends, here’s the data,” and we’d rather give it as a reproducible artefact than a slide deck. Splink is the strongest open-source baseline available: built inside the UK Ministry of Justice, probabilistic, explainable, and free. If we couldn’t stand behind a fair comparison against it, we shouldn’t be making accuracy claims at all.

The discipline we set ourselves was to build the benchmark so that someone who wanted to disprove it could try, with everything they’d need in their hands. That constraint shaped every decision that followed.

What makes a benchmark trustworthy rather than marketing?

A benchmark earns trust by being checkable, not by being flattering. We landed on five principles, and we’d hold a competitor to the same ones.

Principle	What it means in practice
Same inputs	Both tools see the identical dataset and the identical ground-truth pairs. No per-tool dataset, no sampling.
Published config	Every rule, threshold, and parameter is in the repo — including ours, which is proprietary.
Reproducible from scratch	A deterministic generator (seed 42) means anyone regenerates the exact same 1M records and re-runs both tools.
Where the other tool wins	We state plainly that Splink wins on batch throughput, on cost, and on per-pair explainability.
A path to be proven wrong	Open an issue if the method is flawed; open a pull request if you can beat a score on the same data.

The last two are the ones that separate a benchmark from a brochure. A comparison that never lets the competitor win anything is asserting a result, not measuring one. Splink wins clearly on several axes, and saying so is what makes the recall number believable when we report it.

Why publish the proprietary rule config?

This was the decision with the most internal debate. The Tilores linking rules (which fields combine, which comparers and distance thresholds fire a match) are the kind of thing a product company normally keeps to itself. We published them anyway.

The reasoning: a benchmark where one side’s configuration is hidden is unverifiable by construction. If we’d shipped our results with the rule config redacted, every number would have rested on “trust us.” Publishing it means a sceptical reader can see exactly why a pair linked, reproduce it, and challenge it. The config isn’t the moat. The real-time API, the persistent entity graph, and the operational behaviour around it are. A rule list on a synthetic dataset gives away far less than people assume, and it buys the one thing a vendor benchmark can’t otherwise have: the ability to be checked.

It also forced honesty on us. You write rules very differently when you know they’ll be read line by line by people who’d enjoy finding a thumb on the scale.

What was the most interesting technical insight?

The headline gap was recall: Tilores found 19,345 more true duplicates out of 100,000 pairs (+19.4pp), which drove F1 to 0.9949 versus Splink’s 0.8867. Precision was effectively tied, since both tools are cautious about false matches. The interesting part wasn’t that the gap existed. It was where it came from, because it isn’t a tuning artefact.

Splink is a probabilistic record-linkage system, and like every system in that family it relies on a blocking step to keep the candidate-pair count tractable. Blocking is load-bearing: two records that never land in the same blocking bucket cannot be compared, no matter how similar their fields are. On our corrupted data, a duplicate whose blocking fields (say, both postcode and last name) are corrupted becomes a structural false negative that never even enters the candidate set. Blocking is fast and necessary, and it’s also where most of the recall is lost.

Tilores takes a different path. It evaluates overlapping linking rules over phonetic and string-distance comparisons, and merges matches transitively into a persistent entity graph. There’s no blocking gate that can pre-emptively exclude a pair: if any single rule fires (name + date of birth, or name + city, or postcode), the records link. Corruption in one field doesn’t block a match when another field stays clean. The cost is that you enumerate the rules yourself; the benefit is exactly the recall you see in the results.

That’s the insight we’d want a reader to take away even if they never touch our product. In this class of problem, the recall ceiling is often set by the architecture, blocking versus overlapping rules over a persistent graph, long before it’s set by parameter tuning. You can push Splink’s recall up with more blocking keys, but each key enlarges the candidate space and slows the run. It’s a real trade-off, and a fair benchmark should name it rather than hide it.

What did building it the honest way cost us?

We accepted three caveats on the record. Our published Splink config is closer to tutorial-style, and a Splink expert could likely recover another 1–2pp of F1 with custom blocking and m/u-value tuning. We say so in the methodology, and if someone submits a stronger config as a pull request, we re-run and update. The dataset is synthetic: reproducible and honest about its construction, but not a stand-in for your messy production data with its OCR errors, transliteration noise, and locale quirks. And we deliberately do not present Tilores as a Splink replacement. They solve overlapping but distinct problems. Splink produces a one-off pair list at batch time; Tilores is a real-time entity-resolution API that resolves records at ingestion and returns the resolved entity at query time (8.6ms p95 on the 1M-record index here), sitting next to your MDM, CDP, warehouse, or KYC stack rather than replacing them.

None of those caveats help the marketing. All of them help the credibility, which on a benchmark is the currency that matters.

What would we tell someone building their own benchmark?

Publish enough that you could be proven wrong: code, data generator, ground truth, and configs for both sides, no exceptions. Report where you lose, because a benchmark with no losing column isn’t a measurement. And separate the number from the architecture. The most useful finding in ours wasn’t the F1 score. It was understanding that blocking versus overlapping-rules-on-a-graph is what moved recall. If your benchmark only produces a leaderboard and not an explanation, you’ve built marketing, not engineering.

Everything is in the public repository at github.com/tilotech/tilores-splink-benchmark: the scripts, the deterministic data generator, the ground truth, our rule config, and pre-computed results for both tools. The full results write-up lives in our Splink vs. Tilores benchmark post.

FAQ

Why would a vendor publish a benchmark against its own product?

To answer a question practitioners actually ask — when to move from Splink to a real-time API — with a checkable artefact instead of a sales claim. The benchmark is built so a sceptic can reproduce and challenge every number, which is the only way a vendor comparison earns trust.

Why did Tilores publish its proprietary rule configuration?

Because a benchmark where one side’s config is hidden is unverifiable by construction. Publishing the rules lets a reader see exactly why each pair linked and reproduce it. The defensible value is the real-time API and persistent entity graph, not a rule list on a synthetic dataset.

What was the key technical finding?

The +19.4pp recall gap is architectural, not a tuning artefact. Splink’s blocking step excludes pairs whose blocking fields are both corrupted, making them structural false negatives. Tilores uses overlapping rules over a persistent graph with no blocking gate, so a single clean field can still link a pair.

Does the benchmark admit where Splink wins?

Yes — explicitly. Splink wins on raw batch throughput (1M records in 9.24 seconds as an embedded DuckDB library), on cost (it’s free), and on native per-pair probabilistic explainability. A benchmark with no losing column for the author is a brochure, not a measurement.

Where can I see the benchmark and check it myself?

The code, deterministic data generator, ground truth, Tilores rule config, and pre-computed results are all in the public repository at github.com/tilotech/tilores-splink-benchmark. The full results are in the Splink vs. Tilores benchmark post.