150 Million Records, 99.5% Accuracy, 36 Hours: How Tilores Transformed Customer Data at Enterprise Scale
For a major European last-mile logistics company, customer data had become a structural liability. The business shipped parcels on behalf of dozens of retailers — Amazon, eBay, and a mix of regional platforms — each delivering customer records in a different format, under different naming conventions, with different quality standards.
The result was a customer database that had grown to 150 million records, with no reliable way to know how many of those records represented real, unique people.
The problem at scale
When a customer orders through Amazon, eBay, and a retailer's own platform in the same month, they typically appear as three separate records in the logistics operator's systems. Each retailer formats name and address data differently. eBay masks email addresses with proxy formats. Some platforms include mobile numbers; others don't. Address formatting varies by integration.
At low volumes, this kind of duplication is manageable. At 150 million records, it was degrading every downstream operation:
- Delivery accuracy was compromised by inconsistent address data across records for the same customer
- Customer service couldn't answer first-contact queries because representatives were looking at fragments, not complete histories
- Fraud detection was limited to single-source patterns — cross-retailer fraud was invisible
- Personalisation was based on partial customer histories, producing weak relevance signals
Previous deduplication tools had been evaluated. None could handle the combination of volume, cross-retailer complexity, and real-time operational requirements.
What made this dataset difficult
Three characteristics made this dataset unusually challenging.
Masked eBay email addresses. eBay proxies buyer emails — the format buyer@members.ebay.com reveals nothing about the actual customer. Any deduplication system that relies on email as a primary match key fails immediately on eBay-sourced records. The matching logic has to work without it.
Address format inconsistency. Each retailer integration produced addresses in its own format. "Müllerstraße 12" from one source, "Müllerstr. 12" from another, "Muller Str 12" from a third. Exact-match rules miss all of these. The resolution system needed address normalisation before matching — not after.
Scale and accuracy in combination. At 150 million records, even a 1% false positive rate produces 1.5 million phantom merges — customers who appear to be the same person but aren't. The same rate on 100,000 records is a tolerable data quality issue. At this volume, it breaks operational processes.
The Tilores approach
Tilores processes records through a three-stage pipeline: Extract, Transform, Match (ETM).
The Extract stage ingests raw records in any format. The Transform stage normalises them before the matching engine sees them — addresses to a canonical form, phone numbers to E.164, name variants resolved to a standard encoding. For this dataset, ten specialised matchers were configured to handle the specific formats produced by each retailer integration, including masked eBay addresses.
The Match stage then applies ten matching rules against normalised data. These rules are split into two sets with distinct functions:
Linking rules handle high-confidence merges — cases where two records almost certainly represent the same person. These combine multiple strong signals: normalised phone plus fuzzy name match plus normalised address, for example. Address alone is insufficient and blocked by the rule linter.
Search rules handle weak-signal connections — cases where records share a characteristic worth surfacing, but not strong enough to merge. Household members at the same address are linked through search rules, not linking rules. This distinction prevents the most common failure mode at scale: over-merging, where related but distinct people are collapsed into one phantom entity.
Every merge produces an auditable edge recording which rule fired and on what fields. The decision is queryable and reversible.
Processing 150 million records in 36 hours
The full initial load completed in 36 hours — approximately 8 milliseconds per record. Tilores runs serverless on AWS Lambda and DynamoDB, deployed inside the customer's own VPC. There is no in-memory wall, no capacity planning required, and no downtime window.
Before the full load, a 1,000-record sample was run and manually reviewed. The review surfaced 377 data quality issues — malformed addresses, encoding problems, and records with contradictory field values. These were addressed in the ETM normalisation rules before the full load ran. This kind of early-signal sampling is standard practice in Tilores deployments and consistently catches the issues that would otherwise surface as false positives at scale.
The results
Post-load validation confirmed 99.5% accuracy across the resolved dataset.
150 million records were resolved to 38.1 million golden customer profiles — a 3.9:1 deduplication ratio. Each golden profile is the canonical representation of a real customer, assembled from all source records that matched across any combination of retailers.
The 38.1 million profiles are queryable in real time via GraphQL. A delivery system can look up a customer at order time and receive the canonical address, full cross-retailer order history, and household relationship data in a single API call — in under 10 milliseconds.
Beyond deduplication
The golden profiles unlocked capabilities that were not possible with fragmented data.
Household resolution. Search rules surface all customers at a shared delivery address as separate entities. The logistics operator can now identify household delivery preferences — preferred time slots, safe-place instructions, combined order history — without merging family members into a single record.
Cross-retailer fraud detection. Patterns invisible within a single retailer's data become visible when records are resolved across sources. Multiple accounts linked to the same address, contradictory identity fields, and volume anomalies now surface as queryable entity patterns.
AI and LLM integration. Deduplicated, normalised, household-linked profiles provide clean input for predictive routing, proactive exception handling, and personalised delivery communication. The entity resolution layer handles the data quality; the AI systems consume the output.
Timeline
The project moved from initial schema mapping to production in three weeks. The first week covered retailer feed schema review and ETM rule configuration. The second covered rule testing, sample validation, and data quality remediation. The third covered the full 150-million-record load, post-load accuracy validation, and handover to production API endpoints.
This timeline reflects a consistent pattern in Tilores deployments: the majority of implementation effort is schema configuration and data quality investigation, not infrastructure work. Tilores adapts to customer data formats; customers do not need to transform their data to match Tilores.
See what resolved entity data does for your business — and your AI.