Entity Resolution Next to the Lakehouse: Where the Engine Should Actually Run
TL;DR: Put entity resolution beside the lakehouse, not inside a chain of analytical transformations and not inside every consuming application. The lakehouse remains the system for durable analytical data. A separate operational engine resolves records as they arrive, gives each row a stable entity ID and serves the current entity through GraphQL.
Place identity where your applications can use it. Review the architecture with our team or test the engine locally before choosing a production perimeter. Book a Demo or Try Tilores Studio (Free).
On this page
- Where should entity resolution sit?
- Why does warehouse-only resolution become stale?
- What does the engine need?
- What changes with an entity ID?
- How do you keep it current?
- Which deployment model fits?
- Frequently asked questions
A lakehouse can store every customer record an organisation has ever seen and still fail to tell an application which records belong to the same customer now. Storage and identity are related, but they are not the same job.
The placement decision matters because identity resolution sits on a boundary between analytical and operational work. Put it too deep inside warehouse transformations and every new record waits for a load and a model run. Put it inside each application and several incompatible customer views emerge. Put it too far from the data perimeter and the architecture creates movement that governance teams did not intend.
The useful pattern is to run a dedicated entity resolution engine next to the warehouse or lakehouse. Let it consume the records needed for identity, return a stable entity ID to the analytical rows and expose the current resolved entity to authorised applications.
Where should entity resolution sit relative to your warehouse or lakehouse?
Entity resolution should sit beside the lakehouse as a shared operational service. The lakehouse remains responsible for durable storage, transformation, analytical models and historical queries. The resolution engine is responsible for deciding which source records describe the same real-world entity and for keeping that entity current as records change.
This boundary avoids two common extremes. The first is resolving only inside SQL or Spark jobs. That can work for periodic analytics, but the resolved view is only as current as the last successful run. The second is embedding matching logic in a CRM, support application or AI service. That produces a local answer, but other systems cannot rely on the same identity.
In the adjacent pattern, each source record is sent to the resolution engine when it becomes available. The engine applies deterministic, fuzzy and probabilistic matching, updates the entity and returns or exposes an entity ID. The ingestion pipeline writes that ID into the lakehouse alongside the source row. Analytical models can then group history by a durable identity without owning the match logic.
Operational applications query the current entity through GraphQL. They do not wait for a reverse ETL cycle to discover that an address or telephone number changed. The current view can respond at approximately 150ms in the managed path and approximately 1ms when self-hosted, subject to network and workload conditions.
The entity ID should be treated as a governed reference, not as a replacement for source keys. Keep the source system, source record ID and ingestion timestamp. That lineage lets engineers reconstruct how a row entered an entity and lets deletion or correction travel back to the right record.
Why does resolving inside the warehouse break at ingestion speed?
Warehouse-native resolution does not inherently break. It becomes unsuitable when the required freshness is shorter than the transformation cycle or when operational applications need a current entity between loads.
That distinction is visible in the platforms themselves. Snowflake Dynamic Tables express freshness as a target lag rather than a guarantee that every operational change is resolved immediately. Databricks Lakeflow pipelines coordinate streaming tables and materialised views, but they remain data-pipeline constructs. Both can move identity inputs and outputs well. Neither removes the need to decide where the current operational entity is maintained.
For event-driven ingestion, Apache Kafka’s design documentation makes the complementary point: durable ordered logs let consumers process changes independently. The identity engine can consume that change stream without making the warehouse transformation graph responsible for the current entity.
Consider a support agent dealing with an account takeover. A new telephone number arrives through the CRM at 10:03. The warehouse loads at 10:30 and the entity model completes at 10:50. Until then, a customer-service tool that reads the warehouse sees the old profile. A twenty-minute delay may be harmless for monthly reporting and unacceptable for an interaction taking place now.
The same gap appears in event-driven systems. A payment event, device signal or sanctions-screening update may need to change the operational entity immediately. If resolution is expressed as a sequence of analytical joins, the team must either trigger expensive recomputation for every event or accept a stale answer.
There is also a modelling problem. A warehouse transformation normally reads a bounded snapshot and writes a new table. Entity resolution has state. A new record may join an existing entity, split a previous cluster after a correction, or reorganise the profile after a record is deleted. Rebuilding the entire population can obscure which decision changed and why.
None of this means warehouse tools should be removed from the flow. Snowflake and Databricks remain strong places to prepare source data, analyse error patterns and measure the downstream effect of resolution. Splink and Zingg can be valuable when the job is exploratory or batch oriented. AWS Entity Resolution can fit teams whose scope and data flow align with its managed model. The question is whether the production application needs an operational identity service or an analytical output.
What does the engine need from your platform?
Tilores needs three infrastructure primitives: a key-value store, a queue and file storage. On AWS, the reference implementation uses DynamoDB, SQS and S3.
The key-value store maintains the current state needed to locate and update entities. The queue absorbs and orders ingestion work. File storage supports imports, exports and other durable objects. The engine does not introduce a fourth mandatory datastore, a dedicated search cluster or a separate graph database into the platform.
This small primitive set is what makes the adjacent placement portable. Managed Cloud on AWS is the default production model for most teams. The same engine can run in Your AWS Account when infrastructure control or data perimeter requires it. Another Cloud and On-Premise are the remaining deployment models. The application contract stays GraphQL across the four models.
The platform team needs to provide more than infrastructure. It must define source ownership, change capture and deletion. For every source, record:
- which fields may contribute to matching;
- whether a value is trusted, observed or user supplied;
- how corrections and deletions are represented;
- which system remains authoritative for the original record;
- who may query the resolved entity;
- where the returned entity ID will be stored.
Normalisation belongs at a deliberate boundary. Tilores normalises and persists values used by the engine while retaining originals. Source-specific cleaning that reflects business semantics can happen before ingestion. Generic identity comparison should remain in the resolution configuration, where it is visible and testable.
What changes when every row carries an entity id?
An entity ID turns cross-system analysis from a repeated matching exercise into an ordinary join. It does not make every dataset correct, but it gives each authorised row a consistent reference to the same resolved subject.
The effect is easiest to see in a placement table.
| Layer | What it owns | What it receives | What it must not assume |
|---|---|---|---|
| Source system | Original record and business workflow | Optional resolved entity ID | That local duplicates have disappeared |
| Ingestion and streaming | Change capture, mapping and delivery | Entity ID or entity update | That retries may create new identities |
| Entity resolution | Matching, current entity, lineage and entity ID | Source records and corrections | That it owns consent or business truth outside matching |
| Lakehouse | History, models, analytics and feature data | Stable entity ID on each relevant row | That the latest analytical load is the current operational profile |
| Reverse ETL and activation | Movement to destinations | Governed segments keyed by entity | That every destination may see every source attribute |
| Operational application | Customer interaction and decision | Current authorised entity through GraphQL | That a resolved link grants permission to use all linked data |
Analysts can compute lifetime value across commerce and support records without embedding name and address similarity in every model. Product teams can define features against a stable entity. AI teams can retrieve one authorised customer context instead of asking a model to infer identity from document fragments.
There are governance consequences. An entity ID can become a powerful join key, so access should be scoped. The fact that two records resolve to the same person does not mean every business unit or application may see both. Apply attribute-level and source-level controls at the query and consumption layers.
Entity IDs also need lifecycle rules. Downstream tables should tolerate an entity changing after a correction. Keep a mapping or event history so a team can understand whether two IDs merged, one entity split, or a record moved. Do not overwrite history in a way that makes yesterday’s report impossible to reproduce.
How do you keep the resolved view current between loads?
Use the record change as the trigger. When a source creates, updates or deletes a relevant record, publish that event through the ingestion path. The queue gives the engine a reliable unit of work. Resolution updates the operational entity, and downstream consumers can receive the new entity ID or query the current view.
For batch-only sources, use incremental extracts and explicit watermarks. A daily source can coexist with real-time sources, but the freshness contract must be visible. A profile should say that the CRM changed seconds ago while a legacy billing source was last refreshed overnight.
Idempotency is essential. A retried event must update the same source record rather than create a duplicate. Use the source name and source record ID as a durable identity for ingestion. Preserve the event time and processing time separately so late events can be understood.
Reconciliation still belongs in the lakehouse. Run a periodic comparison between expected source counts, ingested records and resolved outputs. Inspect entities that grow unusually quickly and records that remain unlinked. The analytical environment is well suited to this monitoring even when it is not the operational matching engine.
Corrections and deletions need first-class paths. If a source retracts a telephone number or an approved retention rule removes a record, the engine must reorganise the entity rather than leave a stale link. Tilores supports retention rules per source and logs queries, exports and deletions. The controller remains responsible for deciding what must be retained or erased.
Which deployment model fits a lakehouse team?
Start with the data perimeter and operating model, then choose from the four deployment options in order.
Managed Cloud on AWS fits teams that want the quickest supported production route and do not need to operate the underlying primitives. It is the default answer for most organisations, including those whose lakehouse is elsewhere, provided approved data can cross that boundary.
Your AWS Account fits teams that need the engine and data inside their own AWS perimeter while keeping the AWS reference architecture. The platform team owns more of the account controls and integration surface.
Another Cloud fits an organisation whose approved perimeter is in a different cloud. The same three primitive categories are mapped to that environment. The GraphQL contract and matching behaviour remain consistent.
On-Premise fits a perimeter that cannot use a cloud deployment. It is a supported control option, not the default recommendation or a shorthand for greater security.
| Decision condition | Best starting model | Placement beside the lakehouse |
|---|---|---|
| Team wants the lowest operating burden | Managed Cloud on AWS | Stream or batch the identity fields to the managed service and return entity IDs |
| Data and controls must remain in the customer AWS account | Your AWS Account | Run the engine in the same approved account boundary as the lakehouse integrations |
| Approved data perimeter is another cloud | Another Cloud | Place the primitives close to the lakehouse without naming a vendor-specific architecture |
| Network is isolated from cloud services | On-Premise | Run the engine inside the isolated perimeter and exchange only approved outputs |
The right answer is not automatically the environment where the warehouse happens to run. It is the model that minimises unnecessary movement while keeping one operational identity available to every authorised consumer. The deployment options let the engine move to that perimeter instead of forcing the data estate into a new one.
Place resolution next to the lakehouse and the division of labour becomes clear. The lakehouse remembers what happened. The entity engine decides which records belong together now. The entity ID connects the two without asking either system to become the other.
Frequently Asked Questions
Should entity resolution run inside Snowflake or Databricks?
Entity resolution should usually run beside Snowflake or Databricks when applications need a current identity between analytical refreshes. The lakehouse keeps history and analysis; the adjacent engine keeps the operational entity current.
What data should return to the lakehouse after entity resolution?
The lakehouse should receive a stable entity ID, source lineage and any governed resolution events needed for analysis. It should not need to reproduce the matching logic in every model.
How do applications query the current resolved customer?
Applications query the current Tilores entity through GraphQL. They can request only the authorised fields and linked records needed for that workflow.
Can the engine run in the same perimeter as the lakehouse?
Tilores can run as Managed Cloud on AWS, in the customer’s AWS account, in another approved cloud, or on-premise. Choose the first model that satisfies the data perimeter and operating responsibility.
Does an entity ID grant access to every linked attribute?
An entity ID does not grant access to every attribute. It is a governed reference; source-level and attribute-level permissions still determine what each consumer may retrieve.
See what resolved entity data does for your business — and your AI.