Resolving Identities as Events Happen: Entity Resolution in Streaming Pipelines
TL;DR: Streaming entity resolution consumes new and changed records from an event-driven pipeline, resolves and assembles each record at ingestion, and makes the current entity available to applications through GraphQL. Kafka or Kinesis carries the events; the identity layer maintains entity state; query time retrieves that state instead of rebuilding it.
Design the identity step in your event-driven architecture. Book a Demo or Try Tilores Studio (Free).
Event-driven systems are good at moving change. A customer signs up, edits an address, starts a transaction, opens a support case, or changes a company role. Kafka or Amazon Kinesis records the event and lets several consumers react.
Movement is not identity. If each consumer sees only the new record, it still has to know which person or company the event belongs to. A stable source record ID helps with replay, but it does not link records from different systems. A shared email can find candidates, but it does not prove they are the same person.
Streaming entity resolution fills that gap. It turns a flow of source records into a continuously maintained entity graph that downstream applications can query.
What is streaming entity resolution?
Streaming entity resolution is the continuous matching and assembly of records as new events enter the identity layer.
The event bus and identity layer have separate jobs. Apache Kafka or Amazon Kinesis Data Streams captures ordered records and delivers them to consumers. The identity consumer maps each source event into the configured Tilores schema and submits it through GraphQL. Tilores applies deterministic rules, fuzzy matching, and probabilistic scoring at ingestion, then creates, updates, merges, or splits entities as required.
Applications query the resulting entity by entity ID, source record ID, or approved search parameters. They are not asking the query service to compare every raw record again. The current resolved context already exists.
That boundary is the core architecture:
| Layer | Primary job | What it should not own |
|---|---|---|
| Source application | Emit a durable business change with a stable source record ID | Cross-system identity matching |
| Kafka or Kinesis | Transport, retain, partition, and replay events | Entity assembly |
| Identity adapter | Validate, map, and submit the source record through GraphQL | Business decisions |
| Tilores | Resolve and assemble entities at ingestion | MDM governance, KYC cases, or application workflow |
| Consuming service | Query current resolved context and act under its own policy | Rebuilding identity inside each request |
Where should entity resolution sit in a Kafka or Kinesis pipeline?
Entity resolution should sit behind a dedicated consumer or adapter that turns source events into one-record GraphQL submissions.
A common flow looks like this:
- An operational service writes a customer or company change to a topic or stream.
- The event carries a stable source record ID, source name, event time, schema version, and the allowed identity fields.
- An identity adapter consumes the event, validates its shape, and maps source fields into the Tilores schema.
- The adapter submits the record through the Tilores GraphQL API.
- Tilores resolves and assembles the entity at ingestion.
- Downstream services query the current entity context when a decision or interaction needs it.
Kafka topics are multi-producer and multi-subscriber. Kinesis separates producers from consumers in a similar way. That lets the identity adapter evolve without making the checkout, policy, CRM, or support system wait for every other consumer. It also lets teams replay a bounded set of events when mapping logic changes.
The adapter is an integration component, not a second matcher. It should not build its own hidden set of fuzzy rules before calling Tilores. Its job is to preserve source meaning, apply agreed transformations, and submit the record consistently.
What should each identity event contain?
Each identity event should contain enough metadata to process, replay, and trace one source record without pretending that the event itself is a resolved entity.
| Event field | Purpose | Design note |
|---|---|---|
| Source record ID | Identifies the record within its origin | Keep it stable across retries and updates |
| Source system | Preserves provenance | Do not merge namespaces from different systems |
| Event type | Distinguishes create, update, or deletion intent | Map each type to an explicit GraphQL operation |
| Event time | Records when the source change occurred | Keep it separate from processing time |
| Schema version | Lets consumers apply the correct mapping | Reject or quarantine unknown versions |
| Identity fields | Provides the approved name, contact, address, and identifier values | Minimise fields and follow data policy |
| Trace ID | Connects source, stream, ingestion, and query logs | Use it for operations, not as a match key |
Do not place several unrelated records into one GraphQL submit mutation just because the stream consumer received a batch. Tilores can accept an array of records, but records submitted together are automatically connected with a static edge. For ordinary event ingestion, submit one source record at a time unless that fixed connection is deliberate.
This is a subtle but important difference between transport batching and identity semantics. A Kafka consumer or Lambda function may receive several events in one poll. The adapter can still issue one record submission for each event.
How does a GraphQL submission update the entity?
A GraphQL submit mutation adds the source record, then Tilores applies the configured match and consistency rules during ingestion.
A minimal illustrative submission has this shape:
mutation {
submit(input: {
records: [{
id: "crm:customer:123"
email: "example@example.com"
mobile: "+441234567890"
}]
}) {
recordsAdded
}
}The available fields depend on the instance schema. A production adapter should validate that the record ID is a string, protect field types, reject unknown schema versions, and keep credentials outside the event payload. It should also handle the outcome explicitly rather than assuming that transport acknowledgement means identity assembly succeeded.
During ingestion, the record can create a new entity, update an existing entity, or cause existing entities to merge. A later correction or deletion can split or remove entity state. Tilores maintains the entity graph through those changes. Query time then retrieves the current result.
Teams preparing an initial historical load should separate that work from the live stream. The guide to ingesting large volumes into a GraphQL API explains why bulk load, retry, and throughput testing need their own path. Once the history is loaded, the stream keeps new and changed records current.
How should the pipeline handle retries and replay?
The pipeline should make retries safe by keeping source record IDs stable, tracking processing state, and defining how updates and deletions behave.
Event delivery systems can redeliver. Consumers can fail after a request succeeds but before they store the acknowledgement. Operators can replay a topic from an earlier offset. The adapter therefore needs a clear record identity and an observable state transition for every event.
At minimum, log the topic or stream position, source record ID, schema version, GraphQL operation, processing result, and trace ID. Send malformed records to a quarantine path with the exact validation error. Alert on sustained lag, repeated failures, and unexpected changes in create, update, merge, split, or delete volume.
Ordering deserves a policy too. Kafka guarantees order within a topic partition, and Kinesis preserves order within a shard. The key or partition choice should keep changes for the same source record together where practical. Cross-source entity changes still need to tolerate events arriving at different times because separate systems do not share one clock.
Replay should be tested before production. Load a short event sequence, record the resolved entity, replay the same source history under the intended process, and confirm that the resulting context remains correct.
How do downstream services learn that an entity changed?
Downstream services can react to Tilores entity events through AWS SQS or AWS Kinesis, or query the current entity when their workflow runs.
Our entity event stream publishes CREATE, UPDATE, MERGE, SPLIT, and DELETE events. The deployment can expose that stream through AWS SQS or AWS Kinesis. Those events let downstream caches, search indexes, warehouses, and workflow services update when entity state changes.
This gives the architecture two distinct streams:
- The inbound business stream carries source-record changes into the identity adapter.
- The outbound entity stream carries resolved-entity changes to authorised consumers.
Do not treat the outbound event as the full customer record by default. A small event with entity IDs and affected source record IDs can trigger a consumer to query the fields it is allowed to use. That reduces duplicated personal data and keeps field access in the GraphQL layer.
Tilores can therefore sit beside MDM, CDP, KYC and AML systems, warehouses, and operational stores. Those systems remain responsible for their own governance and workflows. Entity events tell them that the resolution state changed.
What does the application query after an event is resolved?
The application queries the current entity context through GraphQL after the ingestion path has updated it.
If the application already knows the entity ID, it can use the entity query. If it knows a source record ID, it can use entityByRecord. If it has approved search parameters, it can use search and inspect the returned hits and hit score. The response can include the entity ID, records, edges, duplicate information, hits, and an entity score.
The article on querying a unified customer profile in real time shows how this serving path works. The query retrieves current resolved context. It does not run a fresh entity assembly inside the request.
That distinction keeps the live application simpler. A support agent, fraud check, AI assistant, or policy service can ask for the entity it needs without reimplementing matching logic. The identity rules stay in one layer and can evolve without changing every consumer.
Is a Kafka consumer enough to make batch entity resolution real time?
No. Putting a batch matcher behind a stream does not make the identity state continuous.
If the consumer only appends events to a file and waits for a scheduled deduplication job, the serving path still sees a snapshot. If it compares the new record only with a limited cache, it can miss older or cross-source evidence. If it rebuilds the entity when queried, the query path inherits variable match work and inconsistent source coverage.
Real-time entity resolution means the record is resolved and assembled as part of ingestion, and the updated entity becomes available for a later query without waiting for a separate matching batch. Batch still has a valid role for historical backfills, reconciliation, and offline analysis.
The guide to real-time versus batch entity resolution gives teams a clean test: use real time when a live application must act on the current resolved record, and use batch when the data and decision can wait.
What should teams test before production?
Teams should test event semantics, identity accuracy, failure recovery, correction handling, and query freshness as separate concerns.
Use a sequence that creates a person, changes an email, adds a record from another source, introduces a difficult non-match, corrects a mistaken link, and deletes a source record. Confirm the expected entity after every event. Then stop the consumer, build lag, restart it, and verify recovery.
Test malformed events, unknown schema versions, duplicate delivery, reordered cross-source events, and a replay from a prior offset. Check that one transport batch does not create accidental static links. Confirm that merge and split events reach downstream consumers and that each consumer can query the new entity state.
Finally, measure the parts independently: stream lag, adapter processing time, ingestion completion, match quality, GraphQL query behaviour, and downstream update time. A fast event bus cannot compensate for weak identity rules, and a good matcher cannot compensate for an unreliable adapter.
FAQ
What is streaming entity resolution?
Streaming entity resolution matches and assembles records as new events are ingested, then keeps the entity context available for current GraphQL queries and downstream updates.
Does Tilores resolve identities when the application queries them?
No. Tilores resolves and assembles entities at ingestion. The application query retrieves the current resolved context that the identity layer already maintains.
Can Kafka send records to Tilores?
Yes, through an application-owned consumer or adapter that maps Kafka events into the configured Tilores schema and submits them through GraphQL. This describes an integration pattern, not a claim that Kafka itself performs entity resolution.
Can Amazon Kinesis fit the same pattern?
Yes. A Kinesis consumer can validate and submit source records through GraphQL. Tilores can also publish resolved entity change events through AWS Kinesis or AWS SQS, depending on deployment.
Should a consumer submit every polled event in one GraphQL array?
Not for ordinary independent records. Records submitted together in one Tilores submit mutation receive a static connection. Process each source record separately unless that fixed link is intentional.
How should the pipeline handle duplicate events?
Keep source record IDs stable, log stream positions and processing outcomes, and test replay. The adapter should define update and deletion semantics and verify the resulting entity state after redelivery.
Does streaming entity resolution replace MDM or a data warehouse?
No. It sits beside MDM, CDP, KYC and AML systems, warehouses, and operational stores. Those systems keep their governance, analytics, and workflow roles while receiving resolved entity context.
When is batch entity resolution still useful?
Batch is useful for historical loads, periodic reconciliation, warehouse preparation, and offline analysis where no live decision waits. Streaming is the better serving path when each new event can change what an application should do now.
See what resolved entity data does for your business — and your AI.