TiloRes vs Zentity for Entity Resolution: A Quick Comparison

The role of entity resolution is becoming increasingly critical in today’s age of “big data”. This is piling additional pressure on top of data teams that are being tasked with extracting more and more value (ad infinitum) from their complicated, bulky and, more often than not, unstructured datasets.

At the same time, these teams are arguably being somewhat held back by the limited capabilities of existing data tooling — that is, unless they can build their own, which is an unrealistic prospect for teams within the majority of firms. And by talking about tooling in the context of entity resolution, we’re of course referring to Elasticsearch.

Elasticsearch limitations

Elasticsearch is a distributed, open-source (under Apache 2.0) full-text search and analytics engine based on Apache Lucene. It is accessible via the RESTful API or the Elasticsearch Java API client and can be used to store, search, and analyse huge volumes of data and return answers in near to real-time due to its document-based structure. We recommend reading our previous blog post, Elasticsearch for entity resolution, if you want more background information on the basics of Elasticsearch.

Although the value of tools like Elasticsearch cannot be understated — it does serve a useful purpose after all, and it does this very well — it’s hard to ignore its scaling issues when it comes to entity resolution. This is unlikely to be an issue for smaller teams but once entity resolution demand grows, Elasticsearch can quickly run out of steam.

Due to this and the fact that Elasticsearch itself was not made for entity resolution, third-party tools have been built on top of it to get around its inherent limitations. One of these plugins is Zentity.

What is Zentity?

Zentity is an Elasticsearch plugin for real-time entity resolution,which operates on data that has already been indexed within Elasticsearch. It is based on Elasticsearch 100% and makes it easier for data teams to carry out entity resolution more easily and get their results faster.

Though it’s a powerful plugin, there are inherent limitations, just like with raw Elasticsearch that will be problematic for larger data teams that have more sophisticated entity resolution requirements.

That’s why we developed TiloRes, our scalable, serverless solution that eliminates many of the pain points associated with entity resolution by building up entities as data is ingested. When searches are subsequently performed, a comparison with all existing datasets is made, and any pre-built entity that the matching dataset belongs to is delivered as a search result. Since a dataset can only belong to one entity, a single search query can return multiple entities more quickly and with high accuracy.

TiloRes vs Zentity

Let’s take a quick look at how TiloRes stacks up against Zentity in terms of scalability, security, and performance.

Scalability

Scalability is arguably the most important consideration for any data team with a growth mindset in the so-called “big data” age. Datasets are inflating at a rapid pace; they are arriving in increasing volumes, contain greater variety, and are coming at us much more quickly.

This can mean that even smaller data teams may inadvertently find themselves growing at a pace they never anticipated. Scalability in tooling is therefore not so much an optional nicety as it is a necessity — and TiloRes delivers on this.

Zentity is 100% based onElasticsearch, which means that its own scalability is limited to Elasticsearch’s scalability. For single instances, entity resolution problems begin to pop up at around 1 million records. On the other hand, TiloRes’ scalability is unlimited because it’s serverless and based on demand; there is no limit to the amount of data that teams can simultaneously process. No matter if you process one request per second or ten thousand, the setup is the same and no manual intervention is needed for the system to scale. Scaling is automatic and requires no downtime or maintenance.

Performance & availability

As we mentioned earlier, Zentity’s performance will degrade as you scale because it’s based on Elasticsearch. Although it’s a fast, high-performance tool for low-volume entity resolution, most data teams will begin to experience serious performance issues which hinder their workflows at around 1 million records.

TiloRes on the other hand can scale from zero to an unlimited number of requests without any noticeable drops in performance. All components of TiloRes are always running on at least three different datacenters, which comes at no additional cost because the logic is only executed when requests are flowing in. In addition, Zentity only indexes records and must create entities during the search process whereas TiloRes matches records to entities. This means it’s possible to have different entities in a single instance and search all entities quickly. In other words, TiloRes builds the pathway to entities only once and remembers it, whereas Zentity must create the pathway each time, which is inefficient,

Cost

ElasticSearch is empirically more expensive to run than TiloRes because ElasticSearch runs on top of servers and nodes. These must always be on and cannot dynamically scale based on periods of increased demand. This leads to overprovisioned systems and a huge server requests-costs ratio, which is an issue for companies that are growing quickly because it can lead to downtime. In addition, the cost of administering and maintaining ElasticSearch is high, as experienced engineers are required for cluster setup.

With TiloRes serverless entity resolution, you only pay for what you use. During periods of low demand and low use, you’ll pay a comparatively small amount to times of peak demand and higher use. It all depends on how many requests have been made. In addition, TiloRes was built to run without the need for administration and maintenance, which eliminates costs further — it just works.

Conclusion

There’s no denying that the Elasticsearch engine coupled with Zentity is a powerful method for searching and analysing huge volumes of data in real-time. Equally, there’s no denying that with growth, Zentity’s ability to carry out entity resolution at the speed, scale, and security needed by larger data teams will begin to degrade.

At TiloTech, we help firms to overcome this challenge with our proprietary data solution, TiloRes: a serverless entity resolution technology. TiloRes offers super-fast searching, unlimited scaling, and real-time deduplication to speed up, simplify, and future-proof the entity resolution process.