What Causes Data Silos & How to Fix Them?

Connecting data with an understanding of human nature

Many people have a romantic view of a business as “one company indivisible”. They imagine today’s enterprise as an army of like minded individuals all working tirelessly towards the same goal, a thousand people in perfect synchronisation as a smoothly functioning hive mind. 

And anyone who runs a business knows … nothing could be further from the truth. 

A company is a population of individuals, inspired by their own motivations, following different goals, for different reasons. To the ultimate benefit of the company, of course – they want the business as a whole to stay successful. But in their day-to-day lives? The focus is on creating value and prestige for themselves and their closest colleagues. Aka “making a name for yourself”. 

This quirk of human behaviour isn’t a bug, it’s a feature. It’s basic division-of-labour theory, people doing what they’re good at so they can extract maximum value from their efforts. Your CFO has different KPIs to your janitorial team, and for excellent reasons.

Trouble is, chasing value has a dark side. It creates a tendency to hoard resources, build fiefdoms, erect defensive barriers around your own workflows. It’s called silo thinking. And while it’s just human nature, it’s problematic for those seeking a Big Picture view on all the data in their business. Because data tends to get sucked into those silos, too, isolated and hidden from view.

Let’s dive into the issues of data silos – and see how technology can solve them.

The data silo problem

A hiring application doesn’t bed in, so the HR team continues keeping most employee data on a single spreadsheet – making reports from the new app misleading. The sales team’s log of customer calls isn’t available to the marketing team that could make use of them, leading to established customers being treated as cold prospects. A merger or acquisition happens, and two CRM platforms continue operating independently because nobody has time to integrate the datasets … resulting in multiple versions of the truth. 

All these are data silo problems. Nobody really means to isolate datasets that’d be more useful if connected up. But the immediate concerns of people for control and responsibility tend to sideline such bigger-picture goals. Just as markets consist of sectors and companies competing for industry advantage, companies are made up of departments and individuals competing for personal advantage first. 

You might think the rise of cloud services, standardised metadata and APIs would make connecting these data silos easier. And in a strict technological sense, they do. But that’s a whole new work requirement, needing people to form new teams across departmental walls for no immediate benefit to themselves. 

Many organisations are hierarchical, without easy “horizontal routes” enabling relationships between departments. And who has time for things that don’t win you any plaudits with your boss?

The key thought here: data is about technology, but the problem of data silos is a very human one.

Why are data silos bad?

The pain points are fivefold. But you’ll see they’re cut from the same cloth: resources, waste, and missed opportunities.

**More IT infrastructure than needed. **Duplicated effort needs duplicated technology – and in large companies it’s not a small issue. Think of the energy bills of hungry servers, twice as much data centre acreage as needed, SaaS subscriptions for far too many seats across the business. 

**Management reports that don’t tell the truth. **Without a single version of the truth, board-level decision making takes place with incomplete information. How can the C-Suite set priorities and allocate resources when the picture of what the business is doing is lopsided?

Mistrust and suspicion between departments. When busy people are asked to collaborate on a goal, there’s a legitimate question of “What’s in it for me?” Especially in hierarchical org charts, the need to influence sideways to build consensus and understanding can slow execution to a crawl. 

**Differences in data quality. **Each team will focus on data quality only to the extent it answers their own KPIs. Missing data points vital to another department will make the dataset useless outside the silo even if it’s successfully shared.  

**Impact on the customer experience. **We’ve all had the jarring unpleasantness of Customer Service treating us like a new customer even though we have a 20-year history. When the data’s not joined up, service to the customer nosedives – a major churn factor in many sectors. 

Solving data silos

So: data silos exist, connecting them is a Hard Problem, and the result may not deliver the outcomes you want anyway. What’s the answer? At Tilores, it’s a technique called entity resolution.

Entity resolution is the process of “comparing” different datasets and seeing the connections between them using rules-based intelligence. The core thought here: doing so avoids the need to build brittle, one-to-one pathways between a database in one department and a database in another that need constant maintenance and updates. 

The Sales team may have one CRM profile of Jack Bell, with records of his purchases stretching back years. The Marketing department may have a J B Bell on its campaign list, with a different email address and records of what PDFs he’s downloaded. While Customer Service may have a Mr Jonathan Bell who phones in regularly each month. Connecting the disparate set of fields together across applications – or even a simple merge-and-purge – carries a lot of risks. What if they’re three different people?

This is where entity resolution comes in. Comparing the three records with each other in creative ways – perhaps seeing connections between the location of a web visitor’s IP address and his postcode, or discovering that times of web visits and service interactions coincide – uncovers insights that allow assumptions about identity. Find enough of them, and entity resolution can say with a high degree of certainty that all these Bells are the same person.

And once that’s established, you’ve brought different datasets together. Without the painstaking work of manually mapping fields from one record onto one of a different shape, with all the scope for errors that implies. 

Data silos are a problem. Human nature is another. But maybe – just maybe – entity resolution can solve both. 

CONCLUSION: don’t think data silos, think entity resolution

Entity resolution may sound a new concept, but in fact it’s solving problems like the silo’ing of data for thousands of companies. And because solutions like Tilores operate without exponential computing overhead – no fresh infrastructure needed – it can work at scale, teasing out deep patterns in inconsistent data to gather disparate data points into a coherent whole. 

Letting the Marketing team know which customers have history. The Customer Service desk understand a problem’s commonalities. The C-Suite make decisions based on complete and accurate information. And so on.