As organisations increasingly depend on data to fuel their analytics and AI initiatives, the challenge of bringing it all together and making it readily accessible has become a growing headache for a lot of them. Many businesses still face a fundamental problem: finding trusted, reliable data when they need it. Data sits in multiple systems, owned by different teams, each with its own rules and levels of governance.
As a result, people spend more time searching for the right dataset and figuring out whether they’re even allowed to use it than actually extracting value from it. Even highly specialised data scientists, whose core role is to model and innovate with data, are affected. Instead of focusing on analytics, they can spend up to 80% of their time simply locating, accessing and preparing data scattered across the organisation.
This inefficiency slows down decision-making, innovation and the business’s ability to respond quickly to new opportunities. And while many businesses have tried to overcome this challenge by building central data lakes and data warehouses, they may ultimately find that the original data should reside with teams who are experts in that data. They may have multiple data warehouses up and running, which they never decommission because there is value in them.
Read Full Article on Business Day
[paywall]
They may also need to have on-premise and cloud data warehouses, creating a more complex environment for businesses to navigate and use. Moving data into a central data lake is time-consuming and specialised work. Every dataset requires its own extract, transform and load flow, which must be built, tested and run.
When millions of records are involved, developers end up waiting for long loads to finish. Data teams lose hours to slow running pipelines, manual source-to-target verifications and repeated cycles of loading and reloading data just to accommodate small changes. It could take six to 12 months to build up a central data repository like a data warehouse or lake.
Another challenge in providing agile, business-ready data is that traditional centralised data architectures often become bottlenecks, with central teams struggling to keep up with the growing demand for data. These factors can hamper efforts to truly centralise data to streamline analytics and AI. A simpler solution to the challenge is to add a data virtualisation layer that brings together all the data users require without moving it.
[/paywall]
