The Modern Data Platform – Challenge 1 – INFLEXIBILITY
In the first part of this blog series I discussed the traditional data warehouse and some of the issues that can occur in the period following the initial project.
I concluded by outlining the 5 common challenges that I hear from customers.
- INFLEXIBILITY
- COMPLEX ARCHITECTURE
- SLOW PERFORMANCE
- OLD TECHNOLOGY
- LACK OF GOVERNANCE
This blog will focus on the first of these issues, INFLEXIBILITY.
For the avoidance of any doubt, let’s begin with a definition of INFLEXIBILITY.
“not permitting change or variation; unalterable”
Perhaps a little more final than I had in mind, however it is clear that we are talking about a solution that does not readily lend itself to being changed and varied.
Let’s begin by considering the 3 most common reasons why we may need to change our existing data warehouse.
- Request to amend an existing data model
- Something has changed in the source system(s)
- Requirement to add a new source data set
Each of these reasons bring their own particular set of challenges that should not be underestimated. However, it is also the case that they have a tendency to hunt in packs. In other words what may start as a simple request to amend a data model may end up leading to new data sources being required.
When the original data models / star schemas / cubes were designed it is likely that they were derived from a set of reporting requirements.
Good practice dictates that some kind of report decomposition matrix (RDM) would have been used. The RDM would catalogue all the data items and then collate them into subject oriented fact tables with corresponding conforming dimensions.
You would hope that the analysts involved in these discussions were able to take a subject oriented view that was wider than the report itself, and that they used modelling tools that supported quick changes, version comparisons and the understanding of relationships to support impact and dependency analysis. But that is unfortunately not always the case.
What if the analysis was not complete or thorough enough?
What if something was missed?
What if the business has a new or shifting focus?
What if the new data we need isn’t in a database, held within our organisation, or, what if the data isn’t conventionally structured at all?
And of course there is always a chance that the business users were simply not able to see into the future and predict all their requirements.
The techniques for designing and developing data warehouses evolved against relatively slow moving structured transactional databases. We were predominantly delivering reporting and drill / slice and dice analysis against reasonably static datasets where intraday reporting requirements were less common.
But that is not the case now. We live in turbulent unpredictable times. Decisions often need to be made “in the moment”. Mergers and acquisitions are rife. New and interesting datasets are made available both inside and outside of our organisations. These are the times of BIG DATA.
The app economy and the consumerisation of IT has led to an information rich culture and an expectation that we can access the data we need as and when we need it.
This demand for data and reporting agility can run up against the complexity of the data warehouse.
Changes to the Business Intelligence requirements could easily mean that each of the layers need to be individually assessed and amended. This is often not a simple task, potentially crossing many individuals and skill sets. Not a recipe for quick responses from the already no doubt overworked and underappreciated data warehouse team (if the organisation has one).
Simply put, there are too may components and complex layers requiring an ETL person, data modeller, a BI person…etc all requiring multiple different tools.
Often this leads to a perception amongst users that their only path to insight is a concrete tunnel that always leads to the same destination.
So what’s the answer?
Can we provide a streamlined, efficient and ultimately more agile data supply chain within our organisations?
Can we address changing Business Intelligence and Analytics requirements without getting lost in an endless cycle of legacy code inspection and the resulting regression testing?
The answer is, yes (of course).
The Modern Data Platform
The Modern Data Platform delivers a foundation for agile modelling, delivery and iteration that rapidly shortens the time to deliver changes, as a result lowering the cost of ownership of your Business Intelligence and Analytics solutions.
Remember to follow the rest of the posts in this blog series where we will explore in detail the four remaining common challenges of traditional data warehousing.
To find out how the Modern Data Platform addresses the INFLEXIBILITY challenge download our webinar recording