Data is the new currency in this day and age. We talked about insights on all data assets in our previous blog and talked about the technology capabilities available that are enablers. In this day and age, consumers have a lot of choices and information available literally at their fingertips, which allow them to make decisions in sub-seconds. This makes speed of insights along with insights across all data a critical component for organizations in this age of digital transformation.
There are multiple factors to focus on when we talk about speed of insights, including:
- Report execution – the time it takes to get the data back to present in the appropriate format
- Speed to Market for Information – the time it takes for the data to be converted and enabled in the appropriate structure and the time it takes for new data sets to be enabled for analysis
- Data availability – The frequency and timeliness of data for enabling companies to make rapid decisions in keeping with the dynamically evolving consumer / customer needs
Report execution is the most frequently citied challenge by end users. The challenge has evolved since the days of the classic data warehousing when query performance and ways to optimize the performance were a major focus. Increasing data volumes, multiple data sets, increased analytical complexity and visual reporting becoming more common place have further contributed to longer running reports.
In the early days of data warehousing, we were predominantly focused on preparing data that ended up in flat reports that were generally scheduled, distributed and consumed on daily, weekly and monthly cycles. Those days are gone. The traditional performance optimization activities such as pre-aggregation, pre-population of queries, indexing, faster disks, etc. will only get you so far.
Further, even the creative ways to speed up the reports were only true for the defined set of data. Straying from the data set caused the performance to significantly deteriorate. And even if reporting tools could connect to various data sets the report was as fast as the slowest data source. This made them highly inflexible. The insight needs of today are not as batch oriented, and as a result far less predictable and needing much more flexibility.
Although most of the BI tools you choose to use for presentation and visualizations do not speed up or slow down your data technically, they do have an impact on the speed at which the insight is presented to your key decision makers, and perhaps more importantly, absorbed and understood by your key decision makers. If they don’t see the data in a way that makes sense to them, is visually understandable, how can they make quick business decisions? This has become clearer as senior executives have more and more information to process daily.
Speed to Market for Information
When raw data is converted into useful business insights, that’s where the real results happen. This is, however, not as simple as stated. In the classic and earlier days of data warehousing, this was where a lot of focus was for not only structuring the data into the information, but also for performance tuning and optimization. The multiple layers of transformation and summarization, the movement of only immediate and focused data sets were all attempts to improve the speed of insights via the query or reporting tools layer.
This however, also was limiting and organizations had to spend time and effort for any changes in data sets or to leverage new data assets. Often by the time these new needs for insights were delivered, the relevance had passed or there was a new set of information needs. This caused a constant cycle of IT development and was also the cause of the frequently voiced concerns of data warehouses being limiting and projects for reporting taking too long etc.
Further, being able to form new relationships and links between data sets and viewing information slices from different angles is critical for organizations to stay ahead of the competition. This makes the speed to market for new information needs critical.
With the information needs of today, not having the flexibility to analyze across all data assets and with the speed of insights in not an option. This can only be achieved if there is data available at the right level of detail, at the right frequency, consistently and accurately to keep pace with the dynamic nature of the business environment.
Adding to this is the need for not just as-is analysis, but to-be and what could-be analysis, which requires larger data sets not only in the type of data but also the duration of the data. This data now needs to be converted to information and be quickly presented at the appropriate level of detail and in the right format and also be enabled for dynamic conversion of data sets for mining new insights.
Fortunately for us today, there are tools and technology which enable this as a standard process versus needing to creatively enable this capability and potentially stressing the systems, which adds overhead in their own way with maintenance and upkeep and could also have business impact.
Is There a Solution for Speed of Insights?
Is it possible to have a data warehouse that can speed the consumption of traditional transactional sources as well as being able to load real time streaming and unstructured data without grinding to a halt?
Can you have a data platform that is able to scale and leverage modern, high power (yet still commodity) hardware to accelerate the preparation of the data so that it can be made available for querying as soon as possible?
Is there a way to evolve your data warehouse in order to adopt new, innovative technologies that have become available in recent times, to support new business workloads such as Predictive and Advanced Analytics?
Would it be possible to replace your existing aging, and frankly struggling data warehouse, without facing prohibitive costs or high barriers to accessing new technical skills? The answer is, yes (of course).
The Modern Analytics Platform
The Modern Analytics Platform delivers on all the requirements for a next generation data warehouse. Enabling organizations to radically simplify their existing legacy or overly complex solutions in order to lower running costs, improve agility and gain breakthrough performance to deliver real business value. Speed is a key factor in the success of any analytics initiative, and a Modern Analytics Platform must be fast.
Remember to follow the rest of the posts in this blog series where we will explore in detail the two remaining common challenges of traditional data warehousing; real-time data and data management.
To find out how the Modern Analytics Platform addresses the challenges of speed of insights, watch the video, The Modern Analytics Platform, to learn more!