In the first part of this blog series I discussed some of the traditional challenges faced by analytics teams over the years, including insights on all data, and how they have evolved to what they are today. They are essentially the same challenges but with a greater or wider definition and meaning.
I concluded by outlining the four common challenges that I hear from customers:
- Insights on all data assets – Structured or unstructured
- Speed of insights – analyse data as and when needed for performance in sub seconds
- Real time – transactional or strategic data analysis
- Data management – even more critical
This blog will focus on the first of these issues, insights on all data assets, both structured and unstructured.
Myriad of Tools Available
Not too long ago Business Intelligence (BI), mainly reporting and analytics, started becoming an increased focus area, which only got magnified with the flood of new BI tools in the market vying for the attention of end users with “eye candy” sales plays and with increased efforts to make the tool agnostic to be able to interface with various sources of data.
At that time, the dominant BI tool was still excel, and in the SAP world Bex for those with business warehouses or ABAP reports dumping data into excel. Yes, Crystal Reports, xcelsius dash boards, etc. were still used in some organizations with varying degree, but not as pervasive as it is today.
BI Begins to Mature
The maturity in BI was starting to evolve. Analysts were starting to drive efficiency into their capability to automate manual reporting or expand reporting by making small-scale changes in bringing together data to optimize the reporting on transactional data. The visual factor was one of the components, but it was not the main driver at that time. The underlying need was to be able to analyze and gain insights across various sources of data.
Business warehouse (BW) started playing a key role in bringing these data sets together. Those in Consumer Packaged Goods (CPG) and Retail sectors would remember the creative ways they came up with to capture and store Point-of-Sale (POS) data and reporting with transactional data to better manage the shelf, etc. This brought about challenges with data movement of large volumes of data, storing large volumes of data, and summarizations to enable reporting on such large volumes of data.
New Challenges Arise
These changes drove discussions on storage costs and virtualizations to bolster the hardware and processing power on the infrastructure side. On the application side, these drove discussions on the ETL windows and efficiencies. The warehouses had to be architected creatively at the database level with table spaces, for example, and summarizations at the application level with aggregations and included as part of the ETL to pre-run or pre-populate reports to improve performance.
As the value and importance of insights across these key data sets started to be realized, the demand for data and information started to grow. The systems and architectures were stretched to expand to this growing need. In the SAP world, we had a warehouse accelerator which was developed to speed up the end product, namely the reports. But that brought its own set of challenges on the ETL side and the inflexibility of going beyond the defined data sets.
Overall, this added to the many layers of data summarizations and storage costs as well. BI projects developed a reputation for taking far too long and being inflexible. The technology had restrictions in capabilities and cost implications which further added to the challenge.
Fast Forward to More Recent Times
The industry as a whole has made great progress in terms of technology capabilities in being able to process and store data. But these had significant cost implications. While it resolved the challenges of performance of the output and ETL window, the costs were staggering for those capabilities and the storage of the large volumes of data. Those could be creatively circumvented but then they added complexity to the data model and landscape. But the value of the insights was significant enough to warrant those decisions.
However, as has been the trend, this just drove the expectations and needs higher.
Fast Forward to Today
In this day and age, analytics has continued to grow in relevance and need. The cost for storage, memory, and hardware in general has continued to see a downward spiral. Cloud continues to grow as it breaks free of its misconceptions and with increased efficiency to be more mainstream. The volume, type, and detail of data continues to grow exponentially. All the components of reduced hardware costs, increased capabilities, availability of cloud, exploding data environment and appetite for analytics has opened up the market for new capabilities and business drivers. Data analysts and companies are no longer driving the needs but the expectations are being set by the consumers like you and I and everyone in the country and across the globe. We are making decisions in sub-seconds and our loyalties are easily transferable based on the speed of service, access to information, and availability of goods.
Data lakes is a concept which enables some of these insights to be able to react and respond as per the users expectations and also to get smarter in being able to pre-empt their expectations. When you think back to years before and even during the time accelerators were making a presence, essentially the analysts were requesting and BI teams were using the tools available at their disposal to essentially get to the concept of data lakes. Multi-providers and universes were created to be able to tie data or analyze data across and down. The level expected was obviously different.
With platforms like Hadoop enabling data lakes and massive data processing to provide deeper insights, they are becoming more main stream. In the SAP world, HANA the database has emerged as a leader with in-memory technology. The applications since the start of the HANA journey have also evolved and kept pace with the dynamic environment. HANA is quickly becoming mainstream. Why stage or summarize data when you can glean better insights from the raw data. Now you can chose what slice of data you want readily available in real-time, quickly and what data needs to be mined and churned for different and unique insights. Now we can pipe data from those Hadoop data lakes and enable fast real-time insights tied to our business data for example.
These are very exciting times. We are at a point where we are not limited by the tool and technology but by our thinking. It’s given a whole new meaning to analytics. Now let’s go find those hidden gems…
Where Do We Go From Here?
The Modern Data Platform delivers on all the requirements for a next generation analytical capability. Enabling organizations to radically simplify their existing legacy or overly complex solutions in order to lower running costs, improve agility and gain breakthrough performance to deliver real business value. Insights are key to using data properly. A vast pool of data without gaining any insights or business value from it is just a waste.
Remember to follow the rest of the posts in this blog series where we will explore in detail the three remaining common challenges of traditional data warehousing; speed of insights, real-time data and data management. To find out how the Modern Data Platform addresses the challenges of insights across your data, attend our webinar, The Modern Data Platform, on Thursday, April 13th at 2 PM EST, 1 PM CST, 11 AM PST to learn more!