Enterprises across several verticals – healthcare, telecoms and IT, energy and utilities, retail/ecommerce, pharmaceuticals, media and entertainment, manufacturing – are embracing data analytics and business intelligence (BI) in a big way, the addressable market size varying from MarketsandMarket’s projected $29.53 billion by 2019 to IDC’s projected $41.5 billion by 2018.
Understanding the scale and volume of BI/analytics
Corporate leaders are beginning to agree that BI tools are now the best game changer ever since the proliferation of the internet as there is a massive business need to transform terabytes of data into meaningful, actionable insights which can in turn, create new revenue streams. To put that into perspective, according to Frost & Sullivan 90% of data in the world was generated in the last two years alone.
To understand the sudden growth and evolution of this market, let’s consider the example of the energy industry. With utility companies announcing smart meter rollouts in their millions (see table), utility providers now have access to a massive data set on consumer behaviour, which according to research by the Edison Foundation, stands at more than 1 billion data points on a daily basis. Information bytes include wattage, demographics and even device-specific power consumption.
It goes without saying that capturing proactive intelligence on consumer behaviour can not only help utilities stretch the dollar through more frugal resource allocation, it also helps roll out innovative pricing models and tariff offers.
Why is data analytics/BI Big?
It has to be remembered that spreadsheets were around long before BI and analytical tools took off. Through in-built charts and pictorial representations, visualizing data was not exactly rocket science. However, considering the business demands of today, the business limitations of spreadsheet visualizations can no longer be ignored thus, necessitating the rapid inclusion of BI/analytical tools in the corporate decision-making landscape.
Some of these limitations are as follows:
The problem of data in motion: So far, corporations were used to dealing with static, stored data which could be collected from various sources before they were analyzed and interpreted for visual results. However, the rapid onset of large data volumes – log files, sensor data, meter readings, social media sentiments, clickstream information (“customer clicks on the website”) - means datasets are no longer expected to reside within a central server or within a fixed place in the cloud. In fact, the very nature of big data refers to business information that is streaming in across hundreds of thousands of disconnected sources, rapidly evolving by the minute and making it nearly impossible for organizations to keep track of spontaneous information bytes. Traditional methods to analyze these information patterns are inadequate giving rise to tools like MongoDB which can process and store billions of bytes of real time data, with hundreds of thousands of transactions per second.
The problem of unstructured data: Analyzing data is a rather simple affair when all data sources collect information based on unified file formats. However, the biggest challenge facing enterprises is the undefined and unpredictable nature of data emerging in multiple formats. Unstructured data can fall into any one of these categories - textual, non-textual, audio, video, presentations, pictures and .rar files. Going hand in hand with the problem of data in motion, it’s nearly impossible to keep track of information formats emerging from multiple sources. This has fueled the rapid emergence of data aggregation tools like Hadoop which process and convert unstructured data into structured, readily-usable, unified formats.
Aesthetics – the need for interactive visualization: Converting unstructured data into structured formats is only half the battle won. An informed business decision can only be taken when the data can be readily converted into relevant charts and pictorial depictions that make “real sense” of all the information pouring in. Accurate, aesthetically-pleasing visuals go a long way in understanding the value of the business data and “consume” it for business decision-making. Data visualization tools like Tableau, Spotfire, Qlikview, etc. have rapidly come to the fore to fill this market need.
The increasing role of predictive analytics in the energy space
One of the biggest catalysts in the big data revolution is the growing significance of predictive analytics. Enterprises having access to the largest source of business data and predictive analytics helps them understand existing patterns in a better way and convert that information into meaningful business value.
In layman’s terms, predictive analytics isn’t very different from weather forecasts based on existing meteorological information. Let us return to the example of energy and utility players. As discussed, despite access to widely available data coming from various sources, they don’t have the knowledge set to convert the data into meaningful business intelligence. For instance, according to the Utility Analytics Institute, each year rebates worth millions of dollars are not communicated in time to the customers because most utilities struggle to keep track of information emerging at the customer end.
A predictive analytics system, hosted either on cloud servers or local servers, can drive the necessary logical, computational and statistical algorithms to analyze large volumes of event data, e.g. correlation, data clustering, regression analysis. At eInfochips, we have designed a comprehensive BI/analytics cycle specifically for energy and utility players, as represented in the above diagram.
Data collection is the first stage of such a framework. Utility companies receive source side data through sensor integration at various points of the energy flow matrix. On the distribution side, each and every transformer, substation and electric pole is fitted with an intelligent electronic device which transfers data to a wireless mesh network. Smart meter records can be added to this mix including fire/smoke alarms, floods, weather-driven network outage, voltage dip/short circuit/overload, peak demand, temperature control for HVAC (heating, ventilation and air conditioning) and smart home alarms.
The next stage is data analysis. The data is brought together in one place and normalized so it can be used by various applications across multiple vendors and types of devices. The analyzed data is then delivered to two types of applications: facility data applications and business intelligence tools for further processing. Predictive analytics tools like MongoDB and reporting tools like Tableau enable pattern creation on the demand side with an aim to improve overall distribution.
In the data visualization and reporting stage, the analytics system should improve interactivity through a graphical, map-based, high-level overview of the whole real estate portfolio along with the ability to drill down to individual regions or facilities for more focused views of performance. To enable prioritized response, the Operations Dashboard organizes performance information in prioritized order so that it’s easy to see which efforts will have the greatest impact tied to specific goals and with the least amount of effort. In order to automate action, the Operations Dashboard offers a simple workflow for responding to events and alerts. Alarms and failures have to be mapped very precisely using a meter analysis tool application which includes a collection of visualization widgets to explore and uncover irregularities within energy and other metered utilities. Demand patterns have to be created based on real-time data integration with weather and geographical locations.