In recent years, we’ve seen streaming analytics—the ability to derive real-time insights from large pools of data—go from serving specific use cases (such as trading and retail recommendation engines) to gaining much broader appeal. Part of the reason why is because of technology advances: Connectivity is faster, edge devices are smarter, and, of course, core processing has become significantly more powerful than in the past. At the same time, business needs have evolved: From being limited to the weekly reporting cycle, executives increasingly demand a more immediate picture of how the business is doing.
At the heart of any streaming analytics solution is the stream of data (yes, the clue is in the name, differentiating from ‘batch’ analytics, in which data is collected and analyzed in batches). This needs to deliver the right data from the right set of sources, with sufficient speed to make it all worthwhile. Getting this right means working within a number of constraints: Bandwidth, storage, and processing resources are greater than before, but are still finite. This means getting a number of elements right, in addition to the usual challenges of bandwidth and processing:
- Edge sensors and smart devices need to be configured to send the right data in the right way. For example, a temperature sensor could be required to deliver a second-by-second feed of absolute values, or it could be enough to indicate when the temperature has dropped below a certain level.
- Gateway devices also serve an important role, in managing the data that will be passed to analytics. It may not be practical to send everything, though. For example, a retail store could be collating point-of-sale (PoS) and inventory data alongside live video. Gateways can compress, cache, and otherwise pre-process data for sending.
- Technologies like Apache Spark* and Flink* are used to champion streaming analytics in scale out cluster computing environments. In-memory computing and databases (e.g., SAP HANA*) are enabling In-memory, real-time analytics and taking this solution space to the next level.
By configuring these elements, engineers, developers, and data specialists can provide the basis for real-time insights to meet a wide variety of use cases. Note, however, this is not a one-way trip. An organization may decide one insight is more important than others—for example, hospital management may see the number of available beds as a priority metric, but may find that a small increase in hospital stays significantly reduces costly re-admissions. Insights such as these will influence both the data models being used and the data that needs to be collected. This also drives new operational models, in this case around patient discharging and outpatient processes.
Streaming analytics enables still-deeper levels of insight to be reached, based on real-time data. Consider, for example, the lowly weather report. We all know how it can be considered as reasonably accurate for the next 24 hours, then can be seen as a guide for a few days—beyond that, more of a cause for optimism or pessimism. Imagine how profound the impact on the enterprise could be were we able to trust the insights about corporate operations to this extent. Many organizations still operate rear-view reporting cycles and are looking to benefit from more direct understanding about their business, the view of what is happening now.
What if, like the weather forecast, you could see what was likely to happen over the next hours, days, and weeks? Some industries are already extending their field of vision in this way, such as credit card companies gaining the ability to predict the potential for fraud. In other industries, such as retail and utilities, manufacturing, and logistics, the potential for predictive business is clear, both to increase efficiency and enable better customer experiences.
While the future of streaming analytics creates clear potential for business differentiation and competitive advantage, delivering on such aspirations is dependent on whether business decision makers trust the insights they have been given. This is why we advise our customers to start on the streaming analytics journey with use cases that are higher priority, well-scoped, and which are likely to deliver clear business value. Organizations can build on such scenarios to increase skills, develop credibility, and grow demand for streaming analytics across the organization.
To understand where to start on the journey to streaming analytics, download the whitepaper.