Smart Grid on the edge – Managing Data Everywhere but the Cloud

Our third post of this series will outline key characteristics to consider for edge-based data processing—addressing security and service level considerations, and addressing data management volume and extensibility of services.


Direct contact between information systems and layers 0 and 1 of industrial networks must be managed very carefully—if allowed at all. This “control plane” (aforementioned layers 0 and 1), in most brownfield scenarios, has limited or no capability for authentication and arbitration of external commands from broadly connected IT/IP systems. Directly put, these systems were not designed for dynamic IT application service demands, and this creates myriad security concerns that are not easily addressed from a retrofit perspective.

Given legitimate risks, executing control decisions at the edge will not be appropriate in many scenarios. Instead, gathering inputs and triggering calculated decisions based on high-fidelity, low-latency edge processing can be a practical first step toward advanced edge-based decision-making. Trace events, identified through local edge processing, can evaluate far higher–fidelity, real-time data streams and arrive at partial or whole decisions without latency and cost-inducing bulk data transfers to remote platforms. A summarized decision can then be efficiently forwarded to a centrally managed system that has appropriate resources for authentication and arbitration of control system inputs.

Service Level

Bringing analytical services closer to the source of machine data at the edge also brings new service level demands and functional requirements. Dynamic empirical data requests, broad enterprise access, and Internet-compatible IP-based access are part-and-parcel with digital IoT initiatives. These service requirements bring high CPU, memory, and storage demands to an industrial edge currently defined by low-power, rugged, and embedded systems. Inevitably, more resources will be demanded at the edge to feed concurrent, high-bandwidth analytic requests from a broad spectrum of stakeholders. This is not a service level that aligns with needs or design of typical industrial control systems. Modern edge-based information systems will need to arbitrate resulting service level conflicts.


Data processing features that reduce transfer volume represent a tide that lifts all boats at the edge. Reducing volume directly mitigates both cost and latency. Furthermore, analytical processing at the edge can preserve fidelity by processing against highly granular data prior to transfer. This provides the benefits of high-fidelity data analysis without the cost and impracticality associated with transferring and storing often-repetitive time-series machine data.

Edge-based data volume management also directly implies the ability to handle large data volumes on the latest rugged, solid-state, fanless computers, alongside assets and well beyond a typical data center environment. Keeping highly granular—but often-repetitive—data with low immediate value at the edge saves cost beyond backhaul transfer rates. It alleviates the need to store low-value data on high-cost central IT or cloud data management platforms, but simultaneously preserves the information and maintains availability if events of interest mandate a closer look.


Edge data processing should intrinsically provide data services that deliver extensible, dynamic data access aligned with broader IT managed application development and services. The established “stream-forward” alternative to an edge-based “store-forward” data architecture provides an illustrative example regarding the concept of extensibility at the edge. Embedding data summarization within a telemetry stream (consider this “advanced stream-forward”) can enable limited but effective local summarization at higher fidelity. Specifically, an embedded algorithm could cache sensor reads every minute for 30 minutes and then calculate mean and variance for transmission in an embedded data- logging device.

The downside is that this approach creates a rigid data model that does not easily accommodate change. For example, what happens when analytical requirements evolve? For an embedded system the answer is replacement, which can be quite costly and impractical in an industrial environment. If the algorithm resides as managed firmware on a local compute device, separate from the data stream, an OTA update is possible but still not entirely practical. There is an additional requirement that is often missed: Underlying the application-based model there must be a local data store that also does not take a prefixed view of data ingestion to complete the dynamic information system. Without the ability to change the data model, only relatively minor tweaks to embedded algorithms are feasible.

In summary, we have defined the role of edge data management as a dynamic information-brokering service. This broker will arbitrate conflicts between security and service level disparities, between critical industrial control systems, and more open IT-centric analytics and application development. Backstopping this will be new technologies, built for industrial environments, that provide local store-forward data services with the ability to deliver any view of all data at the edge. These systems will necessarily support dynamic service levels and concurrent use, against the constant challenge of frequently changing application and analytical demands. Perhaps then we will see solutions catch up to the broad, and inspiring, promise of IoT—and the grid may lead the way.

About the author: Eric Kraemer is the CTO and cofounder of Exara, Inc., with more than 15 years’ experience working with high-performance information systems and software development, both in the US and internationally.

Exara is an Intel Partner that has developed the world’s first edge data services platform for the Industrial IoT. Exara software is designed for Intel®-based edge gateways that can be deployed in substations, electrical plants, and many other industrial environments. For more information, please visit Exara's website.