Streaming Analytics with Provenance in the Advanced Metering Infrastructure

Abstract

The Advanced Metering Infrastructure (AMI) allows electricity to be monitored and billed with fine-grained resolution through the use of smart meters, whose automated measurement results in vast amounts of generated data. Traditionally, this is handled by temporarily storing the sensed data in databases and then performing some analysis, for example, with the goal of finding faulty equipment. After some time, when the data are no longer deemed useful, it can be discarded to make room for more up-to-date measurements.

The amount of data is expected to increase further in the near future due to trends in the industry and national regulations. This presents a challenge to utility companies as database storage becomes more expensive. Motivated by this challenge, utilities can greatly reduce storage requirements by leveraging the stream processing paradigm to analyze unbounded and continuous streams of meter data as soon as it arrives. However, discarding too much data can be counterproductive, since maintaining the associated source data of an analysis result is beneficial to understanding its cause through further analysis. This is enabled by fine-grained data provenance, which connects each detected event with the source data that contribute to it.

Provenance captures have an intrinsic computational overhead associated with them. This thesis aims to give insight into what type of streaming analysis is feasible in the AMI using a state-of-the-art provenance capture. This is done by implementing and evaluating the performance of a number of streaming queries with and without provenance. The thesis was done in close cooperation with the Swedish utility company Göteborg Energi and therefore the queries use historical data from a realworld AMI.

The results show varied performance on a per-query basis, generally displaying an increase of latency between 24–42% and a decrease in throughput between 22–34% compared to its non-provenance counterparts. However, one query shows a more significant degradation, where the overhead reaches above 200% in latency and memory consumption. Despite the overhead, stream processing with provenance is a viable alternative for analysis in the AMI because it has the potential to greatly reduce storage requirements of meter data.