Attention-based Time Series Forecasting with Limited Data

Abstract

Electricity outages are common in electrical power systems, and often caused by natural phenomena, human intervention, or faults in electrical components, such as transformers. A small number of these faults can be predicted by analysing the stream of voltage and current. Forecasting faults in electrical power systems can prevent electricity outages that cause production downtime and capital losses. However, data collected in power systems are usually limited and unbalanced because of the very few historical predictable faults. This study focused on evaluating more recently popular attention-based machine learning models for time series prediction in electrical power systems, in a context where data is a significant limitation. The data was real and consisted of disturbances recorded from power systems over sev eral years, along with documented faults. Two different model architectures were evaluated and compared: the Long short-term memory (LSTM) and the transformer. Three different model instances were trained: using features manually extracted from each disturbance recording, using manually extracted features with pre-training on a similar dataset, and using a signal embedding pipeline attached to each model processing raw waveforms. The results from all six training instances showed that the transformer performed better than the LSTM in terms of evaluation metrics, although the LSTM outputs were more interpretable, because the transformer had higher confidence in its outputs even during false predictions. A bottleneck was found in the small sequence lengths, with improvement shown when utilizing pre training on a similar dataset containing longer sequences. The integrated waveform feature embedding also showed improvement over the manually extracted features.