Exploring the Benefits of Compression to Improve Data-intensive Neural Network Inference

Abstract

Convolutional Neural Networks (CNNs) have become essential for many computer vision tasks such as image classification, object detection and semantic segmentation. State-of-the-art models come with high computational demands that require efficient acceleration. FPGAs are a popular choice, as they can host energy-efficient, high-throughput inference designs. Streaming dataflow designs are particularly efficient, as they create custom per-layer pipelines which confine most of the data movement on chip. As these accelerators also prefer to keep model parameters on chip, on-chip buffer space becomes a limiting factor. Long skip connections, a ubiquitous structure of today’s networks, aggravate this problem further, as they require large amounts of buffer space to synchronize the diverging branches of the model’s architecture. Off-chip eviction instead moves the activations of skip connections temporarily to off-chip memory, but this puts additional load on the memory bus, which can lead to stalls in the compute pipeline if the processing engines cannot be fed. This problem can be alleviated using lossless compression, but lossless compression is fundamentally limited. This thesis explores lossy compression of the evicted activations instead, trading a bounded loss of inference accuracy for a higher compression ratio, without retraining. Starting from SZ1, a prediction-based compressor whose model fits the streaming paradigm, we develop ASZ (Activation SZ) in a design space exploration, which adds block-level error thresholds, new prediction functions, per-channel selection of the prediction functions, and Huffman coding and downsampling stages. We evaluate ASZ against lossless (Huffman coding, RLE) and lossy baselines (quantization with Huffman coding (QH), Lorenzo prediction, SZ1, downsampling, SVD and JPEG) on the 74 skip connections of eight CNN classifiers on ImageNet, CIFAR-10 and CIFAR-100, and estimate the hardware cost of every method. Lossy compression goes well beyond lossless encoding: on every connection, a lossy method exceeds the best lossless one, on 71 of them at a cost of at most 0 1 pp of top-1 accuracy. Within a budget of 1 pp, the streaming-compatible compressors QH, Lorenzo and ASZ reach geometric-mean compression ratios of 3 9, 3 8 and 3 5, 2 3 to 2 6 times the 1 5 of Huffman coding, while RLE does not compress on average. ASZ improves on SZ1 (2 7), but trails the quantizers on most connections, since SZ1-type compressors spend their bits on the choice of a prediction function rather than on the values or their residuals. JPEG reaches 6 6, but buffers eight rows per channel and needs multipliers on the data path. When several connections are compressed, their errors add up and the ratio within the budget falls, least for the error-bounded Lorenzo and SZ1; with three connections, the best lossy method still stays 1 5 to 4 3 times above Huffman coding. Lossy compression can thus substantially reduce the off-chip traffic of evicted skip connections, and QH, the cheapest lossy compressor and the best streaming-compatible one in ratio, offers the best trade-off between compression ratio and hardware cost.