Training Binary Deep Neural Networks Using Knowledge Distillation

Abstract

Binary networks can be used to speed up inference time and make image analysis possible on less powerful devices. When binarizing a network the accuracy drops. The thesis aimed to investigate how the accuracy of a binary network can be improved by using knowledge distillation. Three different knowledge distillation methods were tested for various network types. Additionally, different architectures of a residual block in ResNet were suggested and tested. Test on CIFAR10 showed an 1.5% increase in accuracy when using knowledge distillation and an increase of 1.1% when testing on ImageNet dataset. The results indicate that the suggested knowledge distillation method can improve the accuracy of a binary network. Further testing needs to be done to verify the results, especially longer training. However, there is great potential that knowledge distillation can be used to boost the accuracy of binary networks.