Technical report detail

Pruning Highway Networks For Reducing Memory Usage

by Dhananjay Tomar

In recent years, deep learning has gained a lot of popularity for achieving state-of-the-art results in various tasks like image classification, face recognition, machine translation, speech recognition, etc. This success, however, has come with models getting bigger in size. As they get bigger, they also get computationally more expensive. For example, the ResNet-152 model which won the ILSVRC 2015 image classification task has a size of about 240 MB and requires 11.3 billion FLOPS to classify one image. This prevents them from being used in mobile devices which tend to have much less memory and computational resources. In this work, we propose a novel approach to reduce the size of highway networks (a type of neural network architecture which has been shown to work well in practice) by pruning their nodes. We test our approach on a highway network trained on the CIFAR-10 dataset. Our approach results in reducing the size of the network by 85% (34 MB to 5.2 MB) while incurring a 1.72% loss in accuracy.

Technical report 2017/05, July 2017

BibTex entry

@techreport{17pruning, author = {Dhananjay Tomar}, title = {Pruning Highway Networks For Reducing Memory Usage}, institution = {University of Lugano}, number = {2017/05}, year = 2017, month = jul }