NodeDrop: a method for finding sufficient network architecture size
Files
Accepted manuscript
Date
2020-07
Authors
Jensen, Louis
Harer, Jacob
Chin, Sang
Version
Accepted manuscript
OA Version
Citation
L. Jensen, J. Harer, S. Chin. 2020. "NodeDrop: A Method for Finding Sufficient Network Architecture Size." 2020 International Joint Conference on Neural Networks (IJCNN). 2020 International Joint Conference on Neural Networks (IJCNN). 2020-07-19 - 2020-07-24. https://doi.org/10.1109/ijcnn48605.2020.9206880
Abstract
Determining an appropriate number of features for each layer in a neural network is an important and difficult task. This task is especially important in applications on systems with limited memory or processing power. Many current approaches to reduce network size either utilize iterative procedures, which can extend training time significantly, or require very careful tuning of algorithm parameters to achieve reasonable results. In this paper we propose NodeDrop, a new method for eliminating features in a network. With NodeDrop, we define a condition to identify and guarantee which nodes carry no information, and then use regularization to encourage nodes to meet this condition. We find that NodeDrop drastically reduces the number of features in a network while maintaining high performance. NodeDrop reduces the number of parameters by a factor of 114x for a VGG like network on CIFAR10 without a drop in accuracy.