Time adaptive recurrent neural network
Files
Accepted manuscript
Date
2021
Authors
Kag, Anil
Saligrama, Venkatesh
Version
Accepted manuscript
OA Version
Citation
A. Kag, V. Saligrama. 2021. "Time Adaptive Recurrent Neural Network." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, https://doi.org/10.1109/CVPR46437.2021.01490
Abstract
We propose a learning method that, dynamically modi-
fies the time-constants of the continuous-time counterpart
of a vanilla RNN. The time-constants are modified based
on the current observation and hidden state. Our proposal
overcomes the issues of RNN trainability, by mitigating ex-
ploding and vanishing gradient phenomena based on placing
novel constraints on the parameter space, and by suppress-
ing noise in inputs based on pondering over informative
inputs to strengthen their contribution in the hidden state. As
a result, our method is computationally efficient overcoming
overheads of many existing methods that also attempt to
improve RNN training. Our RNNs, despite being simpler
and having light memory footprint, shows competitive per-
formance against standard LSTMs and baseline RNN models
on many benchmark datasets including those that require
long-term memory.