DeepSITH: efficient learning via decomposition of what and when across time scales
Date
2021
DOI
Authors
Jacques, Brandon G.
Tiganj, Zoran
Howard, Marc W.
Sederberg, Per B.
Version
Published version
OA Version
Citation
B. Jacques, Z. Tiganj, M. Howard, P. Sederberg. "DeepSITH: Efficient learning via decomposition of what and when across time scales." Advances in Neural Information Processing Systems
Abstract
Extracting temporal relationships over a range of scales is a hallmark of human perception and cognition -- and thus it is a critical feature of machine learning applied to real-world problems. Neural networks are either plagued by the exploding/vanishing gradient problem in recurrent neural networks (RNNs) or must adjust their parameters to learn the relevant time scales (e.g., in LSTMs). This paper introduces DeepSITH, a network comprising biologically-inspired Scale-Invariant Temporal History (SITH) modules in series with dense connections between layers. SITH modules respond to their inputs with a geometrically-spaced set of time constants, enabling the DeepSITH network to learn problems along a continuum of time-scales. We compare DeepSITH to LSTMs and other recent RNNs on several time series prediction and decoding tasks. DeepSITH achieves state-of-the-art performance on these problems.