A high probability analysis of adaptive SGD with momentum
Files
Accepted manuscript
Date
2020-07-17
DOI
Authors
Li, Xiaoyu
Orabona, Francesco
Version
OA Version
Accepted manuscript
Citation
Xiaoyu Li, Francesco Orabona. 2020. "A high probability analysis of adaptive SGD with momentum." ICML Workshop on Beyond First Order Methods in ML Systems.
https://arxiv.org/abs/2007.14294
Abstract
Stochastic Gradient Descent (SGD) and its variants are the most used algorithms in machine learning applications. In particular, SGD with adaptive learning rates and momentum is the industry standard to train deep networks. Despite the enormous success of these methods, our theoretical understanding of these variants in the non-convex setting is not complete, with most of the results only proving convergence in expectation and with strong assumptions on the stochastic gradients. In this paper, we present a high probability analysis for adaptive and momentum algorithms, under weak assumptions on the function,stochastic gradients, and learning rates. We use it to prove for the first time the convergence of the gradients to zero in high probability in the smooth non convex setting for Delayed AdaGrad with momentum.