Robust, accurate stochastic optimization for variational inference
Date
2020-12
DOI
Authors
Dhaka, Akash Kumar
Catalina, Alejandro
Andersen, Michael Riis
Magnusson, Måns
Huggins, Jonathan H.
Vehtari, Aki
Version
OA Version
Published version
Citation
Akash Kumar Dhaka, Alejandro Catalina, Michael Riis Andersen, Måns Magnusson, Jonathan H Huggins, Aki Vehtari. 2020. "Robust, Accurate Stochastic Optimization for Variational Inference." Advances in Neural Information Processing Systems.
Abstract
We consider the problem of fitting variational posterior approximations using
stochastic optimization methods. The performance of these approximations depends
on (1) how well the variational family matches the true posterior distribution,
(2) the choice of divergence, and (3) the optimization of the variational objective.
We show that even in the best-case scenario when the exact posterior belongs to
the assumed variational family, common stochastic optimization methods lead to
poor variational approximations if the problem dimension is moderately large. We
also demonstrate that these methods are not robust across diverse model types.
Motivated by these findings, we develop a more robust and accurate stochastic
optimization framework by viewing the underlying optimization algorithm as producing
a Markov chain. Our approach is theoretically motivated and includes a
diagnostic for convergence and a novel stopping rule, both of which are robust to
noisy evaluations of the objective function. We show empirically that the proposed
framework works well on a diverse set of models: it can automatically detect
stochastic optimization failure or inaccurate variational approximation.