Sparse Online Variational Bayesian Inference

This work aims to study variational Bayesian inference for sparse regression. Sparsity promoting priors have proven to be very successful in regression scenarios since it helps to select meaningful features and avoid overfitting (e.g., LASSO regression or total variation regularization in imaging). We focus on a general class of shrinkage priors that can be represented as a scale mixture of normal distributions with a generalized inverse Gaussian distribution and includes such priors as Laplace, Generalized Jeffrey's, Student-t, and others. However, since shrinkage priors are non-Gaussian, a fully Bayesian solution becomes very expensive, requiring Markov Chain Monte Carlo (MCMC) methods. To alleviate this, we employ a variational approach that leverages a generalization of the expectation-maximization algorithm to recover the best Gaussian approximation to the sparsity-promoting posterior. This approach turns out to be especially fast and scalable in the case of linear models, where it provides approximate uncertainty quantification (UQ) for a substantially smaller cost than fully Bayesian approaches yet keeping comparable accuracy. Besides, the proposed approach supports online inference to process the data in batches and strategies for online hyperparameter estimation. The high performance in terms of the variable selection and UQ is demonstrated for complex real and simulated data examples where it competes against MCMC based methods as well as the other approximate approaches.