Abstract: This paper introduces a generalized representation of Bayesian inference. It is derived axiomatically, recovering existing Bayesian methods as special cases. We use it to prove that variational inference (VI) with the variational family Q produces the uniquely optimal Qconstrained approximation to the exact Bayesian inference problem. Surprisingly, this implies that VI dominates any other Q-constrained approximation to the exact Bayesian inference problem. This means that alternative Q-constrained approximations like Expectation Propagation (Minka, 2001; Oppeer & Winther, 2000) can produce better posteriors than VI only by implicitly targeting more appropriate Bayesian inference problems. Inspired by this, we introduce Generalized Variational Inference (GVI), a modular approach for instead solving such alternative inference problems explicitly. We explore some applications of GVI, including robust inference and better approximate posterior variances. Lastly, we derive a black box inference scheme and demonstrate it on Bayesian Neural Networks and Deep Gaussian Processes, where GVI substantially outperforms competing methods.