Bayesian inference for the learning rate in Generalised Bayesian inference

Abstract

In Generalised Bayesian Inference (GBI), the learning rate and hyperparameters of the loss must be estimated. These inference-hyperparameters can't be estimated jointly with the other parameters, from the data, by giving them a prior. However, in some settings there exist unknown ``true'' hyperparameter-values about which it is meaningful to have prior belief. It is then possible to use Bayesian inference with held-out data to get hyperparameter-posteriors. We define two hyperparameter posteriors, one based on an ELPPD-utility and one aiming to cover the pseudo-true parameter. The new framework supports estimation and uncertainty quantification for multiple hyperparameters jointly. Experiments show that the resulting GBI-posteriors out-perform Bayesian inference on simulated test data and select optimal or near optimal hyperparameter values in a large real problem of text analysis. Generalised Bayesian inference is particularly useful for combining multiple data sets and most of our examples belong to that setting. We also give asymptotic results for some of the special ``multi-modular'' Generalised Bayes posteriors which we use in our examples.