The relation of bias with risk in empirically constrained inferences

Abstract

We give some results relating asymptotic characterisations of maximum entropy probability measures to characterisations of Bayes optimal classifiers. Our main theorems show that maximum entropy is a universally Bayes optimal decision rule given constraints on one's knowledge about some observed data in terms of an expected loss. We will extend this result to the case of uncertainty in the observations of expected losses by generalising Sanov's theorem to distributions of constraint values.