Minimax entropy: The statistical physics of optimal models

Abstract

When constructing models of the world, we aim for optimal compressions: models that include as few details as possible while remaining as accurate as possible. But which details -- or features measured in data -- should we choose to include in a model? Here, using the minimum description length principle, we show that the optimal features are the ones that produce the maximum entropy model with minimum entropy, thus yielding a minimax entropy principle. We review applications, which range from machine learning to optimal models of biological networks. Naive implementations, however, are limited to systems with small numbers of states and features. We therefore require new theoretical insights and computational techniques to construct optimal compressions of high-dimensional datasets arising in large-scale experiments.