Statistical Mechanics of Inference

Abstract

Statistical modeling often involves identifying an optimal estimate to some underlying probability distribution known to satisfy some given constraints. I show here that choosing as estimate the centroid, or center of mass, of the set consistent with the constraints formally minimizes an objective measure of the expected error. Further, I obtain a useful approximation to this point, valid in the thermodynamic limit, that immediately provides much information relating to the full solution set's geometry. For weak constraints, the centroid is close to the popular maximum entropy solution, whereas for strong constraints the two are far apart. Because of this, centroid inference is often substantially more accurate. The results I present allow for its straightforward application.