Moduli space of optimization algorithms

Abstract

We introduce a geometric and operator-theoretic formalism viewing optimization algorithms as discrete connections on a space of update operators. Each iterative method is encoded by two coupled channels-drift and diffusion-whose algebraic curvature measures the deviation from ideal reversibility and determines the attainable order of accuracy. Flat connections correspond to methods whose updates commute up to higher order and thus achieve minimal numerical dissipation while preserving stability. The formalism recovers classical gradient, proximal, and momentum schemes as first-order flat cases and extends naturally to stochastic, preconditioned, and adaptive algorithms through perturbations controlled by curvature order. Explicit gauge corrections yield higher-order variants with guaranteed energy monotonicity and noise stability. The associated determinantal and isomonodromic formulations yield exact nonasymptotic bounds and describe acceleration effects via Painlev\'e-type invariants and Stokes corrections.