Revisit the Arimoto-Blahut algorithm: New Analysis with Approximation

Abstract

By the seminal paper of Claude Shannon \cite{Shannon48}, the computation of the capacity of a discrete memoryless channel has been considered as one of the most important and fundamental problems in Information Theory. Nearly 50 years ago, Arimoto and Blahut independently proposed identical algorithms to solve this problem in their seminal papers \cite{Arimoto1972AnAF, Blahut1972ComputationOC}. The Arimoto-Blahut algorithm was proven to converge to the capacity of the channel as $t \to \infty$, with a convergence rate upper bounded by $O\left(\log(m)/t\right)$, where $m$ is the size of the input distribution. Under the assumption that a unique optimal solution is in the interior of the input probability simplex, the convergence becomes inverse exponential after an iteration $t^0$ \cite{Arimoto1972AnAF}. More recently, it was demonstrated in \cite{Nakagawa2020AnalysisOT} that in certain specific cases, the convergence rate is at worst case inverse linear. In this paper, we revisit this fundamental algorithm analyzing its rate of convergence focusing on the approximation of the capacity. Our main result shows that the convergence rate to an $\varepsilon$-optimal solution, for any sufficiently small constant $\varepsilon > 0$, is inverse exponential $O\left(\log(m)/c^t\right)$, for some constant $c > 1$. Given this, we derive new and complementary results for the computation of capacity, particularly in cases where an exact solution is sought.