Variational Equalities of Directed Information and Applications

Abstract

In this paper we introduce two variational equalities of directed information, which are analogous to those of mutual information employed in the Blahut-Arimoto Algorithm (BAA). Subsequently, we introduce nonanticipative Rate Distortion Function (RDF) ${R}^{na}_{0,n}(D)$ defined via directed information introduced in [1], and we establish its equivalence to Gorbunov-Pinsker's nonanticipatory $\epsilon$-entropy $R^{\varepsilon}_{0,n}(D)$. By invoking certain results we first establish existence of the infimizing reproduction distribution for ${R}^{na}_{0,n}(D)$, and then we give its implicit form for the stationary case. Finally, we utilize one of the variational equalities and the closed form expression of the optimal reproduction distribution to provide an algorithm for the computation of ${R}^{na}_{0,n}(D)$.