A Short Note on Stationary Distributions of Unichain Markov Decision Processes

Abstract

Dealing with unichain MDPs, we consider stationary distributions of policies that coincide in all but $n$ states. In these states each policy chooses one of two possible actions. We show that the stationary distributions of n+1 such policies uniquely determine the stationary distributions of all other such policies. An explicit formula for calculation is given.